LLMServer

class ray.serve.llm.LLMServer(llm_config: LLMConfig, *, engine_cls: Type[LLMEngine] | None = None, model_downloader: Type[LoraModelLoader] | None = None)

Bases: LLMServer

The implementation of the vLLM engine deployment.

To build a Deployment object you should use build_llm_deployment function. We also expose a lower level API for more control over the deployment class through serve.deployment function.

Examples

from ray import serve
from ray.serve.llm import LLMConfig
from ray.serve.llm.deployment import LLMServer

# Configure the model
llm_config = LLMConfig(
    model_loading_config=dict(
        served_model_name="llama-3.1-8b",
        model_source="meta-llama/Llama-3.1-8b-instruct",
    ),
    deployment_config=dict(
        autoscaling_config=dict(
            min_replicas=1,
            max_replicas=8,
        )
    ),
)

# Build the deployment directly
serve_options = LLMServer.get_deployment_options(llm_config)
llm_app = serve.deployment(LLMServer).options(
    **serve_options).bind(llm_config)

model_handle = serve.run(llm_app)

# Query the model via `chat` api
from ray.serve.llm.openai_api_models import ChatCompletionRequest
request = ChatCompletionRequest(
    model="llama-3.1-8b",
    messages=[
        {
            "role": "user",
            "content": "Hello, world!"
        }
    ]
)
response = ray.get(model_handle.chat(request))
print(response)

Methods

__init__	Asynchronous constructor that returns a fully started instance.
chat	Runs a chat request to the LLM engine and returns the response.
check_health	Check the health of the replica.
collective_rpc	Execute a collective RPC call on all workers.
completions	Runs a completion request to the LLM engine and returns the response.
detokenize	Detokenize the input token IDs.
embeddings	Runs an embeddings request to the engine and returns the response.
is_paused	Check whether the engine is currently paused.
is_sleeping	Check whether the engine is currently sleeping.
pause	Pause generation on the engine.
record_routing_stats	Serve request-router hook, polled by the controller.
reset_prefix_cache	Reset the KV prefix cache on the engine.
resume	Resume generation on the engine after pause.
score	Runs a score request to the engine and returns the response.
sleep	Put the engine to sleep.
start	Start the underlying engine.
start_profile	Start profiling
stop_profile	Stop profiling
sync_init	Synchronous constructor that returns an unstarted instance.
tokenize	Tokenize the input text.
transcriptions	Runs an transcriptions request to the engine and returns the response.
wakeup	Wake up the engine from sleep mode.