User guides#

How-to guides for deploying, scaling, and operating Ray Serve LLM. If you are new, start with the Quickstart, then come back here to go deeper.

Configure and deploy#

  • Configuration reference: every LLMConfig field, from model loading and engine kwargs to accelerators, placement, and deployment options.

  • Deployment initialization: speed up model loading and replica startup with caching, streaming load formats, and initialization callbacks.

  • Multi-LoRA deployment: serve many LoRA adapters on a shared base model with runtime switching and an LRU cache.

Scale across GPUs and nodes#

Optimize latency and throughput#

  • Prefill/decode disaggregation: split prompt processing and token generation onto separate replicas to tune each independently.

  • Direct streaming: bypass the ingress when streaming tokens to cut per-token latency.

  • Approximate prefix cache aware routing: route requests to replicas that already hold a matching prefix to maximize cache hits.

  • Exact KV cache aware routing: route requests to replicas based on KV cache overlap and token load, accounting for uncached prefill tokens and ongoing decode load.

  • KV cache offloading: extend KV cache capacity with native vLLM CPU offloading, LMCache, or tiered storage backends. Pair it with a router that accounts for KV caches across storage tiers.

Choose an engine#

  • vLLM compatibility: use vLLM features such as embeddings, structured outputs, vision, and reasoning through Ray Serve LLM.

  • Custom vLLM models: serve an out-of-tree architecture with a vLLM plugin, using a Qwen3 reward model as the example.

  • SGLang integration: run SGLang as the inference engine instead of vLLM.

Accelerator-specific serving#

  • TPU serving: serve a model on single-host or multi-host TPU slices with topology-aware placement.

Operate in production#