LLM API

The ray.serve.llm module builds and configures Serve applications that serve large language models.

Builders

serve.llm.build_llm_deployment	Helper to build a single vllm deployment from the given llm config.
serve.llm.build_openai_app	Helper to build an OpenAI compatible app with the llm deployment setup from the given llm serving args.

Configs

serve.llm.LLMConfig	The configuration for starting an LLM deployment.
serve.llm.LLMServingArgs	The configuration for starting an LLM deployment application.
serve.llm.ModelLoadingConfig	The configuration for loading an LLM model.
serve.llm.CloudMirrorConfig	The configuration for mirroring an LLM model from cloud storage.
serve.llm.LoraConfig	The configuration for loading an LLM model with LoRA.

Deployments

serve.llm.LLMServer	The implementation of the vLLM engine deployment.
serve.llm.deployment.PDDecodeServer	Decode-side LLM server for prefill-decode disaggregation.
serve.llm.deployment.PDPrefillServer	Prefill-side LLM server for prefill-decode disaggregation.
serve.llm.deployment.DPServer	Data Parallel LLM Server.