DPServer

class ray.serve.llm.deployment.DPServer(llm_config: LLMConfig, *, engine_cls: Type[LLMEngine] | None = None, model_downloader: Type[LoraModelLoader] | None = None)

Bases: DPServer

Data Parallel LLM Server.

This class is used to serve data parallel attention (DP Attention) deployment paradigm, where the attention layers are replicated and the MoE layers are sharded. DP Attention is typically used for models like DeepSeek-V3.

To build a Deployment object you should use build_dp_deployment function. We also expose a lower level API for more control over the deployment class through serve.deployment function.

Examples

from ray import serve
from ray.serve.llm import LLMConfig, build_dp_deployment

# Configure the model
llm_config = LLMConfig(
    model_loading_config=dict(
        model_id="Qwen/Qwen2.5-0.5B-Instruct",
    ),
    engine_kwargs=dict(
        data_parallel_size=2,
        tensor_parallel_size=1,
    ),
    experimental_configs=dict(
        dp_size_per_node=2,
    ),
    accelerator_type="A10G",
)

# Build the deployment
dp_app = build_dp_deployment(llm_config)

# Deploy the application
model_handle = serve.run(dp_app)

Methods

chat	Runs a chat request to the LLM engine and returns the response.
check_health	Check the health of the replica.
collective_rpc	Execute a collective RPC call on all workers.
completions	Runs a completion request to the LLM engine and returns the response.
detokenize	Detokenize the input token IDs.
embeddings	Runs an embeddings request to the engine and returns the response.
is_paused	Check whether the engine is currently paused.
is_sleeping	Check whether the engine is currently sleeping.
pause	Pause generation on the engine.
record_routing_stats	Serve request-router hook, polled by the controller.
reset_prefix_cache	Reset the KV prefix cache on the engine.
resume	Resume generation on the engine after pause.
score	Runs a score request to the engine and returns the response.
sleep	Put the engine to sleep.
start	Start the underlying engine.
start_profile	Start profiling
stop_profile	Stop profiling
sync_init	Synchronous constructor that returns an unstarted instance.
tokenize	Tokenize the input text.
transcriptions	Runs an transcriptions request to the engine and returns the response.
wakeup	Wake up the engine from sleep mode.