PDDecodeServer#

class ray.serve.llm.deployment.PDDecodeServer(llm_config: LLMConfig, *, prefill_server: DeploymentHandle, engine_cls=None, model_downloader=None)[source]#

Bases: PDDecodeServer

Decode-side LLM server for prefill-decode disaggregation.

This deployment owns a real engine (decode config) and holds a handle to the prefill deployment. For chat/completions it runs remote prefill first, then local decode.

Use build_pd_openai_app to construct the full 3-tier PD graph.

Methods

`check_health`	Check the health of the replica.
`collective_rpc`	Execute a collective RPC call on all workers.
`detokenize`	Detokenize the input token IDs.
`embeddings`	Runs an embeddings request to the engine and returns the response.
`is_paused`	Check whether the engine is currently paused.
`is_sleeping`	Check whether the engine is currently sleeping.
`pause`	Pause generation on the engine.
`record_routing_stats`	Serve request-router hook, polled by the controller.
`reset_prefix_cache`	Reset the KV prefix cache on the engine.
`resume`	Resume generation on the engine after pause.
`score`	Runs a score request to the engine and returns the response.
`sleep`	Put the engine to sleep.
`start`	Start the underlying engine.
`start_profile`	Start profiling
`stop_profile`	Stop profiling
`sync_init`	Synchronous constructor that returns an unstarted instance.
`tokenize`	Tokenize the input text.
`transcriptions`	Runs an transcriptions request to the engine and returns the response.
`wakeup`	Wake up the engine from sleep mode.