PDPrefillServer

class ray.serve.llm.deployment.PDPrefillServer(llm_config: LLMConfig, *, engine_cls: Type[LLMEngine] | None = None, model_downloader: Type[LoraModelLoader] | None = None)

Bases: PDPrefillServer

Prefill-side LLM server for prefill-decode disaggregation.

A standard LLMServer with an additional prewarm_prefill method used during the optional pre-warm handshake.

Methods

__init__	Asynchronous constructor that returns a fully started instance.
chat	Runs a chat request to the LLM engine and returns the response.
check_health	Check the health of the replica.
collective_rpc	Execute a collective RPC call on all workers.
completions	Runs a completion request to the LLM engine and returns the response.
detokenize	Detokenize the input token IDs.
embeddings	Runs an embeddings request to the engine and returns the response.
is_paused	Check whether the engine is currently paused.
is_sleeping	Check whether the engine is currently sleeping.
pause	Pause generation on the engine.
prewarm_prefill	Run one prefill pass and return kv_transfer_params as a dict.
record_replica_metadata	Publish this prefill replica's connector coordination metadata.
record_routing_stats	Serve request-router hook, polled by the controller.
reset_prefix_cache	Reset the KV prefix cache on the engine.
resume	Resume generation on the engine after pause.
score	Runs a score request to the engine and returns the response.
sleep	Put the engine to sleep.
start	Start the underlying engine.
start_profile	Start profiling
stop_profile	Stop profiling
sync_init	Synchronous constructor that returns an unstarted instance.
tokenize	Tokenize the input text.
transcriptions	Runs an transcriptions request to the engine and returns the response.
wakeup	Wake up the engine from sleep mode.