ray.serve.llm.LLMServer.completions

async LLMServer.completions(request: CompletionRequest) → AsyncGenerator[List[str | ErrorResponse] | CompletionResponse, None]

Runs a completion request to the LLM engine and returns the response.

Args:

request: A CompletionRequest object.

Returns:

An AsyncGenerator of the response. If stream is True and batching is enabled, then the generator will yield a list of completion streaming responses (strings of the format data: {response_json}

). Otherwise, it will yield the CompletionResponse object directly.