ray.serve.llm.LLMRouter.chat

async LLMRouter.chat(body: ChatCompletionRequest) → starlette.responses.Response

Given a prompt, the model will return one or more predicted completions, and can also return the probabilities of alternative tokens at each position.

Parameters:

body – The ChatCompletionRequest object.

Returns:

A response object with completions.