tokenize

async PDPrefillServer.tokenize(request: TokenizeRequest, raw_request_info: RawRequestInfo | None = None) → AsyncGenerator[TokenizeResponse | ErrorResponse, None]

Tokenize the input text.

Parameters:

• request (TokenizeRequest) – A TokenizeRequest object (TokenizeCompletionRequest or TokenizeChatRequest).

• raw_request_info (RawRequestInfo | None) – Optional RawRequestInfo containing data from the original HTTP request.

Returns:

An AsyncGenerator over the TokenizeResponse object.

Return type:

AsyncGenerator[TokenizeResponse | ErrorResponse, None]