on_request_routed#
- RequestRouter.on_request_routed(pending_request: PendingRequest, replica_id: ReplicaID, result: ReplicaResult | None) None[source]#
Called when a request is routed to a replica.
This is used as a callback to update the state of the request router after a response is generated.
resultisNonewhen the caller only picks the replica and sends the request itself (for example, the direct-streaming LLM ingress router, which returns the replica to HAProxy).- Parameters:
pending_request (PendingRequest) – The request that was routed.
replica_id (ReplicaID) – ID of the replica that the request was routed to.
result (ReplicaResult | None) – The result of sending the request, or
None.