select_available_replicas#
- RequestRouter.select_available_replicas(candidates: List[RunningReplica] | None = None) List[RunningReplica][source]#
Select available replicas from the list of candidates.
This method is used to select replicas that are available to take more requests based on the queue length cache. If the queue length is not available in the cache, the replica is considered available. It does not actively probe the replicas for their queue length.
If input candidates is
None, all replicas are considered.- Parameters:
candidates (List[RunningReplica] | None) – The replicas to select from. Defaults to all replicas.
- Returns:
The candidates whose cached queue length is unknown or below their
max_ongoing_requests.- Return type: