RequestRouter

class ray.serve.request_router.RequestRouter(deployment_id: DeploymentID, handle_source: DeploymentHandleSource, self_actor_id: str | None = None, self_actor_handle: ActorHandle | None = None, use_replica_queue_len_cache: bool = False, get_curr_time_s: Callable[[], float] | None = None, create_replica_wrapper_func: Callable[[RunningReplicaInfo], RunningReplica] | None = None, initial_backoff_s: float = 0.025, backoff_multiplier: float = 2, max_backoff_s: float = 0.5, *args, **kwargs)

Bases: ABC

Abstract interface for a request router (how the router calls it).

Serve constructs the router, then calls initialize_state with the config’s request_router_kwargs.

Parameters:

• deployment_id (DeploymentID) – ID of the deployment that the router routes to.

• handle_source (DeploymentHandleSource) – Where the handle that owns the router runs, such as a proxy or a replica.

• self_actor_id (str | None) – ID of the actor that the router runs in, if any. Used as a metric tag.

• self_actor_handle (ActorHandle | None) – Handle to the actor that the router runs in. A router on a proxy pushes this handle to each new replica.

• use_replica_queue_len_cache (bool) – Whether to track replica queue lengths in a local cache.

• get_curr_time_s (Callable[[], float] | None) – Function that returns the current time in seconds. Defaults to time.time.

• create_replica_wrapper_func (Callable[[RunningReplicaInfo], RunningReplica] | None) – Function that wraps a RunningReplicaInfo in a RunningReplica.

• initial_backoff_s (float) – Initial backoff, in seconds, before retrying to route a request.

• backoff_multiplier (float) – Multiplier applied to the backoff after each retry.

• max_backoff_s (float) – Maximum backoff, in seconds, between retries.

• *args – Positional arguments passed to mixins.

• **kwargs – Keyword arguments passed to mixins.

PublicAPI (alpha): This API is in alpha and may change before becoming stable.

Methods

choose_replicas	Chooses a subset of candidate replicas from available replicas.
create_replica_wrapper	Wrap replica information in a RunningReplica.
decrement_queue_len_cache	Decrement the queue length cache for a replica.
initialize_state	Initialize the state of the request router.
on_new_queue_len_info	Update queue length cache with new info received from replica.
on_replica_actor_died	Drop replica from replica set so it's not considered for future requests.
on_replica_actor_unavailable	Invalidate cache entry so active probing is required for the next request.
on_request_completed	Called when a request to a replica has completed.
on_request_routed	Called when a request is routed to a replica.
on_send_request	Increment queue length cache when a request is sent to a replica.
select_available_replicas	Select available replicas from the list of candidates.
update_backoff_params	Update the backoff parameters at runtime.
update_replicas	Update the set of available replicas to be considered for routing.

Attributes

app_name	Name of the app this router is serving.
curr_num_routing_tasks	Current number of routing tasks running.
curr_replicas	Current replicas available to be routed.
max_num_routing_tasks	Max number of routing tasks to run at any time.
max_num_routing_tasks_cap	Hard limit on the maximum number of routing tasks to run.
max_queue_len_response_deadline_s	Maximum deadline for receiving queue length info from replicas.
num_pending_requests	Current number of requests pending assignment.
queue_len_response_deadline_s	Deadline for receiving queue length info from replicas.
replica_queue_len_cache	Get the replica queue length cache.
supports_rejection_protocol	Whether this router supports the rejection protocol.
target_num_routing_tasks	Target number of routing tasks to be running based on pending requests.