AutoscalingContext#

class ray.serve.config.AutoscalingContext(deployment_id: DeploymentID, deployment_name: str, app_name: str | None, current_num_replicas: int, target_num_replicas: int, running_replicas: List[ReplicaID], total_num_requests: float | Callable[[], float], total_queued_requests: float | Callable[[], float] | None, aggregated_metrics: Dict[str, Dict[ReplicaID, float]] | Callable[[], Dict[str, Dict[ReplicaID, float]]] | None, raw_metrics: Dict[str, Dict[ReplicaID, List[TimeStampedValue]]] | Callable[[], Dict[str, Dict[ReplicaID, List[TimeStampedValue]]]] | None, capacity_adjusted_min_replicas: int, capacity_adjusted_max_replicas: int, policy_state: Dict[str, Any], last_scale_up_time: float | None, last_scale_down_time: float | None, current_time: float | None, config: Any | None)[source]#

Bases: object

Rich context provided to custom autoscaling policies.

This class provides comprehensive information about a deployment’s current state, metrics, and configuration that can be used by custom autoscaling policies to make intelligent scaling decisions.

The context includes deployment metadata, current replica state, built-in and custom metrics, capacity bounds, policy state, and timing information.

Note: The aggregated_metrics and raw_metrics fields support lazy evaluation. You can pass callables that will be evaluated only when accessed, with results cached for subsequent accesses.

Note: total_num_requests and total_queued_requests are both aggregated over look_back_period_s using the deployment’s aggregation_function, so under min or max they report the window trough or peak rather than the current value.

Serve constructs this object and passes it to the policy on each autoscaling iteration. You don’t construct it yourself.

Parameters:

  • deployment_id (DeploymentID) – Unique identifier of the deployment.

  • deployment_name (str) – Name of the deployment.

  • app_name (str | None) – Name of the application that contains the deployment.

  • current_num_replicas (int) – Number of replicas currently running.

  • target_num_replicas (int) – Number of replicas the autoscaler currently targets.

  • running_replicas (List[ReplicaID]) – IDs of the replicas currently running.

  • total_num_requests (float | Callable[[], float]) – Ongoing requests, running plus queued, across the deployment, or a callable that returns that value on first access.

  • total_queued_requests (float | Callable[[], float] | None) – Requests queued at handles, or a callable that returns that value on first access.

  • aggregated_metrics (Dict[str, Dict[ReplicaID, float]] | Callable[[], Dict[str, Dict[ReplicaID, float]]] | None) – Custom metrics aggregated per replica over the look-back window, keyed by metric name and then replica ID, or a callable that returns them on first access.

  • raw_metrics (Dict[str, Dict[ReplicaID, List[TimeStampedValue]]] | Callable[[], Dict[str, Dict[ReplicaID, List[TimeStampedValue]]]] | None) – Custom metric time series per replica, keyed by metric name and then replica ID, or a callable that returns them on first access.

  • capacity_adjusted_min_replicas (int) – Lower replica bound, scaled by the cluster’s target capacity.

  • capacity_adjusted_max_replicas (int) – Upper replica bound, scaled by the cluster’s target capacity.

  • policy_state (Dict[str, Any]) – State that the policy returned on its previous call, which persists across autoscaling iterations.

  • last_scale_up_time (float | None) – Unix timestamp, in seconds, of the last scale-up, or None if the deployment hasn’t scaled up.

  • last_scale_down_time (float | None) – Unix timestamp, in seconds, of the last scale-down, or None if the deployment hasn’t scaled down.

  • current_time (float | None) – Unix timestamp, in seconds, when Serve built this context.

  • config (Any | None) – The deployment’s autoscaling configuration.

deployment_id#

Unique identifier for the deployment.

deployment_name#

Name of the deployment.

app_name#

Name of the application containing this deployment.

capacity_adjusted_min_replicas#

Minimum replicas adjusted for cluster capacity.

capacity_adjusted_max_replicas#

Maximum replicas adjusted for cluster capacity.

current_time#

Current timestamp.

config#

Autoscaling configuration for this deployment.

property aggregated_metrics: Dict[str, Dict[ReplicaID, float]] | None#

Custom metrics aggregated per replica over the look-back window.

Keyed by metric name, then by replica ID. Evaluated on first access and cached.

property raw_metrics: Dict[str, Dict[ReplicaID, List[TimeStampedValue]]] | None#

Custom metric time series per replica.

Keyed by metric name, then by replica ID. Evaluated on first access and cached.

property total_num_requests: float#

Ongoing requests, running plus queued, across the deployment.

Evaluated on first access and cached.

property total_queued_requests: float#

Requests queued at handles. Evaluated on first access and cached.

property total_running_requests: float#

Approximate number of requests that replicas are processing.

Computed as total_num_requests minus total_queued_requests, clamped at zero.