PrometheusQueryMixin#

class ray.serve.autoscaling_policy.PrometheusQueryMixin(*, prometheus_address: str | None = None, prometheus_queries: List[str] | None = None, fetch_interval_s: float = 5.0, cache_ttl_s: float = 15.0, **kwargs)[source]#

Bases: object

Keeps Prometheus query results fresh for an autoscaling policy.

Mix into a policy and read self.prometheus_results from __call__. Scalar results and instant vectors, including empty and multi-sample vectors, retain their Prometheus result types. self.prometheus_metrics is a convenience view containing only scalars and single-sample vectors. Results are returned as a dict mapping each query string to its value.

The first read starts a daemon thread that evaluates the queries every fetch_interval_s. Reads never block on the network and return None when Prometheus is unset, unreachable, or the cache is older than cache_ttl_s.

prometheus_address defaults to the RAY_PROMETHEUS_HOST environment variable, which Ray’s dashboard and managed clusters already set, so the common case needs no address. HTTP headers are read from the JSON-encoded RAY_PROMETHEUS_HEADERS environment variable used by the dashboard.

Example

from ray.serve.autoscaling_policy import PrometheusQueryMixin

QUERY = "sum(my_queue_depth)"

class QueueDepthPolicy(PrometheusQueryMixin):
    def __init__(self, **kwargs):
        super().__init__(prometheus_queries=[QUERY], **kwargs)

    def __call__(self, ctx):
        metrics = self.prometheus_metrics or {}
        queue_depth = metrics.get(QUERY)
        if queue_depth is None:
            return ctx.target_num_replicas, {}
        desired = ctx.target_num_replicas
        if queue_depth > 10:
            desired += 1
        return desired, {}

Parameters:

prometheus_address (str | None) – Base URL of the Prometheus server. Falls back to the RAY_PROMETHEUS_HOST environment variable.
prometheus_queries (List[str] | None) – PromQL expressions to evaluate on every fetch.
fetch_interval_s (float) – Seconds between completion of background fetch and the next fetch.
cache_ttl_s (float) – Maximum age in seconds of cached results. Reads return None once the cache is older than this.
**kwargs – Forwarded to super().__init__.

PublicAPI (alpha): This API is in alpha and may change before becoming stable.

property prometheus_results: Dict[str, PrometheusScalar | PrometheusVector] | None#: Latest typed scalar and vector results, or None if unavailable.

property prometheus_metrics: Dict[str, float] | None#: Latest unambiguous scalar values, or None if none are available.