BackpressureConfig#

pydantic model ray.serve.config.BackpressureConfig[source]#

Config for the HTTP response returned on backpressure rejections.

When a deployment’s max_queued_requests limit is reached, additional requests are rejected. This class configures the HTTP response for those rejections; requests rejected because the deployment is unavailable (e.g., it failed to deploy) always return 503. The gRPC path is unaffected: backpressure rejections always map to RESOURCE_EXHAUSTED.

Example

from ray import serve
from ray.serve.config import BackpressureConfig

@serve.deployment(
    max_queued_requests=64,
    backpressure_config=BackpressureConfig(
        status_code=429,
        retry_after_s=5,
    ),
)
class Deployment:
    ...

Parameters:

status_code – HTTP status code returned for requests rejected due to backpressure. Must be 503 (default) or 429.
retry_after_s – If set, rejected HTTP responses include a Retry-After header with this value (rounded up to an integer number of seconds). Defaults to None (no header).

PublicAPI (alpha): This API is in alpha and may change before becoming stable.

field retry_after_s: Annotated[float, Ge(ge=0)] | None = None#

Constraints:

allow_inf_nan = False

field status_code: Literal[503, 429] = 503#