Deployment#

class ray.serve.Deployment[source]#

Class (or function) decorated with the @serve.deployment decorator.

This is run on a number of replica actors. Requests to those replicas call this class.

One or more deployments can be composed together into an Application which is then run via serve.run or a config file.

Example:

@serve.deployment
class MyDeployment:
    def __init__(self, name: str):
        self._name = name

    def __call__(self, request):
        return "Hello world!"

app = MyDeployment.bind()
# Run via `serve.run` or the `serve run` CLI command.
serve.run(app)

property name: str#: Unique name of this deployment.

property func_or_class: Callable | str#: Underlying class or function that this deployment wraps.

property num_replicas: int#: Target number of replicas.

property user_config: Any#: Dynamic user-provided config options.

property max_ongoing_requests: int#: Max number of requests a replica can handle at once.

property max_queued_requests: int#: Max number of requests that can be queued in each deployment handle.

property backpressure_config: BackpressureConfig#: Config of the HTTP response for requests rejected due to backpressure.

property ray_actor_options: Dict | None#: Actor options such as resources required for each replica.

bind(*args, **kwargs) → Application[source]#

Bind the arguments to the deployment and return an Application.

The returned Application can be deployed using serve.run (or via config file) or bound to another deployment for composition.

Return a copy of this deployment with updated options.

Only those options passed in will be updated, all others will remain unchanged from the existing deployment.

See the @serve.deployment decorator docs for details on each option.

Parameters:

func_or_class (Callable | None) – The class or function to run as this deployment.
name (DEFAULT | str) – Unique name for this deployment within the application.
version (DEFAULT | str) – Removed. Specifying this argument raises a ValueError.
num_replicas (DEFAULT | int | str | None) – Number of replicas handling requests, or “auto” to autoscale.
ray_actor_options (DEFAULT | Dict | None) – Options for the replica actor, such as resources and runtime environment.
placement_group_bundles (DEFAULT | List[Dict[str, float]]) – Placement group bundles scheduled for each replica. Cannot be combined with max_replicas_per_node.
placement_group_strategy (DEFAULT | str) – Scheduling strategy for those bundles.
placement_group_bundle_label_selector (DEFAULT | List[Dict[str, str]]) – Label selectors applied to the placement group, either one for all bundles or one per bundle.
max_replicas_per_node (DEFAULT | int) – Maximum replicas of this deployment on one node. Cannot be combined with placement_group_bundles.
user_config (DEFAULT | Any | None) – Config passed to the replica’s reconfigure method without restarting replicas.
max_ongoing_requests (DEFAULT | int) – Maximum number of requests a replica can handle at once.
max_queued_requests (DEFAULT | int) – Maximum number of requests queued at each caller. Additional requests are rejected when the queue is full.
backpressure_config (DEFAULT | Dict | BackpressureConfig | None) – HTTP response settings for requests rejected because the queue is full.
autoscaling_config (DEFAULT | Dict | AutoscalingConfig | None) – Autoscaling parameters. Set num_replicas to “auto” or leave it unset when using this.
graceful_shutdown_wait_loop_s (DEFAULT | float) – Seconds between checks for unfinished requests during shutdown.
graceful_shutdown_timeout_s (DEFAULT | float) – Seconds allowed for a graceful shutdown before the replica is forcibly stopped.
health_check_period_s (DEFAULT | float) – Seconds between health checks on a replica.
health_check_timeout_s (DEFAULT | float) – Seconds allowed for a health check to finish before it is considered failed.
logging_config (DEFAULT | Dict | LoggingConfig | None) – Logging options for this deployment’s replicas.
request_router_config (DEFAULT | Dict | RequestRouterConfig | None) – Request routing options for this deployment.
_init_args (DEFAULT | Tuple[Any]) – Internal. Positional arguments for the deployment constructor; use .bind() instead.
_init_kwargs (DEFAULT | Dict[Any, Any]) – Internal. Keyword arguments for the deployment constructor; use .bind() instead.
_internal (bool) – Internal. Set by Serve when it calls this method.
max_constructor_retry_count (DEFAULT | int) – Maximum retries of the deployment constructor before the deployment is marked failed.
gang_scheduling_config (DEFAULT | Dict | GangSchedulingConfig | None) – Options for scheduling replicas together in groups called gangs.
deployment_actors (DEFAULT | List[Dict | DeploymentActorConfig] | None) – Actors shared by all replicas of this deployment.
max_surge_percent (DEFAULT | int) – Extra replicas allowed during a rolling restart, as a percentage of the target count. Must be in [0, 100]. The allowance rounds up to whole replicas or gangs. When positive, replacements start before old replicas stop. If the cluster lacks capacity, old replicas keep serving while replacements wait. Stopping replicas do not count toward the limit, so live actors can exceed the target plus surge while old replicas drain. Lightweight configuration changes to replicas on the new code wait until restarts finish. Setting this to 0 stops old replicas before starting replacements.

Returns:

A copy of this deployment with the given options applied.

Return type:

Deployment