ray.serve.schema.DeploymentSchema

pydantic model ray.serve.schema.DeploymentSchema

Specifies options for one deployment within a Serve application. For each deployment this can optionally be included in ServeApplicationSchema to override deployment options specified in code.

Show JSON schema

{
   "title": "DeploymentSchema",
   "description": "Specifies options for one deployment within a Serve application. For each deployment\nthis can optionally be included in `ServeApplicationSchema` to override deployment\noptions specified in code.",
   "type": "object",
   "properties": {
      "name": {
         "title": "Name",
         "description": "Globally-unique name identifying this deployment.",
         "type": "string"
      },
      "num_replicas": {
         "title": "Num Replicas",
         "description": "The number of processes that handle requests to this deployment. Uses a default if null.",
         "default": 1,
         "exclusiveMinimum": 0,
         "type": "integer"
      },
      "route_prefix": {
         "title": "Route Prefix",
         "description": "[DEPRECATED] Please use route_prefix under ServeApplicationSchema instead.",
         "default": 1,
         "type": "string"
      },
      "max_concurrent_queries": {
         "title": "Max Concurrent Queries",
         "description": "The max number of pending queries in a single replica. Uses a default if null.",
         "default": 1,
         "exclusiveMinimum": 0,
         "type": "integer"
      },
      "user_config": {
         "title": "User Config",
         "description": "Config to pass into this deployment's reconfigure method. This can be updated dynamically without restarting replicas",
         "default": 1,
         "type": "object"
      },
      "autoscaling_config": {
         "title": "Autoscaling Config",
         "description": "Config specifying autoscaling parameters for the deployment's number of replicas. If null, the deployment won't autoscale its number of replicas; the number of replicas will be fixed at num_replicas.",
         "default": 1,
         "type": "object"
      },
      "graceful_shutdown_wait_loop_s": {
         "title": "Graceful Shutdown Wait Loop S",
         "description": "Duration that deployment replicas will wait until there is no more work to be done before shutting down. Uses a default if null.",
         "default": 1,
         "minimum": 0,
         "type": "number"
      },
      "graceful_shutdown_timeout_s": {
         "title": "Graceful Shutdown Timeout S",
         "description": "Serve controller waits for this duration before forcefully killing the replica for shutdown. Uses a default if null.",
         "default": 1,
         "minimum": 0,
         "type": "number"
      },
      "health_check_period_s": {
         "title": "Health Check Period S",
         "description": "Frequency at which the controller will health check replicas. Uses a default if null.",
         "default": 1,
         "exclusiveMinimum": 0,
         "type": "number"
      },
      "health_check_timeout_s": {
         "title": "Health Check Timeout S",
         "description": "Timeout that the controller will wait for a response from the replica's health check before marking it unhealthy. Uses a default if null.",
         "default": 1,
         "exclusiveMinimum": 0,
         "type": "number"
      },
      "ray_actor_options": {
         "title": "Ray Actor Options",
         "description": "Options set for each replica actor.",
         "default": 1,
         "allOf": [
            {
               "$ref": "#/definitions/RayActorOptionsSchema"
            }
         ]
      },
      "placement_group_bundles": {
         "title": "Placement Group Bundles",
         "description": "Define a set of placement group bundles to be scheduled *for each replica* of this deployment. The replica actor will be scheduled in the first bundle provided, so the resources specified in `ray_actor_options` must be a subset of the first bundle's resources. All actors and tasks created by the replica actor will be scheduled in the placement group by default (`placement_group_capture_child_tasks` is set to True).",
         "default": 1,
         "type": "array",
         "items": {
            "type": "object",
            "additionalProperties": {
               "type": "number"
            }
         }
      },
      "placement_group_strategy": {
         "title": "Placement Group Strategy",
         "description": "Strategy to use for the replica placement group specified via `placement_group_bundles`. Defaults to `PACK`.",
         "default": 1,
         "type": "string"
      },
      "max_replicas_per_node": {
         "title": "Max Replicas Per Node",
         "description": "[EXPERIMENTAL] The max number of deployment replicas can run on a single node. Valid values are None (no limitation) or an integer in the range of [1, 100]. Defaults to no limitation.",
         "default": 1,
         "type": "integer"
      }
   },
   "required": [
      "name"
   ],
   "definitions": {
      "RayActorOptionsSchema": {
         "title": "RayActorOptionsSchema",
         "description": "Options with which to start a replica actor.",
         "type": "object",
         "properties": {
            "runtime_env": {
               "title": "Runtime Env",
               "description": "This deployment's runtime_env. working_dir and py_modules may contain only remote URIs.",
               "default": {},
               "type": "object"
            },
            "num_cpus": {
               "title": "Num Cpus",
               "description": "The number of CPUs required by the deployment's application per replica. This is the same as a ray actor's num_cpus. Uses a default if null.",
               "minimum": 0,
               "type": "number"
            },
            "num_gpus": {
               "title": "Num Gpus",
               "description": "The number of GPUs required by the deployment's application per replica. This is the same as a ray actor's num_gpus. Uses a default if null.",
               "minimum": 0,
               "type": "number"
            },
            "memory": {
               "title": "Memory",
               "description": "Restrict the heap memory usage of each replica. Uses a default if null.",
               "minimum": 0,
               "type": "number"
            },
            "object_store_memory": {
               "title": "Object Store Memory",
               "description": "Restrict the object store memory used per replica when creating objects. Uses a default if null.",
               "minimum": 0,
               "type": "number"
            },
            "resources": {
               "title": "Resources",
               "description": "The custom resources required by each replica.",
               "default": {},
               "type": "object"
            },
            "accelerator_type": {
               "title": "Accelerator Type",
               "description": "Forces replicas to run on nodes with the specified accelerator type.",
               "type": "string"
            }
         }
      }
   }
}

Fields

• autoscaling_config (Optional[Dict])

• graceful_shutdown_timeout_s (float)

• graceful_shutdown_wait_loop_s (float)

• health_check_period_s (float)

• health_check_timeout_s (float)

• max_concurrent_queries (int)

• max_replicas_per_node (int)

• name (str)

• num_replicas (Optional[int])

• placement_group_bundles (List[Dict[str, float]])

• placement_group_strategy (str)

• ray_actor_options (ray.serve.schema.RayActorOptionsSchema)

• route_prefix (Optional[str])

• user_config (Optional[Dict])

Validators

• num_replicas_and_autoscaling_config_mutually_exclusive » all fields

field autoscaling_config: Optional[Dict] = DEFAULT.VALUE

Config specifying autoscaling parameters for the deployment’s number of replicas. If null, the deployment won’t autoscale its number of replicas; the number of replicas will be fixed at num_replicas.

Validated by

• num_replicas_and_autoscaling_config_mutually_exclusive

field graceful_shutdown_timeout_s: float = DEFAULT.VALUE

Serve controller waits for this duration before forcefully killing the replica for shutdown. Uses a default if null.

Constraints

• minimum = 0

Validated by

• num_replicas_and_autoscaling_config_mutually_exclusive

field graceful_shutdown_wait_loop_s: float = DEFAULT.VALUE

Duration that deployment replicas will wait until there is no more work to be done before shutting down. Uses a default if null.

Constraints

• minimum = 0

Validated by

• num_replicas_and_autoscaling_config_mutually_exclusive

field health_check_period_s: float = DEFAULT.VALUE

Frequency at which the controller will health check replicas. Uses a default if null.

Constraints

• exclusiveMinimum = 0

Validated by

• num_replicas_and_autoscaling_config_mutually_exclusive

field health_check_timeout_s: float = DEFAULT.VALUE

Timeout that the controller will wait for a response from the replica’s health check before marking it unhealthy. Uses a default if null.

Constraints

• exclusiveMinimum = 0

Validated by

• num_replicas_and_autoscaling_config_mutually_exclusive

field max_concurrent_queries: int = DEFAULT.VALUE

The max number of pending queries in a single replica. Uses a default if null.

Constraints

• exclusiveMinimum = 0

Validated by

• num_replicas_and_autoscaling_config_mutually_exclusive

field max_replicas_per_node: int = DEFAULT.VALUE

[EXPERIMENTAL] The max number of deployment replicas can run on a single node. Valid values are None (no limitation) or an integer in the range of [1, 100]. Defaults to no limitation.

Validated by

• num_replicas_and_autoscaling_config_mutually_exclusive

field name: str [Required]

Globally-unique name identifying this deployment.

Validated by

• num_replicas_and_autoscaling_config_mutually_exclusive

field num_replicas: Optional[int] = DEFAULT.VALUE

The number of processes that handle requests to this deployment. Uses a default if null.

Constraints

• exclusiveMinimum = 0

Validated by

• num_replicas_and_autoscaling_config_mutually_exclusive

field placement_group_bundles: List[Dict[str, float]] = DEFAULT.VALUE

Define a set of placement group bundles to be scheduled for each replica of this deployment. The replica actor will be scheduled in the first bundle provided, so the resources specified in ray_actor_options must be a subset of the first bundle’s resources. All actors and tasks created by the replica actor will be scheduled in the placement group by default (placement_group_capture_child_tasks is set to True).

Validated by

• num_replicas_and_autoscaling_config_mutually_exclusive

field placement_group_strategy: str = DEFAULT.VALUE

Strategy to use for the replica placement group specified via placement_group_bundles. Defaults to PACK.

Validated by

• num_replicas_and_autoscaling_config_mutually_exclusive

field ray_actor_options: ray.serve.schema.RayActorOptionsSchema = DEFAULT.VALUE

Options set for each replica actor.

Validated by

• num_replicas_and_autoscaling_config_mutually_exclusive

field route_prefix: Optional[str] = DEFAULT.VALUE

[DEPRECATED] Please use route_prefix under ServeApplicationSchema instead.

Validated by

• num_replicas_and_autoscaling_config_mutually_exclusive

• _route_prefix_format

field user_config: Optional[Dict] = DEFAULT.VALUE

Config to pass into this deployment’s reconfigure method. This can be updated dynamically without restarting replicas

Validated by

• num_replicas_and_autoscaling_config_mutually_exclusive

classmethod deployment_schema_route_prefix_format(v)

The route_prefix 1. must start with a / character 2. must not end with a / character (unless the entire prefix is just /) 3. cannot contain wildcards (must not have “{” or “}”)

get_user_configured_option_names() → Set[str]

Get set of names for all user-configured options.

Any field not set to DEFAULT.VALUE is considered a user-configured option.

validator num_replicas_and_autoscaling_config_mutually_exclusive  »  all fields