Serve CLI#

The serve command line interface (CLI) lets you deploy, inspect, and manage Serve applications from the command line.

serve#

CLI for managing Serve applications on a Ray cluster.

serve [OPTIONS] COMMAND [ARGS]...

build#

Imports the applications at IMPORT_PATHS and generates a structured, multi-application config for them. If the flag –single-app is set, accepts one application and generates a single-application config. Config outputted from this command can be used by serve deploy or the REST API.

serve build [OPTIONS] IMPORT_PATHS...

Options

-d, --app-dir <app_dir>#

Local directory to look for the IMPORT_PATH (will be inserted into PYTHONPATH). Defaults to ‘.’, meaning that an object in ./main.py can be imported as ‘main.object’. Not relevant if you’re importing from an installed module.

-o, --output-path <output_path>#

Local path where the output config will be written in YAML format. If not provided, the config will be printed to STDOUT.

--grpc-servicer-functions <grpc_servicer_functions>#

Servicer function for adding the method handler to the gRPC server. Defaults to an empty list and no gRPC server is started.

Arguments

IMPORT_PATHS#

Required argument(s)

config#

Gets the current configs of Serve applications on the cluster.

serve config [OPTIONS]

Options

-a, --address <address>#

Address for the Ray dashboard. Defaults to http://localhost:8265. Can also be set using the RAY_DASHBOARD_ADDRESS environment variable.

-n, --name <name>#

Name of an application. Only applies to multi-application mode. If set, this will only fetch the config for the specified application.

controller-health#

Display health metrics for the Ray Serve controller.

Shows performance indicators that help diagnose controller issues, especially as cluster size increases. Metrics include control loop duration statistics, event loop health, component update times, and autoscaling metrics latency.

serve controller-health [OPTIONS]

Options

-a, --address <address>#

Address to use for ray.init(). Can also be set using the RAY_ADDRESS environment variable.

--json#

Output metrics as JSON instead of formatted YAML.

deploy#

Deploy an application from an import path (e.g., main:app) or a group of applications from a YAML config file.

Passed import paths must point to an Application object or a function that returns one. If a function is used, arguments can be passed to it in ‘key=val’ format after the import path, for example:

serve deploy main:app model_path=’/path/to/model.pkl’ num_replicas=5

This command makes a REST API request to a running Ray cluster.

serve deploy [OPTIONS] CONFIG_OR_IMPORT_PATH [ARGUMENTS]...

Options

--runtime-env <runtime_env>#

Path to a local YAML file containing a runtime_env definition. Ignored when deploying from a config file.

--runtime-env-json <runtime_env_json>#

JSON-serialized runtime_env dictionary. Ignored when deploying from a config file.

--working-dir <working_dir>#

Directory containing files that your application(s) will run in. This must be a remote URI to a .zip file (e.g., S3 bucket). This overrides the working_dir in –runtime-env if both are specified. Ignored when deploying from a config file.

--name <name>#

Custom name for the application. Ignored when deploying from a config file.

-a, --address <address>#

Address for the Ray dashboard. Defaults to http://localhost:8265. Can also be set using the RAY_DASHBOARD_ADDRESS environment variable.

Arguments

CONFIG_OR_IMPORT_PATH#

Required argument

ARGUMENTS#

Optional argument(s)

run#

Run an application from an import path (e.g., my_script:app) or a group of applications from a YAML config file.

Passed import paths must point to an Application object or a function that returns one. If a function is used, arguments can be passed to it in ‘key=val’ format after the import path, for example:

serve run my_script:app model_path=’/path/to/model.pkl’ num_replicas=5

If passing a YAML config, existing applications with no code changes will not be updated.

By default, this will block and stream logs to the console. If you Ctrl-C the command, it will shut down Serve on the cluster.

serve run [OPTIONS] CONFIG_OR_IMPORT_PATH [ARGUMENTS]...

Options

--runtime-env <runtime_env>#

Path to a local YAML file containing a runtime_env definition. This will be passed to ray.init() as the default for deployments.

--runtime-env-json <runtime_env_json>#

JSON-serialized runtime_env dictionary. This will be passed to ray.init() as the default for deployments.

--working-dir <working_dir>#

Directory containing files that your application(s) will run in. Can be a local directory or a remote URI to a .zip file (S3, GS, HTTP). This overrides the working_dir in –runtime-env if both are specified. This will be passed to ray.init() as the default for deployments.

-d, --app-dir <app_dir>#

Local directory to look for the IMPORT_PATH (will be inserted into PYTHONPATH). Defaults to ‘.’, meaning that an object in ./main.py can be imported as ‘main.object’. Not relevant if you’re importing from an installed module.

-a, --address <address>#

Address to use for ray.init(). Can also be set using the RAY_ADDRESS environment variable.

--blocking, --non-blocking#

Whether or not this command should be blocking. If blocking, it will loop and log status until Ctrl-C’d, then clean up the app.

-r, --reload#

This is an experimental feature - Listens for changes to files in the working directory, –working-dir or the working_dir in the –runtime-env, and automatically redeploys the application. This will block until Ctrl-C’d, then clean up the app.

--route-prefix <route_prefix>#

Route prefix for the application. This should only be used when running an application specified by import path and will be ignored if running a config file.

--name <name>#

Name of the application. This should only be used when running an application specified by import path and will be ignored if running a config file.

Arguments

CONFIG_OR_IMPORT_PATH#

Required argument

ARGUMENTS#

Optional argument(s)

shutdown#

Shuts down Serve on the cluster, deleting all applications.

serve shutdown [OPTIONS]

Options

-a, --address <address>#

Address for the Ray dashboard. Defaults to http://localhost:8265. Can also be set using the RAY_DASHBOARD_ADDRESS environment variable.

-y, --yes#

Bypass confirmation prompt.

start#

Start Serve on the Ray cluster.

serve start [OPTIONS]

Options

-a, --address <address>#

Address to use for ray.init(). Can also be set using the RAY_ADDRESS environment variable.

--http-host <http_host>#

Host for HTTP proxies to listen on. Defaults to localhost.

--http-port <http_port>#

Port for HTTP proxies to listen on. Defaults to 8000.

--proxy-location <proxy_location>#

Location of the proxies. Defaults to EveryNode.

Options:

ProxyLocation.Disabled | ProxyLocation.HeadOnly | ProxyLocation.EveryNode

--grpc-port <grpc_port>#

Port for gRPC proxies to listen on. Defaults to 9000.

--grpc-servicer-functions <grpc_servicer_functions>#

Servicer function for adding the method handler to the gRPC server. Defaults to an empty list and no gRPC server is started.

status#

Prints status information about all applications on the cluster.

An application may be:

  • NOT_STARTED: the application does not exist.

  • DEPLOYING: the deployments in the application are still deploying and haven’t reached the target number of replicas.

  • RUNNING: all deployments are healthy.

  • DEPLOY_FAILED: the application failed to deploy or reach a running state.

  • DELETING: the application is being deleted, and the deployments in the application are being teared down.

The deployments within each application may be:

  • HEALTHY: all replicas are acting normally and passing their health checks.

  • UNHEALTHY: at least one replica is not acting normally and may not be passing its health check.

  • UPDATING: the deployment is updating.

serve status [OPTIONS]

Options

-a, --address <address>#

Address for the Ray dashboard. Defaults to http://localhost:8265. Can also be set using the RAY_DASHBOARD_ADDRESS environment variable.

-n, --name <name>#

Name of an application. If set, this will display only the status of the specified application.