Cluster Management CLI#

This section contains commands for managing Ray clusters.

ray start#

Start Ray processes manually on the local machine.

ray start [OPTIONS]

Options

--node-ip-address <node_ip_address>#

the IP address of this node

--address <address>#

the address to use for Ray

--port <port>#

the port of the head ray process. If not provided, defaults to 6379; if port is set to 0, we will allocate an available port.

--object-manager-port <object_manager_port>#

the port to use for starting the object manager

--node-manager-port <node_manager_port>#

the port to use for starting the node manager

--gcs-server-port <gcs_server_port>#

Port number for the GCS server.

--min-worker-port <min_worker_port>#

the lowest port number that workers will bind on. If not set, random ports will be chosen.

--max-worker-port <max_worker_port>#

the highest port number that workers will bind on. If set, ‘–min-worker-port’ must also be set.

--worker-port-list <worker_port_list>#

a comma-separated list of open ports for workers to bind on. Overrides ‘–min-worker-port’ and ‘–max-worker-port’.

--ray-client-server-port <ray_client_server_port>#

the port number the ray client server binds on, default to 10001, or None if ray[client] is not installed.

--object-store-memory <object_store_memory>#

The amount of memory (in bytes) to start the object store with. By default, this is 30% (ray_constants.DEFAULT_OBJECT_STORE_MEMORY_PROPORTION) of available system memory capped by the shm size and 200G (ray_constants.DEFAULT_OBJECT_STORE_MAX_MEMORY_BYTES) but can be set higher.

--num-cpus <num_cpus>#

the number of CPUs on this node

--num-gpus <num_gpus>#

the number of GPUs on this node

--resources <resources>#

A JSON serialized dictionary mapping resource name to resource quantity.

--head#

provide this argument for the head node

--include-dashboard <include_dashboard>#

provide this argument to start the Ray dashboard GUI

--dashboard-host <dashboard_host>#

the host to bind the dashboard server to, either localhost (127.0.0.1) or 0.0.0.0 (available from all interfaces). By default, this is 127.0.0.1

--dashboard-port <dashboard_port>#

the port to bind the dashboard server to–defaults to 8265

--dashboard-agent-listen-port <dashboard_agent_listen_port>#

the port for dashboard agents to listen for http on.

--dashboard-agent-grpc-port <dashboard_agent_grpc_port>#

the port for dashboard agents to listen for grpc on.

--dashboard-grpc-port <dashboard_grpc_port>#

The port for the dashboard head to listen for grpc on.

--runtime-env-agent-port <runtime_env_agent_port>#

The port for the runtime enviroment agents to listen for http on.

--block#

provide this argument to block forever in this command

--plasma-directory <plasma_directory>#

object store directory for memory mapped files

--autoscaling-config <autoscaling_config>#

the file that contains the autoscaling config

--no-redirect-output#

do not redirect non-worker stdout and stderr to files

--plasma-store-socket-name <plasma_store_socket_name>#

manually specify the socket name of the plasma store

--raylet-socket-name <raylet_socket_name>#

manually specify the socket path of the raylet process

--temp-dir <temp_dir>#

manually specify the root temporary dir of the Ray process, only works when –head is specified

--storage <storage>#

the persistent storage URI for the cluster. Experimental.

--metrics-export-port <metrics_export_port>#

the port to use to expose Ray metrics through a Prometheus endpoint.

--ray-debugger-external#

Make the Ray debugger available externally to the node. This is only safe to activate if the node is behind a firewall.

--disable-usage-stats#

If True, the usage stats collection will be disabled.

--log-style <log_style>#

If ‘pretty’, outputs with formatting and color. If ‘record’, outputs record-style without formatting. ‘auto’ defaults to ‘pretty’, and disables pretty logging if stdin is not a TTY.

Options:

auto | record | pretty

--log-color <log_color>#

Use color logging. Auto enables color logging if stdout is a TTY.

Options:

auto | false | true

-v, --verbose#

ray stop#

Stop Ray processes manually on the local machine.

ray stop [OPTIONS]

Options

-f, --force#

If set, ray will send SIGKILL instead of SIGTERM.

-g, --grace-period <grace_period>#

The time in seconds ray waits for processes to be properly terminated. If processes are not terminated within the grace period, they are forcefully terminated after the grace period.

--log-style <log_style>#

If ‘pretty’, outputs with formatting and color. If ‘record’, outputs record-style without formatting. ‘auto’ defaults to ‘pretty’, and disables pretty logging if stdin is not a TTY.

Options:

auto | record | pretty

--log-color <log_color>#

Use color logging. Auto enables color logging if stdout is a TTY.

Options:

auto | false | true

-v, --verbose#

ray up#

Create or update a Ray cluster.

ray up [OPTIONS] CLUSTER_CONFIG_FILE

Options

--min-workers <min_workers>#

Override the configured min worker node count for the cluster.

--max-workers <max_workers>#

Override the configured max worker node count for the cluster.

--no-restart#

Whether to skip restarting Ray services during the update. This avoids interrupting running jobs.

--restart-only#

Whether to skip running setup commands and only restart Ray. This cannot be used with ‘no-restart’.

-y, --yes#

Don’t ask for confirmation.

-n, --cluster-name <cluster_name>#

Override the configured cluster name.

--no-config-cache#

Disable the local cluster config cache.

--redirect-command-output#

Whether to redirect command output to a file.

--use-login-shells, --use-normal-shells#

Ray uses login shells (bash –login -i) to run cluster commands by default. If your workflow is compatible with normal shells, this can be disabled for a better user experience.

--disable-usage-stats#

If True, the usage stats collection will be disabled.

--log-style <log_style>#

If ‘pretty’, outputs with formatting and color. If ‘record’, outputs record-style without formatting. ‘auto’ defaults to ‘pretty’, and disables pretty logging if stdin is not a TTY.

Options:

auto | record | pretty

--log-color <log_color>#

Use color logging. Auto enables color logging if stdout is a TTY.

Options:

auto | false | true

-v, --verbose#

Arguments

CLUSTER_CONFIG_FILE#

Required argument

ray down#

Tear down a Ray cluster.

ray down [OPTIONS] CLUSTER_CONFIG_FILE

Options

-y, --yes#

Don’t ask for confirmation.

--workers-only#

Only destroy the workers.

-n, --cluster-name <cluster_name>#

Override the configured cluster name.

--keep-min-workers#

Retain the minimal amount of workers specified in the config.

--log-style <log_style>#

If ‘pretty’, outputs with formatting and color. If ‘record’, outputs record-style without formatting. ‘auto’ defaults to ‘pretty’, and disables pretty logging if stdin is not a TTY.

Options:

auto | record | pretty

--log-color <log_color>#

Use color logging. Auto enables color logging if stdout is a TTY.

Options:

auto | false | true

-v, --verbose#

Arguments

CLUSTER_CONFIG_FILE#

Required argument

ray exec#

Execute a command via SSH on a Ray cluster.

ray exec [OPTIONS] CLUSTER_CONFIG_FILE CMD

Options

--run-env <run_env>#

Choose whether to execute this command in a container or directly on the cluster head. Only applies when docker is configured in the YAML.

Options:

auto | host | docker

--stop#

Stop the cluster after the command finishes running.

--start#

Start the cluster if needed.

--screen#

Run the command in a screen.

--tmux#

Run the command in tmux.

-n, --cluster-name <cluster_name>#

Override the configured cluster name.

--no-config-cache#

Disable the local cluster config cache.

-p, --port-forward <port_forward>#

Port to forward. Use this multiple times to forward multiple ports.

--disable-usage-stats#

If True, the usage stats collection will be disabled.

--log-style <log_style>#

If ‘pretty’, outputs with formatting and color. If ‘record’, outputs record-style without formatting. ‘auto’ defaults to ‘pretty’, and disables pretty logging if stdin is not a TTY.

Options:

auto | record | pretty

--log-color <log_color>#

Use color logging. Auto enables color logging if stdout is a TTY.

Options:

auto | false | true

-v, --verbose#

Arguments

CLUSTER_CONFIG_FILE#

Required argument

CMD#

Required argument

ray submit#

Uploads and runs a script on the specified cluster.

The script is automatically synced to the following location:

os.path.join(“~”, os.path.basename(script))

Example:

ray submit [CLUSTER.YAML] experiment.py – –smoke-test

ray submit [OPTIONS] CLUSTER_CONFIG_FILE SCRIPT [SCRIPT_ARGS]...

Options

--stop#

Stop the cluster after the command finishes running.

--start#

Start the cluster if needed.

--screen#

Run the command in a screen.

--tmux#

Run the command in tmux.

-n, --cluster-name <cluster_name>#

Override the configured cluster name.

--no-config-cache#

Disable the local cluster config cache.

-p, --port-forward <port_forward>#

Port to forward. Use this multiple times to forward multiple ports.

--args <args>#

(deprecated) Use ‘– –arg1 –arg2’ for script args.

--disable-usage-stats#

If True, the usage stats collection will be disabled.

--extra-screen-args <extra_screen_args>#

if screen is enabled, add the provided args to it. A useful example usage scenario is passing –extra-screen-args=’-Logfile /full/path/blah_log.txt’ as it redirects screen output also to a custom file

--log-style <log_style>#

If ‘pretty’, outputs with formatting and color. If ‘record’, outputs record-style without formatting. ‘auto’ defaults to ‘pretty’, and disables pretty logging if stdin is not a TTY.

Options:

auto | record | pretty

--log-color <log_color>#

Use color logging. Auto enables color logging if stdout is a TTY.

Options:

auto | false | true

-v, --verbose#

Arguments

CLUSTER_CONFIG_FILE#

Required argument

SCRIPT#

Required argument

SCRIPT_ARGS#

Optional argument(s)

ray attach#

Create or attach to a SSH session to a Ray cluster.

ray attach [OPTIONS] CLUSTER_CONFIG_FILE

Options

--start#

Start the cluster if needed.

--screen#

Run the command in screen.

--tmux#

Run the command in tmux.

-n, --cluster-name <cluster_name>#

Override the configured cluster name.

--no-config-cache#

Disable the local cluster config cache.

-N, --new#

Force creation of a new screen.

-p, --port-forward <port_forward>#

Port to forward. Use this multiple times to forward multiple ports.

--log-style <log_style>#

If ‘pretty’, outputs with formatting and color. If ‘record’, outputs record-style without formatting. ‘auto’ defaults to ‘pretty’, and disables pretty logging if stdin is not a TTY.

Options:

auto | record | pretty

--log-color <log_color>#

Use color logging. Auto enables color logging if stdout is a TTY.

Options:

auto | false | true

-v, --verbose#

Arguments

CLUSTER_CONFIG_FILE#

Required argument

ray get_head_ip#

Return the head node IP of a Ray cluster.

ray get_head_ip [OPTIONS] CLUSTER_CONFIG_FILE

Options

-n, --cluster-name <cluster_name>#

Override the configured cluster name.

Arguments

CLUSTER_CONFIG_FILE#

Required argument

ray monitor#

Tails the autoscaler logs of a Ray cluster.

ray monitor [OPTIONS] CLUSTER_CONFIG_FILE

Options

--lines <lines>#

Number of lines to tail.

-n, --cluster-name <cluster_name>#

Override the configured cluster name.

--log-style <log_style>#

If ‘pretty’, outputs with formatting and color. If ‘record’, outputs record-style without formatting. ‘auto’ defaults to ‘pretty’, and disables pretty logging if stdin is not a TTY.

Options:

auto | record | pretty

--log-color <log_color>#

Use color logging. Auto enables color logging if stdout is a TTY.

Options:

auto | false | true

-v, --verbose#

Arguments

CLUSTER_CONFIG_FILE#

Required argument