Cluster Management CLI#

This section contains commands for managing Ray clusters.

ray start#

Start Ray processes manually on the local machine.

ray start [OPTIONS]

Options

--node-ip-address <node_ip_address>#: the IP address of this node

--address <address>#: the address to use for Ray

--port <port>#: the port of the head ray process. If not provided, defaults to 6379; if port is set to 0, we will allocate an available port.

--object-manager-port <object_manager_port>#: the port to use for starting the object manager

--node-manager-port <node_manager_port>#: the port to use for starting the node manager

--gcs-server-port <gcs_server_port>#: Port number for the GCS server.

--min-worker-port <min_worker_port>#: the lowest port number that workers will bind on. If not set, random ports will be chosen.

--max-worker-port <max_worker_port>#: the highest port number that workers will bind on. If set, ‘–min-worker-port’ must also be set.

--worker-port-list <worker_port_list>#: a comma-separated list of open ports for workers to bind on. Overrides ‘–min-worker-port’ and ‘–max-worker-port’.

--ray-client-server-port <ray_client_server_port>#: the port number the ray client server binds on, default to 10001, or None if ray[client] is not installed.

--object-store-memory <object_store_memory>#: The amount of memory (in bytes) to start the object store with. By default, this is 30% (ray_constants.DEFAULT_OBJECT_STORE_MEMORY_PROPORTION) of available system memory capped by the shm size and 200G (ray_constants.DEFAULT_OBJECT_STORE_MAX_MEMORY_BYTES) but can be set higher.

--num-cpus <num_cpus>#: the number of CPUs on this node

--num-gpus <num_gpus>#: the number of GPUs on this node

--resources <resources>#: A JSON serialized dictionary mapping resource name to resource quantity.

--head#: provide this argument for the head node

--include-dashboard <include_dashboard>#: provide this argument to start the Ray dashboard GUI

--dashboard-host <dashboard_host>#: the host to bind the dashboard server to, either localhost (127.0.0.1) or 0.0.0.0 (available from all interfaces). By default, this is 127.0.0.1

--dashboard-port <dashboard_port>#: the port to bind the dashboard server to–defaults to 8265

--dashboard-agent-listen-port <dashboard_agent_listen_port>#: the port for dashboard agents to listen for http on.

--dashboard-agent-grpc-port <dashboard_agent_grpc_port>#: the port for dashboard agents to listen for grpc on.

--dashboard-grpc-port <dashboard_grpc_port>#: The port for the dashboard head to listen for grpc on.

--runtime-env-agent-port <runtime_env_agent_port>#: The port for the runtime enviroment agents to listen for http on.

--block#: provide this argument to block forever in this command

--plasma-directory <plasma_directory>#: object store directory for memory mapped files

--autoscaling-config <autoscaling_config>#: the file that contains the autoscaling config

--no-redirect-output#: do not redirect non-worker stdout and stderr to files

--plasma-store-socket-name <plasma_store_socket_name>#: manually specify the socket name of the plasma store

--raylet-socket-name <raylet_socket_name>#: manually specify the socket path of the raylet process

--temp-dir <temp_dir>#: manually specify the root temporary dir of the Ray process, only works when –head is specified

--storage <storage>#: the persistent storage URI for the cluster. Experimental.

--metrics-export-port <metrics_export_port>#: the port to use to expose Ray metrics through a Prometheus endpoint.

--ray-debugger-external#: Make the Ray debugger available externally to the node. This is only safe to activate if the node is behind a firewall.

--disable-usage-stats#: If True, the usage stats collection will be disabled.

--log-style <log_style>#

If ‘pretty’, outputs with formatting and color. If ‘record’, outputs record-style without formatting. ‘auto’ defaults to ‘pretty’, and disables pretty logging if stdin is not a TTY.

Options:: auto | record | pretty

--log-color <log_color>#

Use color logging. Auto enables color logging if stdout is a TTY.

Options:: auto | false | true

-v, --verbose#

ray stop#

Stop Ray processes manually on the local machine.

ray stop [OPTIONS]

Options

-f, --force#: If set, ray will send SIGKILL instead of SIGTERM.

-g, --grace-period <grace_period>#: The time in seconds ray waits for processes to be properly terminated. If processes are not terminated within the grace period, they are forcefully terminated after the grace period.

--log-style <log_style>#

If ‘pretty’, outputs with formatting and color. If ‘record’, outputs record-style without formatting. ‘auto’ defaults to ‘pretty’, and disables pretty logging if stdin is not a TTY.

Options:: auto | record | pretty

--log-color <log_color>#

Use color logging. Auto enables color logging if stdout is a TTY.

Options:: auto | false | true

-v, --verbose#

ray up#

Create or update a Ray cluster.

ray up [OPTIONS] CLUSTER_CONFIG_FILE

Options

--min-workers <min_workers>#: Override the configured min worker node count for the cluster.

--max-workers <max_workers>#: Override the configured max worker node count for the cluster.

--no-restart#: Whether to skip restarting Ray services during the update. This avoids interrupting running jobs.

--restart-only#: Whether to skip running setup commands and only restart Ray. This cannot be used with ‘no-restart’.

-y, --yes#: Don’t ask for confirmation.

-n, --cluster-name <cluster_name>#: Override the configured cluster name.

--no-config-cache#: Disable the local cluster config cache.

--redirect-command-output#: Whether to redirect command output to a file.

--use-login-shells, --use-normal-shells#: Ray uses login shells (bash –login -i) to run cluster commands by default. If your workflow is compatible with normal shells, this can be disabled for a better user experience.

--disable-usage-stats#: If True, the usage stats collection will be disabled.

--log-style <log_style>#

If ‘pretty’, outputs with formatting and color. If ‘record’, outputs record-style without formatting. ‘auto’ defaults to ‘pretty’, and disables pretty logging if stdin is not a TTY.

Options:: auto | record | pretty

--log-color <log_color>#

Use color logging. Auto enables color logging if stdout is a TTY.

Options:: auto | false | true

-v, --verbose#

Arguments

CLUSTER_CONFIG_FILE#: Required argument

ray down#

Tear down a Ray cluster.

ray down [OPTIONS] CLUSTER_CONFIG_FILE

Options

-y, --yes#: Don’t ask for confirmation.

--workers-only#: Only destroy the workers.

-n, --cluster-name <cluster_name>#: Override the configured cluster name.

--keep-min-workers#: Retain the minimal amount of workers specified in the config.

--log-style <log_style>#

If ‘pretty’, outputs with formatting and color. If ‘record’, outputs record-style without formatting. ‘auto’ defaults to ‘pretty’, and disables pretty logging if stdin is not a TTY.

Options:: auto | record | pretty

--log-color <log_color>#

Use color logging. Auto enables color logging if stdout is a TTY.

Options:: auto | false | true

-v, --verbose#

Arguments

CLUSTER_CONFIG_FILE#: Required argument

ray exec#

Execute a command via SSH on a Ray cluster.

ray exec [OPTIONS] CLUSTER_CONFIG_FILE CMD

Options

--run-env <run_env>#

Choose whether to execute this command in a container or directly on the cluster head. Only applies when docker is configured in the YAML.

Options:: auto | host | docker

--stop#: Stop the cluster after the command finishes running.

--start#: Start the cluster if needed.

--screen#: Run the command in a screen.

--tmux#: Run the command in tmux.

-n, --cluster-name <cluster_name>#: Override the configured cluster name.

--no-config-cache#: Disable the local cluster config cache.

-p, --port-forward <port_forward>#: Port to forward. Use this multiple times to forward multiple ports.

--disable-usage-stats#: If True, the usage stats collection will be disabled.

--log-style <log_style>#

If ‘pretty’, outputs with formatting and color. If ‘record’, outputs record-style without formatting. ‘auto’ defaults to ‘pretty’, and disables pretty logging if stdin is not a TTY.

Options:: auto | record | pretty

--log-color <log_color>#

Use color logging. Auto enables color logging if stdout is a TTY.

Options:: auto | false | true

-v, --verbose#

Arguments

CLUSTER_CONFIG_FILE#: Required argument

CMD#: Required argument

ray submit#

Uploads and runs a script on the specified cluster.

The script is automatically synced to the following location:

os.path.join(“~”, os.path.basename(script))

Example:: ray submit [CLUSTER.YAML] experiment.py – –smoke-test

ray submit [OPTIONS] CLUSTER_CONFIG_FILE SCRIPT [SCRIPT_ARGS]...

Options

--stop#: Stop the cluster after the command finishes running.

--start#: Start the cluster if needed.

--screen#: Run the command in a screen.

--tmux#: Run the command in tmux.

-n, --cluster-name <cluster_name>#: Override the configured cluster name.

--no-config-cache#: Disable the local cluster config cache.

-p, --port-forward <port_forward>#: Port to forward. Use this multiple times to forward multiple ports.

--args <args>#: (deprecated) Use ‘– –arg1 –arg2’ for script args.

--disable-usage-stats#: If True, the usage stats collection will be disabled.

--extra-screen-args <extra_screen_args>#: if screen is enabled, add the provided args to it. A useful example usage scenario is passing –extra-screen-args=’-Logfile /full/path/blah_log.txt’ as it redirects screen output also to a custom file

--log-style <log_style>#

If ‘pretty’, outputs with formatting and color. If ‘record’, outputs record-style without formatting. ‘auto’ defaults to ‘pretty’, and disables pretty logging if stdin is not a TTY.

Options:: auto | record | pretty

--log-color <log_color>#

Use color logging. Auto enables color logging if stdout is a TTY.

Options:: auto | false | true

-v, --verbose#

Arguments

CLUSTER_CONFIG_FILE#: Required argument

SCRIPT#: Required argument

SCRIPT_ARGS#: Optional argument(s)

ray attach#

Create or attach to a SSH session to a Ray cluster.

ray attach [OPTIONS] CLUSTER_CONFIG_FILE

Options

--start#: Start the cluster if needed.

--screen#: Run the command in screen.

--tmux#: Run the command in tmux.

-n, --cluster-name <cluster_name>#: Override the configured cluster name.

--no-config-cache#: Disable the local cluster config cache.

-N, --new#: Force creation of a new screen.

-p, --port-forward <port_forward>#: Port to forward. Use this multiple times to forward multiple ports.

--log-style <log_style>#

If ‘pretty’, outputs with formatting and color. If ‘record’, outputs record-style without formatting. ‘auto’ defaults to ‘pretty’, and disables pretty logging if stdin is not a TTY.

Options:: auto | record | pretty

--log-color <log_color>#

Use color logging. Auto enables color logging if stdout is a TTY.

Options:: auto | false | true

-v, --verbose#

Arguments

CLUSTER_CONFIG_FILE#: Required argument

ray get_head_ip#

Return the head node IP of a Ray cluster.

ray get_head_ip [OPTIONS] CLUSTER_CONFIG_FILE

Options

-n, --cluster-name <cluster_name>#: Override the configured cluster name.

Arguments

CLUSTER_CONFIG_FILE#: Required argument

ray monitor#

Tails the autoscaler logs of a Ray cluster.

ray monitor [OPTIONS] CLUSTER_CONFIG_FILE

Options

--lines <lines>#: Number of lines to tail.

-n, --cluster-name <cluster_name>#: Override the configured cluster name.

--log-style <log_style>#

If ‘pretty’, outputs with formatting and color. If ‘record’, outputs record-style without formatting. ‘auto’ defaults to ‘pretty’, and disables pretty logging if stdin is not a TTY.

Options:: auto | record | pretty

--log-color <log_color>#

Use color logging. Auto enables color logging if stdout is a TTY.

Options:: auto | false | true

-v, --verbose#

Arguments

CLUSTER_CONFIG_FILE#: Required argument