Ray token authentication#

Enable token authentication in Ray to secure cluster access and prevent unauthorized use. This guide explains how authentication works and how to set it up for different deployment scenarios.

Note

Token authentication is available in Ray 2.52.0 or later.

How Ray token authentication works#

To enable token authentication, set the environment variable RAY_AUTH_MODE=token before starting your Ray cluster. When you start a Ray cluster with authentication enabled, all external Ray APIs and internal communications are authenticated using the token as a shared secret.

The process for generating and configuring authentication tokens differs depending on how you launch your Ray cluster. When you start a local instance of Ray using ray.init() with token authentication enabled, Ray automatically generates and uses a token.

Other cluster launching methods require that you generate a token before starting the cluster. You can ray get-auth-token [--generate] to retrieve your existing token or generate a new one.

Note

Authentication is disabled by default in Ray 2.52.0. Ray plans to enable token authentication by default in a future release. We recommend enabling token authentication to protect your cluster from unauthorized access.

What token does Ray use?#

You can configure authentication tokens using environment variables or the default path. We recommend using the default path when possible to reduce the chances of committing the token to version control.

Ray checks for tokens in the following order, highest priority first:

RAY_AUTH_TOKEN environment variable.
RAY_AUTH_TOKEN_PATH environment variable, which provides a path to a token file.
The default location, ~/.ray/auth_token.

When managing multiple tokens, we recommend storing them in local files and using the RAY_AUTH_TOKEN_PATH environment variable rather than setting the RAY_AUTH_TOKEN value directly to avoid exposing the token to other code that reads environment variables.

Security considerations#

Ray transmits the authentication token as an HTTP header, which is transmitted in plaintext when using insecure http connections. We recommend enabling some form of encryption whenever exposing a Ray cluster over the network. Consider the following:

Local development: Traffic doesn’t leave your machine, so no additional security is needed.
SSH tunneling: Use SSH tunneling/port forwarding via the ray dashboard command or kubectl port-forward.
TLS termination: Deploy a TLS proxy in front of your Ray cluster.
VPN/Overlay networks: Use network-level encryption for all traffic into and within the cluster.

Warning

Don’t expose Ray clusters directly to the internet without encryption. Tokens alone don’t protect against network eavesdropping.

Tokens have the following properties:

Ray stores tokens by default in plaintext at ~/.ray/auth_token.
Use file permissions to keep token files secure, especially in shared environments.
Don’t commit tokens to version control.
Tokens don’t expire. Your local token remains valid unless you delete and regenerate the token. Ray clusters use the same token for the lifetime of the cluster.

Configure token authentication for local development#

To enable authentication on your local machine for development, set the RAY_AUTH_MODE=token environment variable in your shell or IDE. You can persist this configuration in your .bashrc file or similar.

Local development with ray.init()#

When you run a script that starts a new local Ray cluster with ray.init() (that is, ray.init() without an address, so it starts a fresh cluster), Ray enables token authentication by default and handles the token automatically:

If no token exists at ~/.ray/auth_token, Ray generates one, saves it there, and logs a message confirming the new token.
If a token already exists at ~/.ray/auth_token, Ray reuses it.

This happens even when you haven’t set RAY_AUTH_MODE=token. To start a new local cluster without authentication, set RAY_AUTH_MODE=disabled.

For example:

$ python -c "import ray;ray.init()"

On the first run, this command (or any script that starts Ray) logs a line like:

Generated new authentication token and saved to ~/.ray/auth_token

Connecting to an existing cluster with ray.init(address=...) doesn’t generate a token. As with any client, you must have the cluster’s token configured when the cluster has token authentication enabled, as described in What token does Ray use?.

Local development with ray start#

When you start a local head cluster with ray start --head, Ray enables token authentication automatically if a token is already available from any of the token sources, even when you haven’t set RAY_AUTH_MODE=token. Ray considers a token available when any of the following is set: the RAY_AUTH_TOKEN environment variable, a valid file at the path in RAY_AUTH_TOKEN_PATH, or the default token file ~/.ray/auth_token. Generate a token once with ray get-auth-token --generate and Ray uses it every time you run ray start --head on that machine. To opt out when a token is available, set RAY_AUTH_MODE=disabled.

If no token is available from any source, behavior depends on RAY_AUTH_MODE:

If you didn’t set RAY_AUTH_MODE=token, authentication stays disabled and ray start --head logs a warning that the cluster is unauthenticated.
If you set RAY_AUTH_MODE=token, ray start raises an error with instructions to generate a token.

ray start --head logs a warning whenever it starts a cluster without token authentication, whether because no token is available or because you set RAY_AUTH_MODE=disabled.

If you set RAY_AUTH_MODE=token before a token exists, ray start --head raises an error instead of starting:

$ export RAY_AUTH_MODE=token
$ ray start --head
...
ray.exceptions.AuthenticationError: Token authentication is enabled but no authentication token was found. Ensure that the token for the cluster is available in a local file (e.g., ~/.ray/auth_token or via RAY_AUTH_TOKEN_PATH) or as the `RAY_AUTH_TOKEN` environment variable. To generate a token for local development, use `ray get-auth-token --generate` For remote clusters, ensure that the token is propagated to all nodes of the cluster when token authentication is enabled. For more information, see: https://docs.ray.io/en/latest/ray-security/token-auth.html

Configure token authentication for remote clusters#

When working with remote clusters you must ensure that all nodes in the remote cluster have token authentication enabled and access to the same token. Any clients that interact with the remote cluster, including your local machine, must also have the token configured. The following sections provide an overview of configuring this using the Ray cluster launcher and self-managed clusters.

For instructions on configuring token authentication with KubeRay, see Token authentication with KubeRay.

Note

If you’re using a hosted version of Ray, contact your customer support for authentication questions.

Anyscale manages authentication automatically for users.

Ray clusters on remote virtual machines#

This section provides instructions for using ray up to launch a remote cluster on virtual machines with token authentication enabled.

Step 1: Generate a token#

You must generate a token on your local machine. Run the following command to generate a token:

ray get-auth-token --generate

This command generates a token on your local machine at the path ~/.ray/auth_token and outputs it to the terminal.

Step 2: Specify token authentication values in your cluster configuration YAML#

To enable and configure token authentication, you add the following setting to your cluster configuration YAML:

Use file_mounts to mount your locally generated token file to all nodes in the cluster.
Use initialization_commands to set the environment variable RAY_AUTH_MODE=token for all virtual machines in the cluster.

The following is an example cluster configuration YAML that includes the file_mounts and initialization_commands settings required to enable token authentication and use the token generated on your local machine at the path ~/.ray/auth_token:

cluster_name: my-cluster-name
provider:
    type: aws
    region: us-west-2
max_workers: 2
available_node_types:
    ray.head.default:
        resources: {}
        node_config:
            InstanceType: m5.large
    ray.worker.default:
        min_workers: 2
        max_workers: 2
        resources: {}
        node_config:
            InstanceType: m5.large

# Mount a locally generated token file to all nodes in the Ray cluster.
file_mounts: {
    "/home/ubuntu/.ray/auth_token": "~/.ray/auth_token",
}

# Set the RAY_AUTH_MODE environment variable for all shell sessions on the cluster.
initialization_commands:
  - echo "export RAY_AUTH_MODE=token" >> ~/.bashrc

Step 3: Launch the Ray cluster#

Run the following command to launch a Ray cluster using your cluster configuration YAML:

ray up cluster.yaml

Step 4: Configure the Ray dashboard and port forwarding#

Connecting to the Ray dashboard configures secure SSH port forwarding between your local machine and the Ray cluster. Complete this step even if you don’t plan to use the dashboard for monitoring.

Run the following command to set up port forwarding for the Ray dashboard port (8265 by default):

ray dashboard cluster.yaml

Upon opening the dashboard, a prompt displays requesting your authentication token. To display the token in plaintext, you can run the following on your local machine:

export RAY_AUTH_MODE=token
ray get-auth-token

Paste the token in the prompt and click Submit. The token gets stored as a cookie for a maximum of 30 days. When you open the dashboard for a cluster that uses a different token, a prompt appears to enter the token for that cluster.

Step 5: Submit a Ray job#

You can submit a Ray job with token authentication using secure SSH port forwarding:

export RAY_AUTH_MODE=token
ray job submit --working-dir . -- python script.py

Self-managed clusters#

If you have a custom deployment where you run ray start on multiple nodes, you can use token authentication by generating a token and distributing it to all nodes in the cluster, as shown in the following steps.

Step 1: Generate a token#

Generate a token on a single machine using the following command:

ray get-auth-token --generate

Note

Any machine that needs to interact with the cluster must have the token used to configure authentication.

Step 2: Copy the token to all nodes#

Copy the same token to each node in your Ray cluster. For example, use scp to copy the token:

scp ~/.ray/auth_token user@node1:~/.ray/auth_token;
scp ~/.ray/auth_token user@node2:~/.ray/auth_token;

Step 3: Start Ray with token authentication#

You must set the environment variable RAY_AUTH_MODE=token on each node before running ray start, as in the following example:

ssh user@node1 "RAY_AUTH_MODE=token ray start --head";
ssh user@node2 "RAY_AUTH_MODE=token ray start --address=node1:6379";

Troubleshooting token authentication issues#

You might encounter the following problems with token authentication.

Token authentication isn’t enabled#

Make sure you’ve set the RAY_AUTH_MODE=token environment variable in the environment where you’re launching Ray and in any shell where you are using a client to connect to Ray.

Authentication token not found#

If running locally, run ray get-auth-token --generate to create a token on your local machine. If running a remote cluster, make sure you’ve followed instructions to copy your token into the cluster.

Invalid authentication token#

Any client that tries to interact with a Ray cluster must have the same token as the Ray cluster.

If the token on your local machine doesn’t match the token in a Ray cluster, you can use the RAY_AUTH_TOKEN_PATH or RAY_AUTH_TOKEN environment variable to configure a token for interacting with that cluster. You must work with the creator of the cluster to get the token.

Note

It’s possible to stop and then restart a cluster using a different token. All clients connecting to the cluster must have the updated token to connect successfully.

Next steps#

See overall security guidelines.
Read about KubeRay authentication for Kubernetes-specific configuration.