Configure Ray clusters to use token authentication#

This guide demonstrates how to enable Ray token authentication with KubeRay.

Prerequisites#

  • A Kubernetes cluster. This guide uses GKE, but the concepts apply to other Kubernetes distributions.

  • kubectl installed and configured to interact with your cluster.

  • gcloud CLI installed and configured, if using GKE.

  • Helm installed.

  • Ray 2.52.0 or newer.

Create or use an existing GKE Cluster#

If you don’t have a Kubernetes cluster, create one using the following command, or adapt it for your cloud provider:

gcloud container clusters create kuberay-cluster \
    --num-nodes=2 --zone=us-west1-b --machine-type e2-standard-4

Install the KubeRay Operator#

Follow Deploy a KubeRay operator to install the latest stable KubeRay operator from the Helm repository.

Deploy a Ray cluster with token authentication#

If you are using KubeRay v1.5.1 or newer, you can use the authOptions API in RayCluster to enable token authentication:

kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/refs/heads/master/ray-operator/config/samples/ray-cluster.auth.yaml

When enabled, the KubeRay operator will:

  • Create a Kubernetes Secret containing a randomly generated token.

  • Automatically set the RAY_AUTH_TOKEN and RAY_AUTH_MODE environment variables on all Ray containers.

If you are using a KubeRay version older than v1.5.1, you can enable token authentication by creating a Kubernetes Secret containing your token and configuring the RAY_AUTH_MODE and RAY_AUTH_TOKEN environment variables.

kubectl create secret generic ray-cluster-with-auth --from-literal=auth_token=$(openssl rand -base64 32)
kubectl apply -f https://raw.githubusercontent.com/ray-project/kuberay/refs/heads/master/ray-operator/config/samples/ray-cluster.auth-manual.yaml

Verify initial unauthenticated access#

Attempt to submit a Ray job to the cluster to verify that authentication is required. You should receive a 401 Unauthorized error:

kubectl port-forward svc/ray-cluster-with-auth-head-svc 8265:8265 &
ray job submit --address http://localhost:8265  -- python -c "import ray; ray.init(); print(ray.cluster_resources())"

You should see an error similar to this:

RuntimeError: Authentication required: Unauthorized: Missing authentication token

The Ray cluster requires authentication, but no token was provided.

Please provide an authentication token using one of these methods:
  1. Set the `RAY_AUTH_TOKEN` environment variable.
  2. Set the `RAY_AUTH_TOKEN_PATH` environment variable (pointing to a file containing the token).
  3. Create a token file at the default location: `~/.ray/auth_token`.

This error confirms that the Ray cluster requires authentication.

Accessing your Ray cluster with the Ray CLI#

To access your Ray cluster using the Ray CLI, you need to configure the following environment variables:

  • RAY_AUTH_MODE: this configures the Ray CLI to set the necessary authorization headers for token authentication

  • RAY_AUTH_TOKEN: this contains the token that will be used for authentication.

  • RAY_AUTH_TOKEN_PATH: if RAY_AUTH_TOKEN is not set, the Ray CLI will instead read the token from this path (defaults to ~/.ray/auth_token).

Submit a job with an authenticated Ray CLI:

export RAY_AUTH_MODE=token
export RAY_AUTH_TOKEN=$(kubectl get secrets ray-cluster-with-auth --template={{.data.auth_token}} | base64 -d)
ray job submit --address http://localhost:8265  -- python -c "import ray; ray.init(); print(ray.cluster_resources())"

The job should now succeed and you should see output similar to this:

Job submission server address: http://localhost:8265

-------------------------------------------------------
Job 'raysubmit_...' submitted successfully
-------------------------------------------------------

Next steps
  Query the logs of the job:
    ray job logs raysubmit_n2fq2Ui7cbh3p2Js
  Query the status of the job:
    ray job status raysubmit_n2fq2Ui7cbh3p2Js
  Request the job to be stopped:
    ray job stop raysubmit_n2fq2Ui7cbh3p2Js

Tailing logs until the job exits (disable with --no-wait):

...
{'node:10.112.0.52': 1.0, 'memory': ..., 'node:__internal_head__': 1.0, 'object_store_memory': ..., 'CPU': 4.0, 'node:10.112.1.49': 1.0, 'node:10.112.2.36': 1.0}

------------------------------------------
Job 'raysubmit_...' succeeded
------------------------------------------

Viewing the Ray dashboard (optional)#

To view the Ray dashboard from your browser, first port forward to from your local machine to the cluster:

kubectl port-forward svc/ray-cluster-with-auth-head-svc 8265:8265 &

Then open localhost:8265 in your browser. You will be prompted to provide the auth token for the cluster, which can be retrieved with:

kubectl get secrets ray-cluster-with-auth --template={{.data.auth_token}} | base64 -d