Head node memory management#

When you run a Ray cluster for an extended period, the head node’s memory usage can increase steadily. This growth can lead to out-of-memory (OOM) errors that can make the entire cluster unusable. This page explains what causes head node memory growth and how to mitigate it.

Why head node memory grows #

The Ray dashboard provides a web interface for cluster monitoring and debugging. See Ray dashboard. The dashboard adds to head node memory usage in the following ways:

The dashboard caches cluster events in memory for display and debugging. The RAY_DASHBOARD_MAX_EVENTS_TO_CACHE environment variable controls the cache size. For implementation details, see the event caching code.
The dashboard processes and stores logs and metadata from jobs and workers. This data accumulates in long-running clusters.

Mitigation strategies #

Avoid scheduling on the head node #

Avoid running tasks or actors on the head node, because it hosts critical system components. Keeping work off the head node helps reduce contention and memory pressure.

For head node best practices, see Configuring the head node.

Disable the dashboard #

If you don’t need the dashboard, disable it to remove event caching and the related memory overhead. Disabling the dashboard reduces observability into the system, so Ray doesn’t recommend disabling it on production clusters.

To disable the dashboard with the Python API, run the following:

import ray
ray.init(include_dashboard=False)

To disable the dashboard with the CLI, run the following:

ray start --head --include-dashboard=False

On Kubernetes, set spec.headGroupSpec.rayStartParams.include-dashboard to "false" in your RayCluster configuration.

Warning

Disabling the dashboard prevents KubeRay’s RayJob and RayService features from working properly.

Kubernetes configuration #

Head pod memory settings #

When you deploy on Kubernetes, configure memory requests and limits for the head pod.

Important

Set resource requests equal to their limits. KubeRay uses the container’s resource limits to configure Ray’s logical resource capacities.

The following example sets the head pod’s requests equal to its limits:

headGroupSpec:
  template:
    spec:
      containers:
      - name: ray-head
        resources:
          requests:
            memory: "8Gi"
            cpu: "4"
          limits:
            memory: "8Gi"
            cpu: "4"

Recommended head node specifications #

For large clusters, start with the following head node specification:

CPU: 16 cores
Memory: 64 GB

Actual requirements depend on your workload and cluster size.

To prevent Ray from scheduling tasks on the head node, set num-cpus: "0" in rayStartParams.

Best practices #

Follow these practices to keep head node memory under control:

Avoid scheduling on the head node to reduce contention and memory pressure.
Scale vertically and use a larger head node before you adjust internal settings.
Set Kubernetes resource limits, and set memory and GPU requests to match them.

Note

Disabling the dashboard severely limits observability, so Ray doesn’t recommend it for production. If you choose to disable it, see Disable the dashboard.

Troubleshooting #

If your head node runs into OOM errors, do the following:

Check memory usage with ray memory. See Debugging using ray memory.
Consider increasing the head node’s memory allocation.

For more information on OOM prevention, see Out-of-memory prevention.