Ray Sandboxes#

Ray Sandboxes use gVisor to provide lightweight, kernel-isolated execution environments for running untrusted code and agent tool calls safely on Ray clusters.

Warning

Ray Sandboxes (ray.experimental.sandbox) is an alpha library. The API can change or disappear in any release before it graduates to stable.

Background#

The ability to sandbox model-generated code is critical for agentic reinforcement learning (RL) and large language model (LLM) agents. Executing untrusted code directly in Ray worker processes or host environments introduces security and stability risks. Ray Sandboxes solve this challenge by running lightweight, kernel-isolated sandboxes directly on Ray worker nodes using gVisor (runsc). Scale and manage sandbox environments with familiar Ray concepts and primitives.

What is gVisor?#

gVisor is an open-source application kernel written in Go that provides lightweight, defense-in-depth isolation for containers. Developed by Google, gVisor implements a substantial portion of the Linux system call interface in user space, acting as an isolation barrier between untrusted applications and the host operating system kernel.

Unlike standard container runtimes such as Docker or runc, where containers share the host Linux kernel directly, gVisor intercepts system calls made by containerized processes before they reach the host. gVisor is daemonless and runs as a non-privileged user, so you can deploy and manage it on top of existing container orchestrators such as Kubernetes.

Why gVisor?#

Untrusted code interacts with gVisor’s user-space kernel rather than the host Linux kernel, which shrinks the attack surface for host kernel vulnerabilities and container breakout exploits. gVisor also runs entirely in user space, without host root privileges, the Docker daemon, or nested virtualization hardware extensions, so it runs inside existing Kubernetes Ray worker Pods and cloud container environments.

The runtime cost is low next to full virtual machines (VMs) and MicroVMs, which boot a guest OS kernel and manage heavy disk images. A gVisor sandbox boots in tens of milliseconds, adds minimal memory overhead, and uses near-zero idle CPU, so Ray worker nodes can densely pack hundreds of concurrent sandboxes alongside standard Ray tasks and actors and sustain the high-frequency execution loops that RL rollouts and agent tool calls need.

Requirements#

Ray Sandboxes need the following on every Ray node that runs a sandbox:

  • Linux: x86_64 or arm64.

  • gVisor (runsc): Install the runsc binary on worker nodes and make it reachable from the system $PATH.

  • Ray: version 2.58.0 or later, which includes the ray.experimental.sandbox package.

To install runsc on a Linux worker node, see the gVisor installation guide.

Usage patterns and examples#

Create a basic sandbox and run a command#

Use sandbox.create() to start an isolated environment from any container image. The function returns a Ray ActorHandle representing the sandbox actor.

import ray
from ray.experimental import sandbox

ray.init()

# Create a sandbox with 1 CPU core and 512 MiB RAM
sb = sandbox.create(
    image="python:3.10-slim",
    cpu=1.0,
    memory="512Mi",
    workdir="/workspace",
    timeout_seconds=30.0,
)

# Execute untrusted Python code inside the sandbox
result = ray.get(
    sb.exec.remote("python3 -c 'import sys; print(\"Hello from sandboxed Python:\", sys.version)'")
)

print(f"Exit Code: {result.exit_code}")
print(f"Stdout: {result.stdout.strip()}")
print(f"Execution Duration: {result.duration_ms:.2f} ms")

# Clean up sandbox resources
ray.get(sb.delete.remote())

Read, write, upload, and download files#

Write source files directly into the sandbox, or upload local files from the host before execution. By default, the root filesystem is read-only and the configured workdir, such as /workspace, is the writable scratch space.

import textwrap
import ray
from ray.experimental import sandbox

ray.init()

sb = sandbox.create(
    image="python:3.10-slim",
    workdir="/workspace",
    memory="1Gi",
)

# 1. Write untrusted model-generated script into the sandbox
code = textwrap.dedent("""\
    def fibonacci(n):
        a, b = 0, 1
        for _ in range(n):
            a, b = b, a + b
        return a

    with open('/workspace/output.txt', 'w') as f:
        f.write(f"fib(30) = {fibonacci(30)}")
""")
ray.get(sb.write_file.remote("/workspace/solution.py", code))

# 2. Execute the script inside the sandbox
exec_res = ray.get(sb.exec.remote("python3 /workspace/solution.py"))
print("Execution returncode:", exec_res.exit_code)

# 3. Read generated output file back to the host
output_bytes = ray.get(sb.read_file.remote("/workspace/output.txt"))
print("Result:", output_bytes.decode("utf-8"))

# 4. Alternatively, use upload_file and download_file for host files
# ray.get(sb.upload_file.remote("local_input.json", "/workspace/input.json"))
# ray.get(sb.download_file.remote("/workspace/output.txt", "local_output.txt"))

ray.get(sb.delete.remote())

Schedule a Sandbox actor with custom resources#

Because Sandbox is a standard Ray actor, you can instantiate it directly with Ray actor scheduling options such as num_cpus, memory, and custom accelerator or placement constraints.

import ray
from ray.experimental.sandbox import Sandbox

ray.init()

# Instantiate Sandbox actor with Ray Core resource placement options
sandbox_actor = Sandbox.options(
    num_cpus=2.0,
    memory=2 * 1024 * 1024 * 1024,  # 2 GiB
).remote(
    image="python:3.10-slim",
    workdir="/workspace",
    ttl_seconds=600,  # Automatically terminate after 10 minutes
)

# Run command with a per-command execution timeout
result = ray.get(
    sandbox_actor.exec.remote(
        "python3 -c 'import os; print(\"Worker PID:\", os.getpid())'",
        timeout=5.0,  # 5 second execution timeout
    )
)

print(result.stdout)
ray.get(sandbox_actor.delete.remote())

Manage sandboxes inside custom actors with SandboxRuntime#

If you’re building custom RL environment actors or specialized rollout workers, embed SandboxRuntime directly inside your custom actors for fine-grained sandbox lifecycle control:

import ray
from ray.experimental.sandbox.runtime import SandboxRuntime

@ray.remote
class SandboxPool:
    def __init__(self, size: int = 3, image: str = "python:3.10-slim"):
        self.runtime = SandboxRuntime()
        self.sandboxes = [
            self.runtime.create(image=image, memory="512Mi")
            for _ in range(size)
        ]

    def run_command(self, index: int, command: str):
        return self.runtime.exec(self.sandboxes[index], command)

    def close(self):
        for sb_id in self.sandboxes:
            self.runtime.delete(sb_id)

# Deploy an actor managing a pool of local sandboxes
pool = SandboxPool.remote(size=3)
result = ray.get(pool.run_command.remote(0, "python3 -c 'print(\"Hello from pool!\")'"))
print(result.stdout)
ray.get(pool.close.remote())

Pass custom OCI configurations to gVisor#

For advanced workloads, you might need to configure low-level runtime options such as custom host mounts, Linux capabilities, or custom network and DNS settings. Use the _oci_spec_transform_fn parameter to inspect and modify the generated OCI runtime specification dictionary before Ray passes it to gVisor (runsc).

Note

_oci_spec_transform_fn is an experimental hook for advanced use cases. The Ray project is designing first-class configuration APIs for Ray Sandboxes, such as higher-level volume mount and capability abstractions, and this hook is likely to change once those land. To help shape them, open an issue describing your use case.

The _oci_spec_transform_fn callable receives the fully generated OCI specification dictionary. It can mutate the dictionary in place or return a modified one. Common use cases include the following:

  • Host mounts: Mount host directories, read-only datasets, or model weights into the sandbox container.

  • Networking and DNS: Set up DNS resolution by mounting host configuration files such as /etc/resolv.conf and /etc/hosts, or modify network namespace parameters when using network="sandbox".

  • Linux capabilities: Add or restrict Linux process capabilities such as CAP_NET_RAW or CAP_SYS_PTRACE under spec["process"]["capabilities"].

import ray
from ray.experimental import sandbox

ray.init()


def configure_oci_spec(spec: dict) -> dict:
    # Add a host bind mount (e.g., read-only dataset or cache directory)
    spec.setdefault("mounts", []).append(
        {
            "destination": "/mnt/dataset",
            "source": "/path/to/host/dataset",
            "type": "bind",
            "options": ["rbind", "ro"],
        }
    )

    # Configure networking and DNS resolution
    spec["mounts"].append(
        {
            "destination": "/etc/resolv.conf",
            "source": "/etc/resolv.conf",
            "type": "bind",
            "options": ["rbind", "ro"],
        }
    )

    # Grant additional Linux process capabilities
    caps = spec.setdefault("process", {}).setdefault("capabilities", {})
    for cap_set in ["bounding", "effective", "permitted"]:
        caps.setdefault(cap_set, [])
        if "CAP_NET_RAW" not in caps[cap_set]:
            caps[cap_set].append("CAP_NET_RAW")

    return spec


# Pass the transformation hook when creating the sandbox
sb = sandbox.create(
    image="python:3.10-slim",
    workdir="/workspace",
    network="sandbox",  # Enable network isolation namespace in gVisor
    _oci_spec_transform_fn=configure_oci_spec,
)

# Execute commands within the customized sandbox
result = ray.get(
    sb.exec.remote(
        "python3 -c 'print(\"Sandbox initialized with custom OCI configuration!\")'"
    )
)
print(result.stdout)

# Clean up resources
ray.get(sb.delete.remote())

Architecture#

The Ray Sandboxes subsystem has the following layers:

+-------------------------------------------------------------------+
|               Ray Application / RL Framework                      |
|           (e.g., veRL, SkyRL, RL Rollout Workers, Agents)         |
+-------------------------------------------------------------------+
                                  |
                                  v
+-------------------------------------------------------------------+
|                      ray.experimental.sandbox                     |
|           (High-level create() API & Sandbox Ray Actor)           |
+-------------------------------------------------------------------+
                                  |
                                  v
+-------------------------------------------------------------------+
|                  ray.experimental.sandbox.runtime                 |
|                      SandboxRuntime Interface                     |
+-------------------------------------------------------------------+
                                  |
                                  v
+-------------------------------------------------------------------+
|                 ray.experimental.sandbox.backend                  |
|               GVisorSandboxBackend (runsc OCI)                    |
+-------------------------------------------------------------------+
                                  |
                                  v
+-------------------------------------------------------------------+
|                        Ray Worker Node                            |
|   +-----------------------+       +-----------------------+       |
|   |  gVisor Sandbox 1     |       |  gVisor Sandbox 2     |       |
|   | (python:3.10-slim)    |       | (busybox:latest)      |       |
|   |   CPU: 0.5, Mem: 256M |       |   CPU: 1.0, Mem: 512M |       |
|   +-----------------------+       +-----------------------+       |
+-------------------------------------------------------------------+

Core components#

  • High-level helper (create()): Spawns a Ray actor that encapsulates the sandbox lifecycle and returns an ActorHandle.

  • Sandbox actor (Sandbox): A Ray actor that serves as a proxy to forward command execution and file I/O to the isolated sandbox instance while managing the scheduling and lifecycle of the sandbox.

  • Sandbox runtime (SandboxRuntime): A low-level abstraction that manages the lifecycle of local sandboxes, image pulling and caching, and interactions with the execution backend.

  • gVisor backend (ray.experimental.sandbox.backend.GVisorSandboxBackend): Executes commands and isolates processes through gVisor’s OCI runtime (runsc).

  • Image manager (ray.experimental.sandbox.image_manager.ImageManager): Automatically pulls container images from sources such as Docker Hub, GHCR, or local tar archives, extracts root filesystems into /tmp/ray/sandbox/images, and builds OCI config.json runtime specifications.

Security and isolation model#

Ray Sandboxes implement multi-layered defense-in-depth isolation:

  • System call interception: gVisor’s Sentry application kernel intercepts system calls in user space, isolating untrusted code from the host Linux kernel.

  • Read-only root filesystem: Ray mounts base container filesystems read-only (readonly=True) with an isolated copy-on-write overlay directory per sandbox.

  • Restricted working directory: Only the explicit workdir, such as /workspace, is mounted read-write for application artifacts.

  • Network containment: By default, network="none" disables all outbound network interfaces, which prevents untrusted code from making external API calls or scanning the internal cluster network.

  • Resource quotas: cgroups enforce CPU quotas and memory limits, which prevents CPU starvation and out-of-memory (OOM) conditions from affecting other Ray actors.

API reference#

For detailed signatures, parameters, and return types, see Sandbox API.

Troubleshooting#

  • runsc not found in $PATH: Verify that gVisor’s runsc binary is installed on all Ray worker nodes and sits in a directory on the system $PATH, such as /usr/local/bin/runsc.

  • cgroup or permission errors: In containerized environments such as Kubernetes without root permissions, keep the default rootless=True. Where cgroups are restricted, set RAY_SANDBOX_IGNORE_CGROUPS=1.

  • Image pull failures: Verify that the node can reach the container registry, such as Docker Hub or GHCR, or pre-populate the image cache directory at /tmp/ray/sandbox/images.

Next steps#