Skip to main content
Ctrl+K
Try Ray with $100 credit — Start now
Ray 2.59 enables token authentication by default for local clusters, and Ray 2.61 extends it to all clusters. Read the rollout details

Site Navigation

  • Get Started

  • Use Cases

  • Example Gallery

  • Library

    • Ray CoreScale general Python applications

    • Ray DataScale data ingest and preprocessing

    • Ray TrainScale machine learning training

    • Ray TuneScale hyperparameter tuning

    • Ray ServeScale model serving

    • Ray RLlibScale reinforcement learning

  • APIs

  • Resources

    • Discussion ForumGet your Ray questions answered

    • TrainingHands-on learning

    • BlogUpdates, best practices, user-stories

    • EventsWebinars, meetups, office hours

    • Success StoriesReal-world workload examples

    • EcosystemLibraries integrated with Ray

    • CommunityConnect with us

Try Managed Ray

Site Navigation

  • Get Started

  • Use Cases

  • Example Gallery

  • Library

    • Ray CoreScale general Python applications

    • Ray DataScale data ingest and preprocessing

    • Ray TrainScale machine learning training

    • Ray TuneScale hyperparameter tuning

    • Ray ServeScale model serving

    • Ray RLlibScale reinforcement learning

  • APIs

  • Resources

    • Discussion ForumGet your Ray questions answered

    • TrainingHands-on learning

    • BlogUpdates, best practices, user-stories

    • EventsWebinars, meetups, office hours

    • Success StoriesReal-world workload examples

    • EcosystemLibraries integrated with Ray

    • CommunityConnect with us

Try Managed Ray
  • Overview
  • Getting Started
  • Installation
  • Use Cases
    • Ray for ML Infrastructure
  • Examples
    • Multi-modal AI pipeline
      • Batch inference
      • Distributed training
      • Online serving
    • LLM training and inference
    • Audio batch inference
    • Distributed XGBoost pipeline
      • Distributed training of an XGBoost model
      • Model validation using offline batch inference
      • Scalable online XGBoost inference with Ray Serve
    • Time-series forecasting
      • Distributed training of a DLinear time-series model
      • DLinear model validation using offline batch inference
      • Online serving for DLinear model using Ray Serve
    • Scalable video processing
      • Fine-tuning a face mask detection model with Faster R-CNN
      • Object detection batch inference on test dataset and metrics calculation
      • Video processing with object detection using batch inference
      • Host an object detection model as a service
    • Distributed RAG pipeline
      • Build a Regular RAG Document Ingestion Pipeline (No Ray required)
      • Scalable RAG Data Ingestion and Pagination with Ray Data
      • Deploy LLM with Ray Serve LLM
      • Build Basic RAG App
      • Improve RAG with Prompt Engineering
      • Evaluate RAG with Online Inference
      • Evaluate RAG using Batch Inference with Ray Data LLM
    • Deploy MCP servers
      • Deploying a custom MCP in Streamable HTTP mode with Ray Serve
      • Deploy an MCP Gateway with existing Ray Serve apps
      • Deploying an MCP STDIO Server as a scalable HTTP service with Ray Serve
      • Deploying multiple MCP services with Ray Serve
      • Build a Docker image for an MCP server
    • Build a tool-using agent
    • Build a multi-agent system with the A2A protocol
  • KubeRay
    • Getting Started with KubeRay
      • KubeRay Operator Installation
      • RayCluster Quickstart
      • RayJob Quickstart
      • RayService Quickstart
      • RayCronJob Quickstart
    • User Guides
      • Deploy Ray Serve Apps
      • RayService worker Pods aren’t ready
      • RayService high availability
      • Enable High Throughput on Ray Serve with KubeRay
      • RayService Zero-Downtime Incremental Upgrades
      • KubeRay Observability
      • KubeRay upgrade guide
      • Managed Kubernetes services
        • Start Amazon EKS Cluster with GPUs for KubeRay
        • Start Google Cloud GKE Cluster with GPUs for KubeRay
        • Start a Google Cloud GKE cluster with TPUs for KubeRay
        • Start Azure AKS Cluster with GPUs for KubeRay
        • Start an Aliyun ACK cluster with GPUs for KubeRay
      • Best Practices for Storage and Dependencies
      • RayCluster Configuration
      • Scheduling on Kubernetes
      • KubeRay Autoscaling
      • KubeRay In-Place Pod Resizing (IPPR)
      • KubeRay label-based scheduling
      • GCS fault tolerance in KubeRay
      • Tuning Redis for a Persistent Fault Tolerant GCS
      • GCS fault tolerance with embedded RocksDB (alpha)
      • Configuring KubeRay to use Google Cloud Storage Buckets in GKE
      • Persist KubeRay custom resource logs
      • Persist KubeRay Operator Logs
      • Using GPUs
      • Use TPUs with KubeRay
      • Specify container commands for Ray head/worker Pods
      • Helm Chart RBAC
      • TLS Authentication
      • Configure RayCluster to use NetworkPolicies
      • Configuring mTLS for RayClusters
      • (Advanced) Understanding the Ray Autoscaler in the Context of Kubernetes
      • Use kubectl plugin (beta)
      • Configure Ray clusters to use token authentication
      • Configure Ray clusters to use Kubernetes RBAC authentication
      • Reducing image pull latency on Kubernetes
      • Using uv for Python package management in KubeRay
      • Use KubeRay dashboard (experimental)
      • Resource isolation with writable cgroups on Google Kubernetes Engine (GKE)
      • Ray History Server with KubeRay
      • Enable Ray platform events on Kubernetes
      • RayJob SidecarSubmitterRestart
    • Examples
      • Train a PyTorch model on Fashion MNIST with CPUs on Kubernetes
      • Serve a StableDiffusion text-to-image model on Kubernetes
      • Serve a Stable Diffusion model on GKE with TPUs
      • Serve a MobileNet image classifier on Kubernetes
      • Serve a text summarizer on Kubernetes
      • RayJob Batch Inference Example
      • Priority Scheduling with RayJob and Kueue
      • Gang Scheduling with RayJob and Kueue
      • Distributed checkpointing with KubeRay and GCSFuse
      • Serve a Large Language Model using Ray Serve LLM on Kubernetes
      • Serve Deepseek R1 using Ray Serve LLM
      • Reinforcement Learning with Human Feedback (RLHF) for LLMs with verl on KubeRay
      • Deploying Ray Clusters via ArgoCD
      • Sandboxed code execution with Ray and Agent Sandbox
      • Deploy Ray sandboxes with KubeRay
    • KubeRay Ecosystem
      • Ingress
      • KubeRay metrics references
      • Using Prometheus and Grafana
      • Profiling with py-spy
      • Gang scheduling, queue priority, and GPU sharing for RayClusters using KAI Scheduler
      • KubeRay integration with Volcano
      • KubeRay integration with Apache YuniKorn
      • Gang scheduling, Priority scheduling, and Autoscaling for KubeRay CRDs with Kueue
      • mTLS and L7 observability with Istio
      • KubeRay integration with scheduler plugins
    • KubeRay Benchmarks
      • KubeRay memory and scalability benchmark
    • KubeRay Troubleshooting
      • Troubleshooting guide
      • RayService troubleshooting
    • API Reference
      • KubeRay CRD API reference
  • Ray Core
    • Key Concepts
    • User Guides
      • Tasks
        • Nested remote functions
      • Actors
        • Named actors
        • Terminating actors
        • asyncio and concurrency for actors
        • Limiting concurrency per method with concurrency groups
        • Utility classes
        • Out-of-band communication
        • Actor task execution order
      • Objects
        • Serialization
        • Object spilling
      • Environment dependencies
      • Scheduling
        • Use labels to control scheduling
        • Resources
        • Accelerator support
        • Placement groups
        • Memory management
        • Out-of-memory prevention
      • Fault tolerance
        • Task fault tolerance
        • Actor fault tolerance
        • Object fault tolerance
        • Node fault tolerance
        • GCS fault tolerance
      • Design patterns and anti-patterns
        • Pattern: Using nested tasks to achieve nested parallelism
        • Pattern: Using generators to reduce heap memory usage
        • Pattern: Using ray.wait to limit the number of pending tasks
        • Pattern: Using resources to limit the number of concurrently running tasks
        • Pattern: Using asyncio to run actor methods concurrently
        • Pattern: Using an actor to synchronize other tasks and actors
        • Pattern: Using a supervisor actor to manage a tree of actors
        • Pattern: Using pipelining to increase throughput
        • Anti-pattern: Returning ray.put() ObjectRefs from a task harms performance and fault tolerance
        • Anti-pattern: Calling ray.get on task arguments harms performance
        • Anti-pattern: Calling ray.get in a loop harms parallelism
        • Anti-pattern: Calling ray.get unnecessarily harms performance
        • Anti-pattern: Processing results in submission order using ray.get increases runtime
        • Anti-pattern: Fetching too many objects at once with ray.get causes failure
        • Anti-pattern: Over-parallelizing with too fine-grained tasks harms speedup
        • Anti-pattern: Redefining the same remote function or class harms performance
        • Anti-pattern: Passing the same large argument by value repeatedly harms performance
        • Anti-pattern: Closure capturing large objects harms performance
        • Anti-pattern: Using global variables to share state between tasks and actors
        • Anti-pattern: Serializing ray.ObjectRef out of band
        • Anti-pattern: Forking new processes in application code
      • Ray Direct Transport (RDT)
        • Implementing a custom tensor transport (Advanced)
      • Ray Compiled Graph (beta)
        • Quickstart
        • Profiling
        • Experimental: Overlapping communication and computation
        • Troubleshooting
        • Compiled Graph API
      • Resource isolation with cgroup v2
      • Ray Sandboxes
      • Advanced topics
        • Tips for first-time users
        • Type hints in Ray
        • Starting Ray
        • Ray generators
        • Use namespaces
        • Cross-language programming
        • Use Ray with Jupyter Notebook and JupyterLab
        • Lazy computation graphs with the Ray DAG API
        • Miscellaneous topics
        • Authenticate remote URIs in runtime_env
        • Lifetimes of user-spawned processes
        • Head node memory management
    • Examples
      • Batch Prediction with Ray Core
      • A Gentle Introduction to Ray Core by Example
      • Using Ray for highly parallelizable tasks
      • A Simple MapReduce Example with Ray Core
      • Monte Carlo estimation of π
      • Simple Parallel Model Selection
      • Parameter Server
      • Learning to Play Pong
      • Speed up your web crawler by parallelizing it with Ray
    • Internals
      • Task lifecycle
      • Streaming generator
      • Autoscaler v2
      • RPC fault tolerance
      • Token authentication
      • Metric exporter infrastructure
      • Ray event exporter infrastructure
      • Port service discovery
      • Object spilling
  • Ray Data
    • Ray Data quickstart
    • Key concepts
    • User guides
      • Loading data
      • Inspecting data
      • Transforming Data
      • Aggregating data
      • Iterating over data
      • Joining data
      • Shuffling data
      • Weighted dataset mixing
      • Saving data
      • Working with images
      • Working with text
      • Working with tensors and NumPy
      • Working with PyTorch
      • Working with LLMs
      • How to avoid out-of-memory errors (OOMs)
      • Monitoring your workload
      • Execution configurations
      • Run multiple Datasets in one cluster
      • End-to-end: Offline Batch Inference
      • Advanced: Performance tips and tuning
      • Advanced: Scaling out expensive collate functions
      • Advanced: Read and write custom file types
    • Examples
    • Contributing to Ray Data
      • Contributing Guide
      • How to write tests
    • Comparing Ray Data to other systems
    • Ray Data benchmarks
    • Ray Data internals
  • Ray Train
    • Overview
    • PyTorch guide
    • PyTorch Lightning guide
    • Hugging Face Transformers guide
    • XGBoost guide
    • JAX guide
    • More frameworks
      • Hugging Face Accelerate guide
      • DeepSpeed guide
      • TensorFlow and Keras guide
      • LightGBM guide
      • Horovod guide
    • User guides
      • Data loading and preprocessing
      • Configure scale and accelerators
      • Configure persistent storage
      • Monitor and log metrics
      • Save and load checkpoints
      • Validate checkpoints asynchronously
      • Experiment tracking
      • Inspect training results
      • Handle failures and node preemption
      • Elastic training
      • Detect NCCL hangs
      • Ray Train metrics
      • Local mode
      • Reproducibility
      • Hyperparameter optimization
    • Tutorials
      • Introduction to Ray Train workloads
      • Computer vision pattern
      • Tabular workload pattern
      • Time series workload pattern
      • Generative computer vision pattern
      • Diffusion policy pattern
      • Recommendation system pattern
    • Examples
    • Benchmarks
  • Ray Tune
    • Getting started
    • Key concepts
    • User guides
      • Running basic experiments
      • Logging and outputs in Tune
      • Setting trial resources
      • Using search spaces
      • How to define stopping criteria for a Ray Tune experiment
      • How to save and load trial checkpoints
      • Configure persistent storage in Ray Tune
      • How to enable fault tolerance in Ray Tune
      • Using callbacks and metrics
      • Get data in and out of Tune
      • Analyzing Tune Experiment Results
      • A Guide to Population Based Training with Tune
        • Visualizing and Understanding PBT
      • Deploying Tune in the cloud
      • Tune architecture
      • Scalability benchmarks
    • Ray Tune Examples
      • PyTorch Example
      • PyTorch Lightning Example
      • XGBoost Example
      • LightGBM Example
      • Hugging Face Transformers Example
      • Ray RLlib Example
      • Keras Example
      • PyTorch with ASHA
      • Weights & Biases Example
      • MLflow Example
      • Aim Example
      • Comet Example
      • Ax Example
      • HyperOpt Example
      • Bayesopt Example
      • BOHB Example
      • Nevergrad Example
      • Optuna Example
    • Ray Tune FAQ
  • Ray Serve
    • Getting Started
    • Key Concepts
    • Develop and Deploy an ML Application
    • Deploy Compositions of Models
    • Deploy Multiple Applications
    • Model Multiplexing
    • Model Registry Integration
    • Configure Ray Serve deployments
    • Set Up FastAPI and HTTP
    • Serving LLMs
      • Quickstart
      • Examples
        • Deploy a small-sized LLM
        • Deploy a medium-sized LLM
        • Deploy a large-sized LLM
        • Deploy a vision LLM
        • Deploy a reasoning LLM
        • Deploy a hybrid reasoning LLM
        • Deploy gpt-oss
        • Deploy NVIDIA Nemotron-3-Super-120B-A12B-FP8
      • User Guides
        • Configuration reference
        • Deployment initialization
        • Multi-LoRA deployment
        • Cross-node parallelism
        • Data parallel attention
        • Fractional GPU serving
        • Prefill/decode disaggregation
        • Direct streaming
        • Prefix-aware routing
        • KV-aware routing
        • KV cache offloading
        • vLLM compatibility
        • Custom vLLM models
        • SGLang integration
        • TPU serving
        • Observability and monitoring
      • Architecture
        • Architecture overview
        • Core components
        • Serving patterns
        • Request routing
      • Benchmarks
      • Troubleshooting
    • Production Guide
      • Serve Config Files
      • Deploy on Kubernetes
      • Custom Docker Images
      • Add End-to-End Fault Tolerance
      • Handle Dependencies
      • Best practices in production
    • Monitor Your Application
    • Resource Allocation
    • Ray Serve Autoscaling
    • Asynchronous Inference
    • Advanced Guides
      • Pass Arguments to Applications
      • Advanced Ray Serve Autoscaling
      • Asyncio and concurrency best practices in Ray Serve
      • Performance Tuning
      • Dynamic Request Batching
      • Updating Applications In-Place
      • Development Workflow
      • Set Up a gRPC Service
      • Replica ranks
      • Replica scheduling
      • Gang scheduling
      • Experimental Java API
      • Deploy on VM
      • Run Multiple Applications in Different Containers
      • Use Custom Algorithm for Request Routing
      • Use deployment-scoped actors
      • Troubleshoot multi-node GPU serving on KubeRay
    • Architecture
    • Examples
  • Ray RLlib
    • Getting Started
    • Key concepts
    • Environments
      • Multi-agent environments
      • Hierarchical environments
      • External environments and applications
    • AlgorithmConfig API
    • Algorithms
    • User Guides
      • Advanced Python APIs
      • Callbacks
      • Checkpointing
      • MetricsLogger API
      • Episodes
      • ConnectorV2 and ConnectorV2 pipelines
        • Env-to-module pipelines
        • Learner connector pipelines
      • Replay buffers
      • Working with offline data
      • RL Modules
      • Learner (Alpha)
      • Fault tolerance and elastic training
      • Install RLlib for development
      • RLlib scaling guide
    • Examples
    • New API stack migration guide
  • More Libraries
    • Distributed Scikit-learn / Joblib
    • Distributed multiprocessing.Pool
    • Ray Collective Communication Lib
      • Custom Collective Backends
    • Using Dask on Ray
    • Using Spark on Ray (RayDP)
    • Using Mars on Ray
    • Using Pandas on Ray (Modin)
    • Distributed Data Processing in Data-Juicer
  • APIs
    • Ray Data
      • Loading Data API
      • Saving Data API
      • Dataset API
      • DataIterator API
      • ExecutionOptions API
      • Checkpoint API
      • Aggregation API
      • GroupedData API
      • Expressions API
      • Data types
      • Global configuration
      • Preprocessor
      • Large Language Model (LLM) API
      • API Guide for Users from Other Data Libraries
    • Ray Train
      • PyTorch
      • PyTorch Lightning
      • Hugging Face Transformers
      • TensorFlow and Keras
      • XGBoost
      • LightGBM
      • JAX
      • Configuration
      • Training function utilities
      • Results
      • Exceptions
      • Ray Tune integration
      • Developer APIs
    • Ray Tune
      • Tune Execution (tune.Tuner)
      • Tune Experiment Results (tune.ResultGrid)
      • Training in Tune (tune.Trainable, tune.report)
      • Tune Search Space API
      • Tune Search Algorithms (tune.search)
      • Tune Trial Schedulers (tune.schedulers)
      • Tune Stopping Mechanisms (tune.stopper)
      • Tune Console Output (Reporters)
      • Syncing in Tune
      • Tune Loggers (tune.logger)
      • Tune Callbacks (tune.Callback)
      • Environment variables used by Ray Tune
      • External library integrations for Ray Tune
      • Tune Internals
      • Tune CLI (Experimental)
    • Ray Serve
      • Writing applications
      • Deployment handles
      • Running applications
      • Configuration
      • Runtime context
      • Request router
      • Schemas
      • Exceptions
      • Observability
      • LLM API
      • CLI
      • REST API
    • Ray RLlib
      • Algorithm Configuration API
      • Algorithms
      • Callback APIs
      • Environments
        • EnvRunner API
        • SingleAgentEnvRunner API
        • SingleAgentEpisode API
        • MultiAgentEnv API
        • MultiAgentEnvRunner API
        • MultiAgentEpisode API
        • External Envs
        • Env Utils
      • RLModule APIs
      • Distribution API
      • LearnerGroup API
      • Offline RL API
      • ConnectorV2 API
      • Replay Buffer API
      • RLlib Utilities
    • Ray Core
      • Core API
      • Scheduling API
      • Runtime Env API
      • Utility
      • Exceptions
      • Ray Core CLI
      • Sandbox API
      • State CLI
      • State API
      • Ray Direct Transport (RDT) API
      • Python SDK API Reference
      • Compiled Graph API
      • Programmatic Cluster Scaling
    • Ray integrations
      • Ray on Spark API
      • Ray Collective API
      • Dask on Ray API
  • Ecosystem
  • Ray Clusters
    • Key Concepts
    • Deploying on Kubernetes
    • Deploying on VMs
      • Getting Started
      • User Guides
        • Launching Ray Clusters on AWS, GCP, Azure, vSphere, On-Prem
        • Best practices for deploying large clusters
        • Configuring Autoscaling
        • Log Persistence
        • Community Supported Cluster Managers
      • Examples
        • Ray Train XGBoostTrainer on VMs
      • API References
        • Cluster Launcher Commands
        • Cluster YAML Configuration Options
    • Collecting and monitoring metrics
    • Configuring and managing Ray dashboard
    • Applications Guide
      • Ray Jobs Overview
        • Quickstart using the Ray Jobs CLI
        • Python SDK Overview
        • Python SDK API Reference
        • Ray Jobs CLI API Reference
        • Ray Jobs REST API
        • Ray Jobs REST API specification
        • Ray Client
      • Programmatic Cluster Scaling
    • FAQ
    • Ray Cluster Management API
      • Cluster Management CLI
      • Python SDK API Reference
      • Ray Jobs CLI API Reference
      • Programmatic Cluster Scaling
    • Usage Stats Collection
  • Monitoring and Debugging
    • Ray dashboard
    • Ray Distributed Debugger
    • Key Concepts
    • User Guides
      • Debugging Applications
        • Common Issues
        • Debugging Memory Issues
        • Debugging Hangs
        • Debugging Failures
        • Debugging high worker thread counts
        • Optimizing Performance
        • Using the Ray Debugger
      • Monitoring with the CLI or SDK
      • Configuring Logging
      • Profiling
      • JAX profiler for Ray on Kubernetes
      • Adding Application-Level Metrics
      • Tracing
      • Ray Event Export
    • Reference
      • State API
      • State CLI
      • System Metrics
  • Developer Guides
    • API stability
    • API policy
    • Getting involved and contributing
      • Building Ray from source
      • CI testing workflow on PRs
      • Editing and managing Python dependencies
      • Contributing to the Ray documentation
      • Publishing an example
      • Publishing an example with the in-tree notebook flow (legacy)
      • Ray documentation style guide
      • How to write code snippets
      • Testing autoscaling locally
      • Tips for testing Ray programs
      • Debugging for Ray developers
      • Profiling for Ray developers
      • Using agents for development
    • Configure Ray
    • Architecture whitepapers
  • Glossary
  • Security
    • Ray token authentication
  • Project Governance
    • People
  • What’s Ray Core?
  • Internals

Internals#

This section describes some of the internals of Ray Core. It’s primarily for advanced users and for developers who work on Ray Core. For a high-level overview of the architecture, see the Ray 2.0 Architecture white paper.

  • Task lifecycle
  • Streaming generator
  • Autoscaler v2
  • RPC fault tolerance
  • Token authentication
  • Metric exporter infrastructure
  • Ray event exporter infrastructure
  • Port service discovery
  • Object spilling

previous

Speed up your web crawler by parallelizing it with Ray

next

Task lifecycle
Edit on GitHub
Thanks for the feedback!
Was this helpful?
Yes
No
Feedback
Submit

© Copyright 2026, The Ray Team.

Created using Sphinx 8.2.3.

Built with the PyData Sphinx Theme 0.18.0.