Skip to main content
Ctrl+K
Try Ray with $100 credit — Start now

Site Navigation

  • Get Started

  • Use Cases

  • Example Gallery

  • Library

    • Ray CoreScale general Python applications

    • Ray DataScale data ingest and preprocessing

    • Ray TrainScale machine learning training

    • Ray TuneScale hyperparameter tuning

    • Ray ServeScale model serving

    • Ray RLlibScale reinforcement learning

  • APIs

  • Resources

    • Discussion ForumGet your Ray questions answered

    • TrainingHands-on learning

    • BlogUpdates, best practices, user-stories

    • EventsWebinars, meetups, office hours

    • Success StoriesReal-world workload examples

    • EcosystemLibraries integrated with Ray

    • CommunityConnect with us

Try Managed Ray

Site Navigation

  • Get Started

  • Use Cases

  • Example Gallery

  • Library

    • Ray CoreScale general Python applications

    • Ray DataScale data ingest and preprocessing

    • Ray TrainScale machine learning training

    • Ray TuneScale hyperparameter tuning

    • Ray ServeScale model serving

    • Ray RLlibScale reinforcement learning

  • APIs

  • Resources

    • Discussion ForumGet your Ray questions answered

    • TrainingHands-on learning

    • BlogUpdates, best practices, user-stories

    • EventsWebinars, meetups, office hours

    • Success StoriesReal-world workload examples

    • EcosystemLibraries integrated with Ray

    • CommunityConnect with us

Try Managed Ray
  • Overview
  • Getting Started
  • Installation
  • Use Cases
    • Ray for ML Infrastructure
  • Examples
    • Multi-modal AI pipeline
      • Batch inference
      • Distributed training
      • Online serving
    • LLM training and inference
    • Audio batch inference
    • Distributed XGBoost pipeline
      • Distributed training of an XGBoost model
      • Model validation using offline batch inference
      • Scalable online XGBoost inference with Ray Serve
    • Time-series forecasting
      • Distributed training of a DLinear time-series model
      • DLinear model validation using offline batch inference
      • Online serving for DLinear model using Ray Serve
    • Scalable video processing
      • Fine-tuning a face mask detection model with Faster R-CNN
      • Object detection batch inference on test dataset and metrics calculation
      • Video processing with object detection using batch inference
      • Host an object detection model as a service
    • Distributed RAG pipeline
      • Build a Regular RAG Document Ingestion Pipeline (No Ray required)
      • Scalable RAG Data Ingestion and Pagination with Ray Data
      • Deploy LLM with Ray Serve LLM
      • Build Basic RAG App
      • Improve RAG with Prompt Engineering
      • Evaluate RAG with Online Inference
      • Evaluate RAG using Batch Inference with Ray Data LLM
    • Deploy MCP servers
      • Deploying a custom MCP in Streamable HTTP mode with Ray Serve
      • Deploy an MCP Gateway with existing Ray Serve apps
      • Deploying an MCP STDIO Server as a scalable HTTP service with Ray Serve
      • Deploying multiple MCP services with Ray Serve
      • Build a Docker image for an MCP server
    • Build a tool-using agent
    • Build a multi-agent system with the A2A protocol
  • Ecosystem
  • Ray Core
    • Key Concepts
    • User Guides
      • Tasks
        • Nested Remote Functions
      • Actors
        • Named Actors
        • Terminating Actors
        • AsyncIO / Concurrency for Actors
        • Limiting Concurrency Per-Method with Concurrency Groups
        • Utility Classes
        • Out-of-band Communication
        • Actor Task Execution Order
      • Objects
        • Serialization
        • Object Spilling
      • Environment Dependencies
      • Scheduling
        • Use labels to control scheduling
        • Resources
        • Accelerator Support
        • Placement Groups
        • Memory Management
        • Out-Of-Memory Prevention
      • Fault tolerance
        • Task Fault Tolerance
        • Actor Fault Tolerance
        • Object Fault Tolerance
        • Node Fault Tolerance
        • GCS Fault Tolerance
      • Design Patterns & Anti-patterns
        • Pattern: Using nested tasks to achieve nested parallelism
        • Pattern: Using generators to reduce heap memory usage
        • Pattern: Using ray.wait to limit the number of pending tasks
        • Pattern: Using resources to limit the number of concurrently running tasks
        • Pattern: Using asyncio to run actor methods concurrently
        • Pattern: Using an actor to synchronize other tasks and actors
        • Pattern: Using a supervisor actor to manage a tree of actors
        • Pattern: Using pipelining to increase throughput
        • Anti-pattern: Returning ray.put() ObjectRefs from a task harms performance and fault tolerance
        • Anti-pattern: Calling ray.get on task arguments harms performance
        • Anti-pattern: Calling ray.get in a loop harms parallelism
        • Anti-pattern: Calling ray.get unnecessarily harms performance
        • Anti-pattern: Processing results in submission order using ray.get increases runtime
        • Anti-pattern: Fetching too many objects at once with ray.get causes failure
        • Anti-pattern: Over-parallelizing with too fine-grained tasks harms speedup
        • Anti-pattern: Redefining the same remote function or class harms performance
        • Anti-pattern: Passing the same large argument by value repeatedly harms performance
        • Anti-pattern: Closure capturing large objects harms performance
        • Anti-pattern: Using global variables to share state between tasks and actors
        • Anti-pattern: Serialize ray.ObjectRef out of band
        • Anti-pattern: Forking new processes in application code
      • Ray Direct Transport (RDT)
        • Implementing a custom tensor transport (Advanced)
      • Ray Compiled Graph (beta)
        • Quickstart
        • Profiling
        • Experimental: Overlapping communication and computation
        • Troubleshooting
        • Compiled Graph API
      • Resource Isolation With Cgroup v2
      • Advanced topics
        • Tips for first-time users
        • Type hints in Ray
        • Starting Ray
        • Ray Generators
        • Using Namespaces
        • Cross-language programming
        • Working with Jupyter Notebooks & JupyterLab
        • Lazy Computation Graphs with the Ray DAG API
        • Miscellaneous Topics
        • Authenticating Remote URIs in runtime_env
        • Lifetimes of a User-Spawn Process
        • Head Node Memory Management
    • Examples
      • Batch Prediction with Ray Core
      • A Gentle Introduction to Ray Core by Example
      • Using Ray for Highly Parallelizable Tasks
      • A Simple MapReduce Example with Ray Core
      • Monte Carlo Estimation of π
      • Simple Parallel Model Selection
      • Parameter Server
      • Learning to Play Pong
      • Speed up your web crawler by parallelizing it with Ray
    • Internals
      • Task Lifecycle
      • Streaming Generator
      • Autoscaler v2
      • RPC Fault Tolerance
      • Token Authentication
      • Metric Exporter Infrastructure
      • Ray Event Exporter Infrastructure
      • Port Service Discovery
      • Object Spilling
  • Ray Data
    • Ray Data Quickstart
    • Key Concepts
    • User Guides
      • Loading Data
      • Inspecting Data
      • Transforming Data
      • Aggregating Data
      • Iterating over Data
      • Joining Data
      • Shuffling Data
      • Weighted Dataset Mixing
      • Saving Data
      • Working with Images
      • Working with Text
      • Working with Tensors / NumPy
      • Working with PyTorch
      • Working with LLMs
      • How to avoid out-of-memory errors (OOMs)
      • Monitoring Your Workload
      • Execution Configurations
      • Run multiple Datasets in one cluster
      • End-to-end: Offline Batch Inference
      • Advanced: Performance Tips and Tuning
      • Advanced: Scaling out expensive collate functions
      • Advanced: Read and Write Custom File Types
    • Examples
    • Contributing to Ray Data
      • Contributing Guide
      • How to write tests
    • Comparing Ray Data to other systems
    • Ray Data Benchmarks
    • Ray Data Internals
  • Ray Train
    • Overview
    • PyTorch Guide
    • PyTorch Lightning Guide
    • Hugging Face Transformers Guide
    • XGBoost Guide
    • JAX Guide
    • More Frameworks
      • Hugging Face Accelerate Guide
      • DeepSpeed Guide
      • TensorFlow and Keras Guide
      • LightGBM Guide
      • Horovod Guide
    • User Guides
      • Data Loading and Preprocessing
      • Configuring Scale and Accelerators
      • Configuring Persistent Storage
      • Monitoring and Logging Metrics
      • Saving and Loading Checkpoints
      • Validating checkpoints asynchronously
      • Experiment Tracking
      • Inspecting Training Results
      • Handling Failures and Node Preemption
      • Elastic training
      • Ray Train Metrics
      • Local Mode
      • Reproducibility
      • Hyperparameter Optimization
    • Tutorials
      • Introduction to Ray Train workloads
      • Computer vision pattern
      • Tabular workload pattern
      • Time series workload pattern
      • Generative computer vision pattern
      • Diffusion policy pattern
      • Recommendation system pattern
    • Examples
    • Benchmarks
  • Ray Tune
    • Getting Started
    • Key Concepts
    • User Guides
      • Running Basic Experiments
      • Logging and Outputs in Tune
      • Setting Trial Resources
      • Using Search Spaces
      • How to Define Stopping Criteria for a Ray Tune Experiment
      • How to Save and Load Trial Checkpoints
      • How to Configure Persistent Storage in Ray Tune
      • How to Enable Fault Tolerance in Ray Tune
      • Using Callbacks and Metrics
      • Getting Data in and out of Tune
      • Analyzing Tune Experiment Results
      • A Guide to Population Based Training with Tune
        • Visualizing and Understanding PBT
      • Deploying Tune in the Cloud
      • Tune Architecture
      • Scalability Benchmarks
    • Ray Tune Examples
      • PyTorch Example
      • PyTorch Lightning Example
      • XGBoost Example
      • LightGBM Example
      • Hugging Face Transformers Example
      • Ray RLlib Example
      • Keras Example
      • PyTorch with ASHA
      • Weights & Biases Example
      • MLflow Example
      • Aim Example
      • Comet Example
      • Ax Example
      • HyperOpt Example
      • Bayesopt Example
      • BOHB Example
      • Nevergrad Example
      • Optuna Example
    • Ray Tune FAQ
  • Ray Serve
    • Getting Started
    • Key Concepts
    • Develop and Deploy an ML Application
    • Deploy Compositions of Models
    • Deploy Multiple Applications
    • Model Multiplexing
    • Model Registry Integration
    • Configure Ray Serve deployments
    • Set Up FastAPI and HTTP
    • Serving LLMs
      • Quickstart
      • Examples
        • Deploy a small-sized LLM
        • Deploy a medium-sized LLM
        • Deploy a large-sized LLM
        • Deploy a vision LLM
        • Deploy a reasoning LLM
        • Deploy a hybrid reasoning LLM
        • Deploy gpt-oss
      • User Guides
        • Configuration reference
        • Deployment initialization
        • Multi-LoRA deployment
        • Cross-node parallelism
        • Data parallel attention
        • Fractional GPU serving
        • Prefill/decode disaggregation
        • KV cache offloading
        • Prefix-aware routing
        • Direct streaming
        • vLLM compatibility
        • SGLang integration
        • Observability and monitoring
      • Architecture
        • Architecture overview
        • Core components
        • Serving patterns
        • Request routing
      • Benchmarks
      • Troubleshooting
    • Production Guide
      • Serve Config Files
      • Deploy on Kubernetes
      • Custom Docker Images
      • Add End-to-End Fault Tolerance
      • Handle Dependencies
      • Best practices in production
    • Monitor Your Application
    • Resource Allocation
    • Ray Serve Autoscaling
    • Asynchronous Inference
    • Advanced Guides
      • Pass Arguments to Applications
      • Advanced Ray Serve Autoscaling
      • Asyncio and concurrency best practices in Ray Serve
      • Performance Tuning
      • Dynamic Request Batching
      • Updating Applications In-Place
      • Development Workflow
      • Set Up a gRPC Service
      • Replica ranks
      • Replica scheduling
      • Gang scheduling
      • Experimental Java API
      • Deploy on VM
      • Run Multiple Applications in Different Containers
      • Use Custom Algorithm for Request Routing
      • Use deployment-scoped actors
      • Troubleshoot multi-node GPU serving on KubeRay
    • Architecture
    • Examples
  • Ray RLlib
    • Getting Started
    • Key concepts
    • Environments
      • Multi-Agent Environments
      • Hierarchical Environments
      • External Environments and Applications
    • AlgorithmConfig API
    • Algorithms
    • User Guides
      • Advanced Python APIs
      • Callbacks
      • Checkpointing
      • MetricsLogger API
      • Episodes
      • ConnectorV2 and ConnectorV2 pipelines
        • Env-to-module pipelines
        • Learner connector pipelines
      • Replay Buffers
      • Working with offline data
      • RL Modules
      • Learner (Alpha)
      • Fault Tolerance And Elastic Training
      • Install RLlib for Development
      • RLlib scaling guide
    • Examples
    • New API stack migration guide
  • More Libraries
    • Distributed Scikit-learn / Joblib
    • Distributed multiprocessing.Pool
    • Ray Collective Communication Lib
      • Custom Collective Backends
    • Using Dask on Ray
      • RayDaskCallback
        • ray_active
      • _ray_presubmit
      • _ray_postsubmit
      • _ray_pretask
      • _ray_posttask
      • _ray_postsubmit_all
      • _ray_finish
    • Using Spark on Ray (RayDP)
    • Using Mars on Ray
    • Using Pandas on Ray (Modin)
    • Distributed Data Processing in Data-Juicer
  • APIs
    • Ray Data
      • Loading Data API
      • Saving Data API
      • Dataset API
      • DataIterator API
      • ExecutionOptions API
      • Checkpoint API
      • Aggregation API
      • GroupedData API
      • Expressions API
      • Data types
      • Global configuration
      • Preprocessor
      • Large Language Model (LLM) API
      • API Guide for Users from Other Data Libraries
    • Ray Train
    • Ray Tune
      • Tune Execution (tune.Tuner)
      • Tune Experiment Results (tune.ResultGrid)
      • Training in Tune (tune.Trainable, tune.report)
      • Tune Search Space API
      • Tune Search Algorithms (tune.search)
      • Tune Trial Schedulers (tune.schedulers)
      • Tune Stopping Mechanisms (tune.stopper)
      • Tune Console Output (Reporters)
      • Syncing in Tune
      • Tune Loggers (tune.logger)
      • Tune Callbacks (tune.Callback)
      • Environment variables used by Ray Tune
      • External library integrations for Ray Tune
      • Tune Internals
      • Tune CLI (Experimental)
    • Ray Serve
    • Ray RLlib
      • Algorithm Configuration API
        • AlgorithmConfig
        • build_algo
        • build_learner_group
        • build_learner
        • is_multi_agent
        • is_offline
        • learner_class
        • model_config
        • rl_module_spec
        • total_train_batch_size
        • get_default_learner_class
        • get_default_rl_module_spec
        • get_evaluation_config_object
        • get_multi_rl_module_spec
        • get_multi_agent_setup
        • get_rollout_fragment_length
        • copy
        • validate
        • freeze
      • Algorithms
        • Algorithm
        • setup
        • get_default_config
        • env_runner
        • eval_env_runner
        • train
        • training_step
        • save_to_path
        • restore_from_path
        • from_checkpoint
        • get_state
        • set_state
        • evaluate
        • get_module
        • add_policy
        • remove_policy
      • Callback APIs
        • RLlibCallback
        • on_algorithm_init
        • on_sample_end
        • on_train_result
        • on_evaluate_start
        • on_evaluate_end
        • on_env_runners_recreated
        • on_checkpoint_loaded
        • on_environment_created
        • on_episode_created
        • on_episode_start
        • on_episode_step
        • on_episode_end
      • Environments
        • EnvRunner API
        • SingleAgentEnvRunner API
        • SingleAgentEpisode API
        • MultiAgentEnv API
        • MultiAgentEnvRunner API
        • MultiAgentEpisode API
        • External Envs
        • Env Utils
      • RLModule APIs
        • RLModuleSpec
        • build
        • module_class
        • observation_space
        • action_space
        • inference_only
        • learner_only
        • model_config
        • MultiRLModuleSpec
        • build
        • DefaultModelConfig
        • RLModule
        • observation_space
        • action_space
        • inference_only
        • model_config
        • setup
        • as_multi_rl_module
        • forward_exploration
        • forward_inference
        • forward_train
        • _forward
        • _forward_exploration
        • _forward_inference
        • _forward_train
        • save_to_path
        • restore_from_path
        • from_checkpoint
        • get_state
        • set_state
        • MultiRLModule
        • setup
        • as_multi_rl_module
        • add_module
        • remove_module
        • save_to_path
        • restore_from_path
        • from_checkpoint
        • get_state
        • set_state
      • Distribution API
        • Distribution
        • from_logits
        • sample
        • rsample
        • logp
        • kl
      • LearnerGroup API
        • learners
        • build_learner_group
        • LearnerGroup
      • Offline RL API
        • offline_data
        • learners
        • env_runners
        • OfflineSingleAgentEnvRunner
        • OfflineData
        • __init__
        • sample
        • default_map_batches_kwargs
        • default_iter_batches_kwargs
        • OfflinePreLearner
        • __init__
        • SCHEMA
        • __call__
        • _map_to_episodes
        • _map_sample_batch_to_episode
        • _should_module_be_updated
      • ConnectorV2 API
      • Replay Buffer API
        • StorageUnit
        • ReplayBuffer
        • PrioritizedReplayBuffer
        • ReservoirReplayBuffer
        • sample
        • add
        • get_state
        • set_state
        • MultiAgentReplayBuffer
        • MultiAgentPrioritizedReplayBuffer
        • update_priorities_in_replay_buffer
        • sample_min_n_steps_from_buffer
      • RLlib Utilities
        • MetricsLogger
        • peek
        • log_value
        • log_dict
        • aggregate
        • log_time
        • Scheduler
        • validate
        • get_current_value
        • update
        • _create_tensor_variable
        • try_import_torch
        • clip_gradients
        • compute_global_norm
        • convert_to_torch_tensor
        • explained_variance
        • flatten_inputs_to_1d_tensor
        • global_norm
        • one_hot
        • reduce_mean_ignore_inf
        • sequence_mask
        • set_torch_seed
        • softmax_cross_entropy_with_logits
        • update_target_network
        • aligned_array
        • concat_aligned
        • convert_to_numpy
        • fc
        • flatten_inputs_to_1d_tensor
        • make_action_immutable
        • huber_loss
        • l2_loss
        • lstm
        • one_hot
        • relu
        • sigmoid
        • softmax
        • try_import_msgpack
        • Checkpointable
    • Ray Core
      • Core API
      • Scheduling API
      • Runtime Env API
      • Utility
      • Exceptions
      • Ray Core CLI
      • State CLI
      • State API
      • Ray Direct Transport (RDT) API
  • Ray Clusters
    • Key Concepts
    • Deploying on Kubernetes
      • Getting Started with KubeRay
        • KubeRay Operator Installation
        • RayCluster Quickstart
        • RayJob Quickstart
        • RayService Quickstart
        • RayCronJob Quickstart
      • User Guides
        • Deploy Ray Serve Apps
        • RayService worker Pods aren’t ready
        • RayService high availability
        • Enable High Throughput on Ray Serve with KubeRay
        • RayService Zero-Downtime Incremental Upgrades
        • KubeRay Observability
        • KubeRay upgrade guide
        • Managed Kubernetes services
        • Best Practices for Storage and Dependencies
        • RayCluster Configuration
        • KubeRay Autoscaling
        • KubeRay In-Place Pod Resizing (IPPR)
        • KubeRay label-based scheduling
        • GCS fault tolerance in KubeRay
        • Tuning Redis for a Persistent Fault Tolerant GCS
        • Configuring KubeRay to use Google Cloud Storage Buckets in GKE
        • Persist KubeRay custom resource logs
        • Persist KubeRay Operator Logs
        • Using GPUs
        • Use TPUs with KubeRay
        • Specify container commands for Ray head/worker Pods
        • Helm Chart RBAC
        • TLS Authentication
        • (Advanced) Understanding the Ray Autoscaler in the Context of Kubernetes
        • Use kubectl plugin (beta)
        • Configure Ray clusters to use token authentication
        • Configure Ray clusters to use Kubernetes RBAC authentication
        • Reducing image pull latency on Kubernetes
        • Using uv for Python package management in KubeRay
        • Use KubeRay dashboard (experimental)
        • Resource Isolation with Writable Cgroups on Google Kubernetes Engine (GKE)
        • Ray History Server with KubeRay
      • Examples
        • Train a PyTorch model on Fashion MNIST with CPUs on Kubernetes
        • Serve a StableDiffusion text-to-image model on Kubernetes
        • Serve a Stable Diffusion model on GKE with TPUs
        • Serve a MobileNet image classifier on Kubernetes
        • Serve a text summarizer on Kubernetes
        • RayJob Batch Inference Example
        • Priority Scheduling with RayJob and Kueue
        • Gang Scheduling with RayJob and Kueue
        • Distributed checkpointing with KubeRay and GCSFuse
        • Serve a Large Language Model using Ray Serve LLM on Kubernetes
        • Serve Deepseek R1 using Ray Serve LLM
        • Reinforcement Learning with Human Feedback (RLHF) for LLMs with verl on KubeRay
        • Deploying Ray Clusters via ArgoCD
        • Sandboxed Code Execution with Ray and Agent Sandbox
      • KubeRay Ecosystem
        • Ingress
        • KubeRay metrics references
        • Using Prometheus and Grafana
        • Profiling with py-spy
        • Gang scheduling, queue priority, and GPU sharing for RayClusters using KAI Scheduler
        • KubeRay integration with Volcano
        • KubeRay integration with Apache YuniKorn
        • Gang scheduling, Priority scheduling, and Autoscaling for KubeRay CRDs with Kueue
        • mTLS and L7 observability with Istio
        • KubeRay integration with scheduler plugins
      • KubeRay Benchmarks
        • KubeRay memory and scalability benchmark
      • KubeRay Troubleshooting
        • Troubleshooting guide
        • RayService troubleshooting
      • API Reference
    • Deploying on VMs
      • Getting Started
      • User Guides
        • Launching Ray Clusters on AWS, GCP, Azure, vSphere, On-Prem
        • Best practices for deploying large clusters
        • Configuring Autoscaling
        • Log Persistence
        • Community Supported Cluster Managers
      • Examples
        • Ray Train XGBoostTrainer on VMs
      • API References
        • Cluster Launcher Commands
        • Cluster YAML Configuration Options
    • Collecting and monitoring metrics
    • Configuring and Managing Ray Dashboard
    • Applications Guide
      • Ray Jobs Overview
        • Quickstart using the Ray Jobs CLI
        • Python SDK Overview
        • Python SDK API Reference
        • Ray Jobs CLI API Reference
        • Ray Jobs REST API
        • Ray Client
      • Programmatic Cluster Scaling
    • FAQ
    • Ray Cluster Management API
      • Cluster Management CLI
      • Python SDK API Reference
      • Ray Jobs CLI API Reference
      • Programmatic Cluster Scaling
    • Usage Stats Collection
  • Monitoring and Debugging
    • Ray Dashboard
    • Ray Distributed Debugger
    • Key Concepts
    • User Guides
      • Debugging Applications
        • Common Issues
        • Debugging Memory Issues
        • Debugging Hangs
        • Debugging Failures
        • Optimizing Performance
        • Ray Distributed Debugger
        • Using the Ray Debugger
      • Monitoring with the CLI or SDK
      • Configuring Logging
      • Profiling
      • Adding Application-Level Metrics
      • Tracing
      • Ray Event Export
    • Reference
      • State API
      • State CLI
      • System Metrics
  • Developer Guides
    • API stability
    • API policy
    • Getting involved and contributing
      • Building Ray from source
      • CI testing workflow on PRs
      • Contributing to the Ray documentation
      • How to write code snippets
      • Testing autoscaling locally
      • Tips for testing Ray programs
      • Debugging for Ray developers
      • Profiling for Ray developers
      • Using agents for development
    • Configuring Ray
    • Architecture whitepapers
  • Glossary
  • Security
    • Ray token authentication
  • Project Governance
    • People
  • Developer guides
  • Architecture whitepapers

Architecture whitepapers#

For an in-depth overview of Ray internals, check out the Ray 2.0 Architecture whitepaper. For the previous version, see the v1.0 whitepaper.

For more about the scalability and performance of the Ray dataplane, see the Exoshuffle paper.

previous

Configuring Ray

next

Ray Glossary
Edit on GitHub
Thanks for the feedback!
Was this helpful?
Yes
No
Feedback
Submit

© Copyright 2026, The Ray Team.

Created using Sphinx 8.2.3.

Built with the PyData Sphinx Theme 0.18.0.