Skip to main content
Ctrl+K
Try Ray with $100 credit — Start now
Ray 2.59 enables token authentication by default for local clusters, and Ray 2.61 extends it to all clusters. Read the rollout details

Site Navigation

  • Get Started

  • Use Cases

  • Example Gallery

  • Library

    • Ray CoreScale general Python applications

    • Ray DataScale data ingest and preprocessing

    • Ray TrainScale machine learning training

    • Ray TuneScale hyperparameter tuning

    • Ray ServeScale model serving

    • Ray RLlibScale reinforcement learning

  • APIs

  • Resources

    • Discussion ForumGet your Ray questions answered

    • TrainingHands-on learning

    • BlogUpdates, best practices, user-stories

    • EventsWebinars, meetups, office hours

    • Success StoriesReal-world workload examples

    • EcosystemLibraries integrated with Ray

    • CommunityConnect with us

Try Managed Ray

Site Navigation

  • Get Started

  • Use Cases

  • Example Gallery

  • Library

    • Ray CoreScale general Python applications

    • Ray DataScale data ingest and preprocessing

    • Ray TrainScale machine learning training

    • Ray TuneScale hyperparameter tuning

    • Ray ServeScale model serving

    • Ray RLlibScale reinforcement learning

  • APIs

  • Resources

    • Discussion ForumGet your Ray questions answered

    • TrainingHands-on learning

    • BlogUpdates, best practices, user-stories

    • EventsWebinars, meetups, office hours

    • Success StoriesReal-world workload examples

    • EcosystemLibraries integrated with Ray

    • CommunityConnect with us

Try Managed Ray
  • Overview
  • Getting Started
  • Installation
  • Use Cases
    • Ray for ML Infrastructure
  • Examples
    • Multi-modal AI pipeline
      • Batch inference
      • Distributed training
      • Online serving
    • LLM training and inference
    • Audio batch inference
    • Distributed XGBoost pipeline
      • Distributed training of an XGBoost model
      • Model validation using offline batch inference
      • Scalable online XGBoost inference with Ray Serve
    • Time-series forecasting
      • Distributed training of a DLinear time-series model
      • DLinear model validation using offline batch inference
      • Online serving for DLinear model using Ray Serve
    • Scalable video processing
      • Fine-tuning a face mask detection model with Faster R-CNN
      • Object detection batch inference on test dataset and metrics calculation
      • Video processing with object detection using batch inference
      • Host an object detection model as a service
    • Distributed RAG pipeline
      • Build a Regular RAG Document Ingestion Pipeline (No Ray required)
      • Scalable RAG Data Ingestion and Pagination with Ray Data
      • Deploy LLM with Ray Serve LLM
      • Build Basic RAG App
      • Improve RAG with Prompt Engineering
      • Evaluate RAG with Online Inference
      • Evaluate RAG using Batch Inference with Ray Data LLM
    • Deploy MCP servers
      • Deploying a custom MCP in Streamable HTTP mode with Ray Serve
      • Deploy an MCP Gateway with existing Ray Serve apps
      • Deploying an MCP STDIO Server as a scalable HTTP service with Ray Serve
      • Deploying multiple MCP services with Ray Serve
      • Build a Docker image for an MCP server
    • Build a tool-using agent
    • Build a multi-agent system with the A2A protocol
  • KubeRay
    • Getting Started with KubeRay
      • KubeRay Operator Installation
      • RayCluster Quickstart
      • RayJob Quickstart
      • RayService Quickstart
      • RayCronJob Quickstart
    • User Guides
      • Deploy Ray Serve Apps
      • RayService worker Pods aren’t ready
      • RayService high availability
      • Enable High Throughput on Ray Serve with KubeRay
      • RayService Zero-Downtime Incremental Upgrades
      • KubeRay Observability
      • KubeRay upgrade guide
      • Managed Kubernetes services
        • Start Amazon EKS Cluster with GPUs for KubeRay
        • Start Google Cloud GKE Cluster with GPUs for KubeRay
        • Start a Google Cloud GKE cluster with TPUs for KubeRay
        • Start Azure AKS Cluster with GPUs for KubeRay
        • Start an Aliyun ACK cluster with GPUs for KubeRay
      • Best Practices for Storage and Dependencies
      • RayCluster Configuration
      • Scheduling on Kubernetes
      • KubeRay Autoscaling
      • KubeRay In-Place Pod Resizing (IPPR)
      • KubeRay label-based scheduling
      • GCS fault tolerance in KubeRay
      • Tuning Redis for a Persistent Fault Tolerant GCS
      • GCS fault tolerance with embedded RocksDB (alpha)
      • Configuring KubeRay to use Google Cloud Storage Buckets in GKE
      • Persist KubeRay custom resource logs
      • Persist KubeRay Operator Logs
      • Using GPUs
      • Use TPUs with KubeRay
      • Specify container commands for Ray head/worker Pods
      • Helm Chart RBAC
      • TLS Authentication
      • Configure RayCluster to use NetworkPolicies
      • Configuring mTLS for RayClusters
      • (Advanced) Understanding the Ray Autoscaler in the Context of Kubernetes
      • Use kubectl plugin (beta)
      • Configure Ray clusters to use token authentication
      • Configure Ray clusters to use Kubernetes RBAC authentication
      • Reducing image pull latency on Kubernetes
      • Using uv for Python package management in KubeRay
      • Use KubeRay dashboard (experimental)
      • Resource isolation with writable cgroups on Google Kubernetes Engine (GKE)
      • Ray History Server with KubeRay
      • Enable Ray platform events on Kubernetes
      • RayJob SidecarSubmitterRestart
    • Examples
      • Train a PyTorch model on Fashion MNIST with CPUs on Kubernetes
      • Serve a StableDiffusion text-to-image model on Kubernetes
      • Serve a Stable Diffusion model on GKE with TPUs
      • Serve a MobileNet image classifier on Kubernetes
      • Serve a text summarizer on Kubernetes
      • RayJob Batch Inference Example
      • Priority Scheduling with RayJob and Kueue
      • Gang Scheduling with RayJob and Kueue
      • Distributed checkpointing with KubeRay and GCSFuse
      • Serve a Large Language Model using Ray Serve LLM on Kubernetes
      • Serve Deepseek R1 using Ray Serve LLM
      • Reinforcement Learning with Human Feedback (RLHF) for LLMs with verl on KubeRay
      • Deploying Ray Clusters via ArgoCD
      • Sandboxed code execution with Ray and Agent Sandbox
      • Deploy Ray sandboxes with KubeRay
    • KubeRay Ecosystem
      • Ingress
      • KubeRay metrics references
      • Using Prometheus and Grafana
      • Profiling with py-spy
      • Gang scheduling, queue priority, and GPU sharing for RayClusters using KAI Scheduler
      • KubeRay integration with Volcano
      • KubeRay integration with Apache YuniKorn
      • Gang scheduling, Priority scheduling, and Autoscaling for KubeRay CRDs with Kueue
      • mTLS and L7 observability with Istio
      • KubeRay integration with scheduler plugins
    • KubeRay Benchmarks
      • KubeRay memory and scalability benchmark
    • KubeRay Troubleshooting
      • Troubleshooting guide
      • RayService troubleshooting
    • API Reference
      • KubeRay CRD API reference
  • Ray Core
    • Key Concepts
    • User Guides
      • Tasks
        • Nested remote functions
      • Actors
        • Named actors
        • Terminating actors
        • asyncio and concurrency for actors
        • Limiting concurrency per method with concurrency groups
        • Utility classes
        • Out-of-band communication
        • Actor task execution order
      • Objects
        • Serialization
        • Object spilling
      • Environment dependencies
      • Scheduling
        • Use labels to control scheduling
        • Resources
        • Accelerator support
        • Placement groups
        • Memory management
        • Out-of-memory prevention
      • Fault tolerance
        • Task fault tolerance
        • Actor fault tolerance
        • Object fault tolerance
        • Node fault tolerance
        • GCS fault tolerance
      • Design patterns and anti-patterns
        • Pattern: Using nested tasks to achieve nested parallelism
        • Pattern: Using generators to reduce heap memory usage
        • Pattern: Using ray.wait to limit the number of pending tasks
        • Pattern: Using resources to limit the number of concurrently running tasks
        • Pattern: Using asyncio to run actor methods concurrently
        • Pattern: Using an actor to synchronize other tasks and actors
        • Pattern: Using a supervisor actor to manage a tree of actors
        • Pattern: Using pipelining to increase throughput
        • Anti-pattern: Returning ray.put() ObjectRefs from a task harms performance and fault tolerance
        • Anti-pattern: Calling ray.get on task arguments harms performance
        • Anti-pattern: Calling ray.get in a loop harms parallelism
        • Anti-pattern: Calling ray.get unnecessarily harms performance
        • Anti-pattern: Processing results in submission order using ray.get increases runtime
        • Anti-pattern: Fetching too many objects at once with ray.get causes failure
        • Anti-pattern: Over-parallelizing with too fine-grained tasks harms speedup
        • Anti-pattern: Redefining the same remote function or class harms performance
        • Anti-pattern: Passing the same large argument by value repeatedly harms performance
        • Anti-pattern: Closure capturing large objects harms performance
        • Anti-pattern: Using global variables to share state between tasks and actors
        • Anti-pattern: Serializing ray.ObjectRef out of band
        • Anti-pattern: Forking new processes in application code
      • Ray Direct Transport (RDT)
        • Implementing a custom tensor transport (Advanced)
      • Ray Compiled Graph (beta)
        • Quickstart
        • Profiling
        • Experimental: Overlapping communication and computation
        • Troubleshooting
        • Compiled Graph API
      • Resource isolation with cgroup v2
      • Ray Sandboxes
      • Advanced topics
        • Tips for first-time users
        • Type hints in Ray
        • Starting Ray
        • Ray generators
        • Use namespaces
        • Cross-language programming
        • Use Ray with Jupyter Notebook and JupyterLab
        • Lazy computation graphs with the Ray DAG API
        • Miscellaneous topics
        • Authenticate remote URIs in runtime_env
        • Lifetimes of user-spawned processes
        • Head node memory management
    • Examples
      • Batch Prediction with Ray Core
      • A Gentle Introduction to Ray Core by Example
      • Using Ray for Highly Parallelizable Tasks
      • A Simple MapReduce Example with Ray Core
      • Monte Carlo estimation of π
      • Simple Parallel Model Selection
      • Parameter Server
      • Learning to Play Pong
      • Speed up your web crawler by parallelizing it with Ray
    • Internals
      • Task lifecycle
      • Streaming generator
      • Autoscaler v2
      • RPC fault tolerance
      • Token authentication
      • Metric exporter infrastructure
      • Ray event exporter infrastructure
      • Port service discovery
      • Object spilling
  • Ray Data
    • Ray Data quickstart
    • Key concepts
    • User guides
      • Loading data
      • Inspecting data
      • Transforming Data
      • Aggregating data
      • Iterating over data
      • Joining data
      • Shuffling data
      • Weighted dataset mixing
      • Saving data
      • Working with images
      • Working with text
      • Working with tensors and NumPy
      • Working with PyTorch
      • Working with LLMs
      • How to avoid out-of-memory errors (OOMs)
      • Monitoring your workload
      • Execution configurations
      • Run multiple Datasets in one cluster
      • End-to-end: Offline Batch Inference
      • Advanced: Performance tips and tuning
      • Advanced: Scaling out expensive collate functions
      • Advanced: Read and write custom file types
    • Examples
    • Contributing to Ray Data
      • Contributing Guide
      • How to write tests
    • Comparing Ray Data to other systems
    • Ray Data benchmarks
    • Ray Data internals
  • Ray Train
    • Overview
    • PyTorch guide
    • PyTorch Lightning guide
    • Hugging Face Transformers guide
    • XGBoost guide
    • JAX guide
    • More frameworks
      • Hugging Face Accelerate guide
      • DeepSpeed guide
      • TensorFlow and Keras guide
      • LightGBM guide
      • Horovod guide
    • User guides
      • Data loading and preprocessing
      • Configure scale and accelerators
      • Configure persistent storage
      • Monitor and log metrics
      • Save and load checkpoints
      • Validate checkpoints asynchronously
      • Experiment tracking
      • Inspect training results
      • Handle failures and node preemption
      • Elastic training
      • Detect NCCL hangs
      • Ray Train metrics
      • Local mode
      • Reproducibility
      • Hyperparameter optimization
    • Tutorials
      • Introduction to Ray Train workloads
      • Computer vision pattern
      • Tabular workload pattern
      • Time series workload pattern
      • Generative computer vision pattern
      • Diffusion policy pattern
      • Recommendation system pattern
    • Examples
    • Benchmarks
  • Ray Tune
    • Getting Started
    • Key Concepts
    • User Guides
      • Running Basic Experiments
      • Logging and Outputs in Tune
      • Setting Trial Resources
      • Using Search Spaces
      • How to Define Stopping Criteria for a Ray Tune Experiment
      • How to Save and Load Trial Checkpoints
      • How to Configure Persistent Storage in Ray Tune
      • How to Enable Fault Tolerance in Ray Tune
      • Using Callbacks and Metrics
      • Getting Data in and out of Tune
      • Analyzing Tune Experiment Results
      • A Guide to Population Based Training with Tune
        • Visualizing and Understanding PBT
      • Deploying Tune in the Cloud
      • Tune Architecture
      • Scalability Benchmarks
    • Ray Tune Examples
      • PyTorch Example
      • PyTorch Lightning Example
      • XGBoost Example
      • LightGBM Example
      • Hugging Face Transformers Example
      • Ray RLlib Example
      • Keras Example
      • PyTorch with ASHA
      • Weights & Biases Example
      • MLflow Example
      • Aim Example
      • Comet Example
      • Ax Example
      • HyperOpt Example
      • Bayesopt Example
      • BOHB Example
      • Nevergrad Example
      • Optuna Example
    • Ray Tune FAQ
  • Ray Serve
    • Getting Started
    • Key Concepts
    • Develop and Deploy an ML Application
    • Deploy Compositions of Models
    • Deploy Multiple Applications
    • Model Multiplexing
    • Model Registry Integration
    • Configure Ray Serve deployments
    • Set Up FastAPI and HTTP
    • Serving LLMs
      • Quickstart
      • Examples
        • Deploy a small-sized LLM
        • Deploy a medium-sized LLM
        • Deploy a large-sized LLM
        • Deploy a vision LLM
        • Deploy a reasoning LLM
        • Deploy a hybrid reasoning LLM
        • Deploy gpt-oss
        • Deploy NVIDIA Nemotron-3-Super-120B-A12B-FP8
      • User Guides
        • Configuration reference
        • Deployment initialization
        • Multi-LoRA deployment
        • Cross-node parallelism
        • Data parallel attention
        • Fractional GPU serving
        • Prefill/decode disaggregation
        • Direct streaming
        • Prefix-aware routing
        • KV-aware routing
        • KV cache offloading
        • vLLM compatibility
        • Custom vLLM models
        • SGLang integration
        • TPU serving
        • Observability and monitoring
      • Architecture
        • Architecture overview
        • Core components
        • Serving patterns
        • Request routing
      • Benchmarks
      • Troubleshooting
    • Production Guide
      • Serve Config Files
      • Deploy on Kubernetes
      • Custom Docker Images
      • Add End-to-End Fault Tolerance
      • Handle Dependencies
      • Best practices in production
    • Monitor Your Application
    • Resource Allocation
    • Ray Serve Autoscaling
    • Asynchronous Inference
    • Advanced Guides
      • Pass Arguments to Applications
      • Advanced Ray Serve Autoscaling
      • Asyncio and concurrency best practices in Ray Serve
      • Performance Tuning
      • Dynamic Request Batching
      • Updating Applications In-Place
      • Development Workflow
      • Set Up a gRPC Service
      • Replica ranks
      • Replica scheduling
      • Gang scheduling
      • Experimental Java API
      • Deploy on VM
      • Run Multiple Applications in Different Containers
      • Use Custom Algorithm for Request Routing
      • Use deployment-scoped actors
      • Troubleshoot multi-node GPU serving on KubeRay
    • Architecture
    • Examples
  • Ray RLlib
    • Getting Started
    • Key concepts
    • Environments
      • Multi-agent environments
      • Hierarchical environments
      • External environments and applications
    • AlgorithmConfig API
    • Algorithms
    • User Guides
      • Advanced Python APIs
      • Callbacks
      • Checkpointing
      • MetricsLogger API
      • Episodes
      • ConnectorV2 and ConnectorV2 pipelines
        • Env-to-module pipelines
        • Learner connector pipelines
      • Replay buffers
      • Working with offline data
      • RL Modules
      • Learner (Alpha)
      • Fault tolerance and elastic training
      • Install RLlib for development
      • RLlib scaling guide
    • Examples
    • New API stack migration guide
  • More Libraries
    • Distributed Scikit-learn / Joblib
    • Distributed multiprocessing.Pool
    • Ray Collective Communication Lib
      • Custom Collective Backends
    • Using Dask on Ray
    • Using Spark on Ray (RayDP)
    • Using Mars on Ray
    • Using Pandas on Ray (Modin)
    • Distributed Data Processing in Data-Juicer
  • APIs
    • Ray Data
      • Loading Data API
      • Saving Data API
      • Dataset API
      • DataIterator API
      • ExecutionOptions API
      • Checkpoint API
      • Aggregation API
      • GroupedData API
      • Expressions API
      • Data types
      • Global configuration
      • Preprocessor
      • Large Language Model (LLM) API
      • API Guide for Users from Other Data Libraries
    • Ray Train
      • PyTorch
      • PyTorch Lightning
      • Hugging Face Transformers
      • TensorFlow and Keras
      • XGBoost
      • LightGBM
      • JAX
      • Configuration
      • Training function utilities
      • Results
      • Exceptions
      • Ray Tune integration
      • Developer APIs
    • Ray Tune
      • Tune Execution (tune.Tuner)
      • Tune Experiment Results (tune.ResultGrid)
      • Training in Tune (tune.Trainable, tune.report)
      • Tune Search Space API
      • Tune Search Algorithms (tune.search)
      • Tune Trial Schedulers (tune.schedulers)
      • Tune Stopping Mechanisms (tune.stopper)
      • Tune Console Output (Reporters)
      • Syncing in Tune
      • Tune Loggers (tune.logger)
      • Tune Callbacks (tune.Callback)
      • Environment variables used by Ray Tune
      • External library integrations for Ray Tune
      • Tune Internals
      • Tune CLI (Experimental)
    • Ray Serve
      • Writing applications
      • Deployment handles
      • Running applications
      • Configuration
      • Runtime context
      • Request router
      • Schemas
      • Exceptions
      • Observability
      • LLM API
      • CLI
      • REST API
    • Ray RLlib
      • Algorithm Configuration API
      • Algorithms
      • Callback APIs
      • Environments
        • EnvRunner API
        • SingleAgentEnvRunner API
        • SingleAgentEpisode API
        • MultiAgentEnv API
        • MultiAgentEnvRunner API
        • MultiAgentEpisode API
        • External Envs
        • Env Utils
      • RLModule APIs
      • Distribution API
      • LearnerGroup API
      • Offline RL API
      • ConnectorV2 API
      • Replay Buffer API
      • RLlib Utilities
    • Ray Core
      • Core API
      • Scheduling API
      • Runtime Env API
      • Utility
      • Exceptions
      • Ray Core CLI
      • Sandbox API
      • State CLI
      • State API
      • Ray Direct Transport (RDT) API
      • Python SDK API Reference
      • Compiled Graph API
      • Programmatic Cluster Scaling
    • Ray integrations
      • Ray on Spark API
      • Ray Collective API
      • Dask on Ray API
  • Ecosystem
  • Ray Clusters
    • Key Concepts
    • Deploying on Kubernetes
    • Deploying on VMs
      • Getting Started
      • User Guides
        • Launching Ray Clusters on AWS, GCP, Azure, vSphere, On-Prem
        • Best practices for deploying large clusters
        • Configuring Autoscaling
        • Log Persistence
        • Community Supported Cluster Managers
      • Examples
        • Ray Train XGBoostTrainer on VMs
      • API References
        • Cluster Launcher Commands
        • Cluster YAML Configuration Options
    • Collecting and monitoring metrics
    • Configuring and managing Ray dashboard
    • Applications Guide
      • Ray Jobs Overview
        • Quickstart using the Ray Jobs CLI
        • Python SDK Overview
        • Python SDK API Reference
        • Ray Jobs CLI API Reference
        • Ray Jobs REST API
        • Ray Jobs REST API specification
        • Ray Client
      • Programmatic Cluster Scaling
    • FAQ
    • Ray Cluster Management API
      • Cluster Management CLI
      • Python SDK API Reference
      • Ray Jobs CLI API Reference
      • Programmatic Cluster Scaling
    • Usage Stats Collection
  • Monitoring and Debugging
    • Ray dashboard
    • Ray Distributed Debugger
    • Key Concepts
    • User Guides
      • Debugging Applications
        • Common Issues
        • Debugging Memory Issues
        • Debugging Hangs
        • Debugging Failures
        • Debugging high worker thread counts
        • Optimizing Performance
        • Using the Ray Debugger
      • Monitoring with the CLI or SDK
      • Configuring Logging
      • Profiling
      • JAX profiler for Ray on Kubernetes
      • Adding Application-Level Metrics
      • Tracing
      • Ray Event Export
    • Reference
      • State API
      • State CLI
      • System Metrics
  • Developer Guides
    • API stability
    • API policy
    • Getting involved and contributing
      • Building Ray from source
      • CI testing workflow on PRs
      • Editing and managing Python dependencies
      • Contributing to the Ray documentation
      • Publishing an example
      • Publishing an example with the in-tree notebook flow (legacy)
      • Ray documentation style guide
      • How to write code snippets
      • Testing autoscaling locally
      • Tips for testing Ray programs
      • Debugging for Ray developers
      • Profiling for Ray developers
      • Using agents for development
    • Configure Ray
    • Architecture whitepapers
  • Glossary
  • Security
    • Ray token authentication
  • Project Governance
    • People
  • Ray Train: Scalable model training
  • More frameworks

More frameworks#

Hugging Face Accelerate

Get started with distributed training using Hugging Face Accelerate

DeepSpeed

Get started with DeepSpeed

TensorFlow and Keras

Get started with distributed training using TensorFlow and Keras

LightGBM

Get started with distributed training using LightGBM

Horovod

Get started with distributed training using Horovod

previous

Get started with distributed training using JAX

next

Get started with distributed training using Hugging Face Accelerate
Edit on GitHub
Thanks for the feedback!
Was this helpful?
Yes
No
Feedback
Submit

© Copyright 2026, The Ray Team.

Created using Sphinx 8.2.3.

Built with the PyData Sphinx Theme 0.18.0.