Train Examples

Below are examples for using Ray Train with a variety of frameworks and use cases. Ray Train makes it easy to scale out each of these examples to a large cluster of GPUs.

Beginner

Framework	Example
PyTorch	Distributing your PyTorch Training Code with Ray Train and Ray Data
Lightning	Train an image classifier with Lightning
TensorFlow	Train an image classifier with TensorFlow
JAX, Flax	Train a GPT-2-style Transformer with JAX and Flax
PyTorch	Train ResNet model with Intel Gaudi
PyTorch	Profiling a Ray Train Workload with PyTorch Profiler
XGBoost	Train a tabular model with XGBoost

Intermediate

Framework	Example
PyTorch	Get started with PyTorch Fully Sharded Data Parallel (FSDP2) and Ray Train
PyTorch, DeepSpeed	Get started with Tensor Parallelism (DeepSpeed AutoTP) and Ray Train
PyTorch	Get started with 2D Parallelism (Tensor + Data Parallelism) using FSDP2 and Ray Train
PyTorch, DeepSpeed	Fine-tune an LLM with Ray Train and DeepSpeed
Accelerate, Transformers	Finetune Stable Diffusion and generate images with Intel Gaudi
Lightning	Train a text classifier with PyTorch Lightning and Ray Data
Transformers	Train a text classifier with Hugging Face Transformers
Transformers, TRL	RL Post-Train an LLM using HuggingFace TRL with GRPO
Accelerate, Transformers	Fine-tune Llama-2-7b and Llama-2-70b with Intel Gaudi

Advanced

Framework	Example
PyTorch, AWS Neuron	Fine-tune Llama3.1 with AWS Trainium