Ray Train Examples#

Below are examples for using Ray Train with a variety of frameworks and use cases.

Beginner#

Framework

Example

PyTorch

Train a Fashion MNIST Image Classifier with PyTorch

Lightning

Train an MNIST Image Classifier with Lightning

Transformers

Fine-tune a Text Classifier on the Yelp Reviews Dataset with Hugging Face Transformers

Accelerate

Distributed Data Parallel Training with Hugging Face Accelerate

DeepSpeed

Train with DeepSpeed ZeRO-3

TensorFlow

Train an MNIST Image Classifier with TensorFlow

Horovod

Train with Horovod and PyTorch

Intermediate#

Framework

Example

PyTorch

Fine-tune of Stable Diffusion with DreamBooth and Ray Train

Lightning

Train with PyTorch Lightning and Ray Data

Transformers

Fine-tune a Text Classifier on GLUE Benchmark with Hugging Face Accelerate

Advanced#

Framework

Example

Accelerate, DeepSpeed

Fine-tune Llama-2 series models with Deepspeed, Accelerate, and Ray Train TorchTrainer

Transformers, DeepSpeed

Fine-tune GPT-J-6B with Ray Train and DeepSpeed

Lightning, DeepSpeed

Fine-tune vicuna-13b with PyTorch Lightning and DeepSpeed

Lightning

Fine-tune dolly-v2-7b with PyTorch Lightning and FSDP