Ray Use Cases#

This page indexes common Ray use cases for scaling ML. It contains highlighted references to blogs, examples, and tutorials also located elsewhere in the Ray documentation.

LLMs and Gen AI#

Large language models (LLMs) and generative AI are rapidly changing industries, and demand compute at an astonishing pace. Ray provides a distributed compute framework for scaling these models, allowing developers to train and deploy models faster and more efficiently. With specialized libraries for data streaming, training, fine-tuning, hyperparameter tuning, and serving, Ray simplifies the process of developing and deploying large-scale AI models.


Learn more about how Ray scales LLMs and generative AI with the following resources.

Batch Inference#

Batch inference refers to generating model predictions over a set of input observations. The model could be a regression model, neural network, or simply a Python function. Ray can scale batch inference from single GPU machines to large clusters.

Performing inference on incoming batches of data can be parallelized by exporting the architecture and weights of a trained model to the shared object store. Using these model replicas, Ray AIR’s Batch Predictor scales predictions on batches across workers.


Using Ray AIR’s BatchPredictor for batch inference.#

Learn more about batch inference with the following resources.

Many Model Training#

Many model training is common in ML use cases such as time series forecasting, which require fitting of models on multiple data batches corresponding to locations, products, etc. The focus is on training many models on subsets of a dataset. This is in contrast to training a single model on the entire dataset.

When any given model you want to train can fit on a single GPU, Ray can assign each training run to a separate Ray Task. In this way, all available workers are utilized to run independent remote training rather than one worker running jobs sequentially.


Data parallelism pattern for distributed training on large datasets.#

How do I do many model training on Ray?#

To train multiple independent models, use the Ray Tune (Tutorial) library. This is the recommended library for most cases.

You can use Tune with your current data preprocessing pipeline if your data source fits into the memory of a single machine (node). If you need to scale your data, or you want to plan for future scaling, use the Ray Data library. Your data must be a supported format, to use Ray Data.

Alternative solutions exist for less common cases:

  1. If your data is not in a supported format, use Ray Core (Tutorial) for custom applications. This is an advanced option and requires and understanding of design patterns and anti-patterns.

  2. If you have a large preprocessing pipeline, you can use the Ray Data library to train multiple models (Tutorial).

Learn more about many model training with the following resources.

Model Serving#

Ray Serve is well suited for model composition, enabling you to build a complex inference service consisting of multiple ML models and business logic all in Python code.

It supports complex model deployment patterns requiring the orchestration of multiple Ray actors, where different actors provide inference for different models. Serve handles both batch and online inference and can scale to thousands of models in production.


Deployment patterns with Ray Serve. (Click image to enlarge.)#

Learn more about model serving with the following resources.

Hyperparameter Tuning#

The Ray Tune library enables any parallel Ray workload to be run under a hyperparameter tuning algorithm.

Running multiple hyperparameter tuning experiments is a pattern apt for distributed computing because each experiment is independent of one another. Ray Tune handles the hard bit of distributing hyperparameter optimization and makes available key features such as checkpointing the best result, optimizing scheduling, and specifying search patterns.


Distributed tuning with distributed training per trial.#

Learn more about the Tune library with the following talks and user guides.

Distributed Training#

The Ray Train library integrates many distributed training frameworks under a simple Trainer API, providing distributed orchestration and management capabilities out of the box.

In contrast to training many models, model parallelism partitions a large model across many machines for training. Ray Train has built-in abstractions for distributing shards of models and running training in parallel.


Model parallelism pattern for distributed large model training.#

Learn more about the Train library with the following talks and user guides.

Reinforcement Learning#

RLlib is an open-source library for reinforcement learning (RL), offering support for production-level, highly distributed RL workloads while maintaining unified and simple APIs for a large variety of industry applications. RLlib is used by industry leaders in many different verticals, such as climate control, industrial control, manufacturing and logistics, finance, gaming, automobile, robotics, boat design, and many others.


Decentralized distributed proximal polixy optimiation (DD-PPO) architecture.#

Learn more about reinforcement learning with the following resources.

ML Platform#

Merlin is Shopify’s ML platform built on Ray. It enables fast-iteration and scaling of distributed applications such as product categorization and recommendations.


Shopify’s Merlin architecture built on Ray.#

Spotify uses Ray for advanced applications that include personalizing content recommendations for home podcasts, and personalizing Spotify Radio track sequencing.


How Ray ecosystem empowers ML scientists and engineers at Spotify.#

The following highlights feature companies leveraging Ray’s unified API to build simpler, more flexible ML platforms.

End-to-End ML Workflows#

The following highlights examples utilizing Ray AIR to implement end-to-end ML workflows.

Large Scale Workload Orchestration#

The following highlights feature projects leveraging Ray Core’s distributed APIs to simplify the orchestration of large scale workloads.