RLlib: Industry-grade, scalable reinforcement learning#

RLlib is an open source library for reinforcement learning (RL). It supports production-grade, scalable, fault-tolerant RL workloads and keeps simple, unified APIs across a wide range of industry applications.

Whether you train policies in a multi-agent setup, from historic offline data, or with externally connected simulators, RLlib covers each of these autonomous decision-making cases, so you can start running experiments quickly.

Industry leaders use RLlib in production in many different verticals, such as gaming, robotics, finance, climate and industrial control, manufacturing and logistics, automobile, and boat design.

RLlib in 60 seconds#

A few steps get your first RLlib workload running on your laptop. Install RLlib and PyTorch:

pip install "ray[rllib]" torch

Note

To run the Atari or MuJoCo examples, install these additional packages:

pip install "gymnasium[atari,accept-rom-license,mujoco]"

That’s all you need to start coding against RLlib. This example runs the PPO algorithm on the Taxi domain. First, create a config for the algorithm. The config defines the RL environment and any other settings the algorithm needs.

from ray.rllib.algorithms.ppo import PPOConfig
from ray.rllib.connectors.env_to_module import FlattenObservations

# Configure the algorithm.
config = (
    PPOConfig()
    .environment("Taxi-v3")
    .env_runners(
        num_env_runners=2,
        # Observations are discrete (ints) -> We need to flatten (one-hot) them.
        env_to_module_connector=lambda env: FlattenObservations(),
    )
    .evaluation(evaluation_num_env_runners=1)
)

Next, build the algorithm and train it for two iterations. One training iteration includes parallel, distributed sample collection by the EnvRunner actors, followed by loss calculation on the collected data, and a model update step.

from pprint import pprint

# Build the algorithm.
algo = config.build_algo()

# Train it for 2 iterations ...
for _ in range(2):
    pprint(algo.train())

At the end of your script, evaluate the trained algorithm and release its resources:

# ... and evaluate it.
pprint(algo.evaluate())

# Release the algo's resources (remote actors, like EnvRunners and Learners).
algo.stop()

You can use any Farama-Foundation Gymnasium registered environment with the env argument.

In config.env_runners(), you can specify the number of parallel EnvRunner actors that collect samples from the environment, among many other settings.

You can also change the neural network architecture with RLlib’s DefaultModelConfig, and set up a separate config for the evaluation EnvRunner actors through the config.evaluation() method.

To learn more about the RLlib training APIs, see the RLlib Python API. For an example of an action inference loop after training, see this example script.

For a quick preview of which algorithms and environments RLlib supports, expand the dropdowns below.

Why choose RLlib?#

Learn more#

RLlib Key Concepts

Learn the core concepts of RLlib, such as algorithms, environments, models, and learners.

Key Concepts

RL Environments

Get started with environments RLlib supports, such as the Farama Foundation’s Gymnasium, PettingZoo, and custom formats for vectorized and multi-agent environments.

Environments

Models (RLModule)

Learn how to configure RLlib’s default models and implement your own custom models through the RLModule APIs, which support arbitrary architectures with PyTorch, complex multi-model setups, and multi-agent models with components shared between agents.

Models (RLModule)

Algorithms

See the RL algorithms RLlib provides for on-policy and off-policy training, offline and model-based RL, multi-agent RL, and more.

Algorithms

Customize RLlib#

RLlib provides APIs for customizing every part of your experimental and production training workflows. For example, you can code your own environments in Python with the Farama Foundation’s Gymnasium or DeepMind’s OpenSpiel, provide custom PyTorch models, write your own optimizer setups and loss definitions, or define custom exploratory behavior.

../_images/rllib-new-api-stack-simple.svg — **RLlib’s API stack:** Built on Ray, RLlib provides off-the-shelf, distributed, fault-tolerant algorithms and loss functions, PyTorch default models, multi-GPU training, and multi-agent support. You customize your experiments by subclassing the existing abstractions.#

Cite RLlib#

If RLlib helps with your academic research, the Ray RLlib team encourages you to cite these papers:

@inproceedings{liang2021rllib,
    title={{RLlib} Flow: Distributed Reinforcement Learning is a Dataflow Problem},
    author={
        Wu, Zhanghao and
        Liang, Eric and
        Luo, Michael and
        Mika, Sven and
        Gonzalez, Joseph E. and
        Stoica, Ion
    },
    booktitle={Conference on Neural Information Processing Systems ({NeurIPS})},
    year={2021},
    url={https://proceedings.neurips.cc/paper/2021/file/2bce32ed409f5ebcee2a7b417ad9beed-Paper.pdf}
}

@inproceedings{liang2018rllib,
    title={{RLlib}: Abstractions for Distributed Reinforcement Learning},
    author={
        Eric Liang and
        Richard Liaw and
        Robert Nishihara and
        Philipp Moritz and
        Roy Fox and
        Ken Goldberg and
        Joseph E. Gonzalez and
        Michael I. Jordan and
        Ion Stoica,
    },
    booktitle = {International Conference on Machine Learning ({ICML})},
    year={2018},
    url={https://arxiv.org/pdf/1712.09381}
}

On-Policy
PPO (Proximal Policy Optimization)
Off-Policy
SAC (Soft Actor Critic)
DQN/Rainbow (Deep Q Networks)
High-throughput Architectures
APPO (Asynchronous Proximal Policy Optimization)
IMPALA (Importance Weighted Actor-Learner Architecture)
Model-based RL
DreamerV3
Offline RL and Imitation Learning
BC (Behavior Cloning)
CQL (Conservative Q-Learning)
MARWIL (Advantage Re-Weighted Imitation Learning)