.rst .pdf

repository open issue suggest edit

Contents

RLlib Table of Contents

Training APIs

• Command-line

  • Evaluating Trained Policies

• Configuration

  • Specifying Parameters

  • Specifying Resources

  • Common Parameters

  • Scaling Guide

  • Tuned Examples

• Basic Python API

  • Computing Actions

  • Accessing Policy State

  • Accessing Model State

• Advanced Python APIs

  • Custom Training Workflows

  • Global Coordination

  • Callbacks and Custom Metrics

  • Customizing Exploration Behavior

  • Customized Evaluation During Training

  • Rewriting Trajectories

  • Curriculum Learning

• Debugging

  • Gym Monitor

  • Eager Mode

  • Episode Traces

  • Log Verbosity

  • Stack Traces

• External Application API

Environments

• RLlib Environments Overview

• OpenAI Gym

• Vectorized

• Multi-Agent and Hierarchical

• External Agents and Applications

  • External Application Clients

• Advanced Integrations

Models, Preprocessors, and Action Distributions

• RLlib Models, Preprocessors, and Action Distributions Overview

• TensorFlow Models

• PyTorch Models

• Custom Preprocessors

• Custom Action Distributions

• Supervised Model Losses

• Self-Supervised Model Losses

• Variable-length / Complex Observation Spaces

• Variable-length / Parametric Action Spaces

• Autoregressive Action Distributions

Algorithms

• High-throughput architectures

  • Distributed Prioritized Experience Replay (Ape-X)

  • Importance Weighted Actor-Learner Architecture (IMPALA)

  • Asynchronous Proximal Policy Optimization (APPO)

  • Decentralized Distributed Proximal Policy Optimization (DD-PPO)

• Gradient-based

  • Advantage Actor-Critic (A2C, A3C)

  • Deep Deterministic Policy Gradients (DDPG, TD3)

  • Deep Q Networks (DQN, Rainbow, Parametric DQN)

  • Policy Gradients

  • Proximal Policy Optimization (PPO)

  • Soft Actor Critic (SAC)

  • Slate Q-Learning (SlateQ)

• Derivative-free

  • Augmented Random Search (ARS)

  • Evolution Strategies

• Model-based / Meta-learning

  • Single-Player AlphaZero (contrib/AlphaZero)

  • Model-Agnostic Meta-Learning (MAML)

  • Model-Based Meta-Policy-Optimization (MBMPO)

  • Dreamer (DREAMER)

• Multi-agent

  • QMIX Monotonic Value Factorisation (QMIX, VDN, IQN)

  • Multi-Agent Deep Deterministic Policy Gradient (contrib/MADDPG)

• Offline

  • Advantage Re-Weighted Imitation Learning (MARWIL)

• Contextual bandits

  • Linear Upper Confidence Bound (contrib/LinUCB)

  • Linear Thompson Sampling (contrib/LinTS)

• Exploration-based plug-ins (can be combined with any algo)

  • Curiosity (ICM: Intrinsic Curiosity Module)

Sample Collection

• The SampleCollector Class is Used to Store and Retrieve Temporary Data

• Trajectory View API

Offline Datasets

• Working with Offline Datasets

• Input Pipeline for Supervised Losses

• Input API

• Output API

Concepts and Custom Algorithms

• Policies

  • Policies in Multi-Agent

  • Building Policies in TensorFlow

  • Building Policies in TensorFlow Eager

  • Building Policies in PyTorch

  • Extending Existing Policies

• Policy Evaluation

• Execution Plans

• Trainers

Examples

• Tuned Examples

• Training Workflows

• Custom Envs and Models

• Serving and Offline

• Multi-Agent and Hierarchical

• Community Examples

Development

• Development Install

• API Stability

• Features

• Benchmarks

• Contributing Algorithms

Package Reference

• ray.rllib.agents

• ray.rllib.env

• ray.rllib.evaluation

• ray.rllib.execution

• ray.rllib.models

• ray.rllib.utils

Troubleshooting

If you encounter errors like blas_thread_init: pthread_create: Resource temporarily unavailable when using many workers, try setting OMP_NUM_THREADS=1. Similarly, check configured system limits with ulimit -a for other resource limit errors.

For debugging unexpected hangs or performance problems, you can run ray stack to dump the stack traces of all Ray workers on the current node, ray timeline to dump a timeline visualization of tasks to a file, and ray memory to list all object references in the cluster.

TensorFlow 2.0

RLlib currently runs in tf.compat.v1 mode. This means eager execution is disabled by default, and RLlib imports TF with import tensorflow.compat.v1 as tf; tf.disable_v2_behaviour(). Eager execution can be enabled manually by calling tf.enable_eager_execution() or setting the "eager": True trainer config.

RLlib: Scalable Reinforcement Learning RLlib Training APIs