Algorithms

Tip

Check out the environments page to learn more about different environment types.

Available Algorithms - Overview

Algorithm

Frameworks

Discrete Actions

Continuous Actions

Multi-Agent

Model Support

Multi-GPU

A2C

tf + torch

Yes +parametric

Yes

Yes

+RNN, +LSTM auto-wrapping, +Attention, +autoreg

A2C: tf + torch

A3C

tf + torch

Yes +parametric

Yes

Yes

+RNN, +LSTM auto-wrapping, +Attention, +autoreg

No

AlphaZero

torch

Yes +parametric

No

No

No

APPO

tf + torch

Yes +parametric

Yes

Yes

+RNN, +LSTM auto-wrapping, +Attention, +autoreg

tf + torch

ARS

tf + torch

Yes

Yes

No

No

Bandits (TS & LinUCB)

torch

Yes +parametric

No

Yes

No

BC

tf + torch

Yes +parametric

Yes

Yes

+RNN

torch

CQL

tf + torch

No

Yes

No

tf + torch

CRR

torch

Yes +parametric

Yes

Yes

torch

DDPG

tf + torch

No

Yes

Yes

torch

APEX-DDPG

tf + torch

No

Yes

Yes

torch

ES

tf + torch

Yes

Yes

No

No

Dreamer

torch

No

Yes

No

+RNN

torch

DQN, Rainbow

tf + torch

Yes +parametric

No

Yes

tf + torch

APEX-DQN

tf + torch

Yes +parametric

No

Yes

torch

IMPALA

tf + torch

Yes +parametric

Yes

Yes

+RNN, +LSTM auto-wrapping, +Attention, +autoreg

tf + torch

MAML

tf + torch

No

Yes

No

torch

MARWIL

tf + torch

Yes +parametric

Yes

Yes

+RNN

torch

MBMPO

torch

No

Yes

No

torch

PG

tf + torch

Yes +parametric

Yes

Yes

+RNN, +LSTM auto-wrapping, +Attention, +autoreg

tf + torch

PPO

tf + torch

Yes +parametric

Yes

Yes

+RNN, +LSTM auto-wrapping, +Attention, +autoreg

tf + torch

R2D2

tf + torch

Yes +parametric

No

Yes

+RNN, +LSTM auto-wrapping, +autoreg

torch

SAC

tf + torch

Yes

Yes

Yes

torch

SlateQ

tf + torch

Yes (multi-discr. slates)

No

No

torch

TD3

tf + torch

No

Yes

Yes

torch

Multi-Agent only Methods

Algorithm

Frameworks

Discrete Actions

Continuous Actions

Multi-Agent

Model Support

QMIX

torch

Yes +parametric

No

Yes

+RNN

MADDPG

tf

Yes

Partial

Yes

Parameter Sharing

Depends on bootstrapped algorithm

Fully Independent Learning

Depends on bootstrapped algorithm

Shared Critic Methods

Depends on bootstrapped algorithm

Exploration-based plug-ins (can be combined with any algo)

Algorithm

Frameworks

Discrete Actions

Continuous Actions

Multi-Agent

Model Support

Curiosity

tf + torch

Yes +parametric

No

Yes

+RNN

Offline

Behavior Cloning (BC; derived from MARWIL implementation)

pytorch tensorflow [paper] [implementation]

Our behavioral cloning implementation is directly derived from our MARWIL implementation, with the only difference being the beta parameter force-set to 0.0. This makes BC try to match the behavior policy, which generated the offline data, disregarding any resulting rewards. BC requires the offline datasets API to be used.

Tuned examples: CartPole-v1

BC-specific configs (see also common configs):

class ray.rllib.algorithms.bc.bc.BCConfig(algo_class=None)[source]

Defines a configuration class from which a new BC Trainer can be built

Example

>>> from ray.rllib.algorithms.bc import BCConfig
>>> # Run this from the ray directory root.
>>> config = BCConfig().training(lr=0.00001, gamma=0.99)
>>> config = config.offline_data(  
...     input_="./rllib/tests/data/cartpole/large.json")
>>> print(config.to_dict())  
>>> # Build a Trainer object from the config and run 1 training iteration.
>>> algo = config.build()  
>>> algo.train()  

Example

>>> from ray.rllib.algorithms.bc import BCConfig
>>> from ray import tune
>>> config = BCConfig()
>>> # Print out some default values.
>>> print(config.beta)  
>>> # Update the config object.
>>> config.training(  
...     lr=tune.grid_search([0.001, 0.0001]), beta=0.75
... )
>>> # Set the config object's data path.
>>> # Run this from the ray directory root.
>>> config.offline_data(  
...     input_="./rllib/tests/data/cartpole/large.json"
... )
>>> # Set the config object's env, used for evaluation.
>>> config.environment(env="CartPole-v1")  
>>> # Use to_dict() to get the old-style python config dict
>>> # when running with tune.
>>> tune.Tuner(   
...     "BC",
...     param_space=config.to_dict(),
... ).fit()
training(*, beta: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, bc_logstd_coeff: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, moving_average_sqd_adv_norm_update_rate: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, moving_average_sqd_adv_norm_start: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, use_gae: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, vf_coeff: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, grad_clip: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, **kwargs) ray.rllib.algorithms.marwil.marwil.MARWILConfig

Sets the training related configuration.

Parameters
  • beta – Scaling of advantages in exponential terms. When beta is 0.0, MARWIL is reduced to behavior cloning (imitation learning); see bc.py algorithm in this same directory.

  • bc_logstd_coeff – A coefficient to encourage higher action distribution entropy for exploration.

  • moving_average_sqd_adv_norm_start – Starting value for the squared moving average advantage norm (c^2).

  • use_gae – If True, use the Generalized Advantage Estimator (GAE) with a value function, see https://arxiv.org/pdf/1506.02438.pdf in case an input line ends with a non-terminal timestep.

  • vf_coeff – Balancing value estimation loss and policy optimization loss. moving_average_sqd_adv_norm_update_rate: Update rate for the squared moving average advantage norm (c^2).

  • grad_clip – If specified, clip the global norm of gradients by this amount.

Returns

This updated AlgorithmConfig object.

Critic Regularized Regression (CRR)

pytorch [paper] [implementation]

CRR is another offline RL algorithm based on Q-learning that can learn from an offline experience replay. The challenge in applying existing Q-learning algorithms to offline RL lies in the overestimation of the Q-function, as well as, the lack of exploration beyond the observed data. The latter becomes increasingly important during bootstrapping in the bellman equation, where the Q-function queried for the next state’s Q-value(s) does not have support in the observed data. To mitigate these issues, CRR implements a simple and yet powerful idea of “value-filtered regression”. The key idea is to use a learned critic to filter-out the non-promising transitions from the replay dataset. For more details, please refer to the paper (see link above).

Tuned examples: CartPole-v1, Pendulum-v1

class ray.rllib.algorithms.crr.crr.CRRConfig(algo_class=None)[source]
training(*, weight_type: Optional[str] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, temperature: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, max_weight: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, advantage_type: Optional[str] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, n_action_sample: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, twin_q: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, target_network_update_freq: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, actor_hiddens: Optional[List[int]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, actor_hidden_activation: Optional[str] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, critic_hiddens: Optional[List[int]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, critic_hidden_activation: Optional[str] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, tau: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, td_error_loss_fn: Optional[str] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, categorical_distribution_temperature: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, **kwargs) ray.rllib.algorithms.crr.crr.CRRConfig[source]

CRR training configuration

Parameters
  • weight_type – weight type to use bin | exp.

  • temperature – the exponent temperature used in exp weight type.

  • max_weight – the max weight limit for exp weight type.

  • advantage_type

    The way we reduce q values to v_t values max | mean | expectation. max and mean work for both discrete and continuous action spaces while expectation only works for discrete action spaces. max: Uses max over sampled actions to estimate the value.

    \[A(s_t, a_t) = Q(s_t, a_t) - \max_{a^j} Q(s_t, a^j)\]

    where \(a^j\) is n_action_sample times sampled from the policy \(\pi(a | s_t)\) mean: Uses mean over sampled actions to estimate the value.

    \[A(s_t, a_t) = Q(s_t, a_t) - \frac{1}{m}\sum_{j=1}^{m} [Q(s_t, a^j)]\]

    where \(a^j\) is n_action_sample times sampled from the policy \(\pi(a | s_t)\) expectation: This uses categorical distribution to evaluate the expectation of the q values directly to estimate the value.

    \[A(s_t, a_t) = Q(s_t, a_t) - E_{a^j\sim \pi(a|s_t)}[Q(s_t,a^j)]\]

  • n_action_sample – the number of actions to sample for v_t estimation.

  • twin_q – if True, uses pessimistic q estimation.

  • target_network_update_freq – The frequency at which we update the target copy of the model in terms of the number of gradient updates applied to the main model.

  • actor_hiddens – The number of hidden units in the actor’s fc network.

  • actor_hidden_activation – The activation used in the actor’s fc network.

  • critic_hiddens – The number of hidden units in the critic’s fc network.

  • critic_hidden_activation – The activation used in the critic’s fc network.

  • tau – Polyak averaging coefficient (making it 1 is reduces it to a hard update).

  • td_error_loss_fn – “huber” or “mse”. Loss function for calculating critic error.

  • categorical_distribution_temperature – Set the temperature parameter used by Categorical action distribution. A valid temperature is in the range of [0, 1]. Note that this mostly affects evaluation since critic error uses selected action for return calculation.

  • **kwargs – forward compatibility kwargs

Returns

This updated CRRConfig object.

Conservative Q-Learning (CQL)

pytorch tensorflow [paper] [implementation]

In offline RL, the algorithm has no access to an environment, but can only sample from a fixed dataset of pre-collected state-action-reward tuples. In particular, CQL (Conservative Q-Learning) is an offline RL algorithm that mitigates the overestimation of Q-values outside the dataset distribution via conservative critic estimates. It does so by adding a simple Q regularizer loss to the standard Bellman update loss. This ensures that the critic does not output overly-optimistic Q-values. This conservative correction term can be added on top of any off-policy Q-learning algorithm (here, we provide this for SAC).

RLlib’s CQL is evaluated against the Behavior Cloning (BC) benchmark at 500K gradient steps over the dataset. The only difference between the BC- and CQL configs is the bc_iters parameter in CQL, indicating how many gradient steps we perform over the BC loss. CQL is evaluated on the D4RL benchmark, which has pre-collected offline datasets for many types of environments.

Tuned examples: HalfCheetah Random, Hopper Random

CQL-specific configs (see also common configs):

class ray.rllib.algorithms.cql.cql.CQLConfig(algo_class=None)[source]

Defines a configuration class from which a CQL Trainer can be built.

Example

>>> from ray.rllib.algorithms.cql import CQLConfig
>>> config = CQLConfig().training(gamma=0.9, lr=0.01)
>>> config = config.resources(num_gpus=0)
>>> config = config.rollouts(num_rollout_workers=4)
>>> print(config.to_dict())  
>>> # Build a Trainer object from the config and run 1 training iteration.
>>> algo = config.build(env="CartPole-v1")  
>>> algo.train()  
training(*, bc_iters: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, temperature: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, num_actions: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, lagrangian: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, lagrangian_thresh: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, min_q_weight: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, **kwargs) ray.rllib.algorithms.cql.cql.CQLConfig[source]

Sets the training-related configuration.

Parameters
  • bc_iters – Number of iterations with Behavior Cloning pretraining.

  • temperature – CQL loss temperature.

  • num_actions – Number of actions to sample for CQL loss

  • lagrangian – Whether to use the Lagrangian for Alpha Prime (in CQL loss).

  • lagrangian_thresh – Lagrangian threshold.

  • min_q_weight – in Q weight multiplier.

Returns

This updated AlgorithmConfig object.

Monotonic Advantage Re-Weighted Imitation Learning (MARWIL)

pytorch tensorflow [paper] [implementation]

MARWIL is a hybrid imitation learning and policy gradient algorithm suitable for training on batched historical data. When the beta hyperparameter is set to zero, the MARWIL objective reduces to vanilla imitation learning (see BC). MARWIL requires the offline datasets API to be used.

Tuned examples: CartPole-v1

MARWIL-specific configs (see also common configs):

class ray.rllib.algorithms.marwil.marwil.MARWILConfig(algo_class=None)[source]

Defines a configuration class from which a MARWIL Algorithm can be built.

Example

>>> from ray.rllib.algorithms.marwil import MARWILConfig
>>> # Run this from the ray directory root.
>>> config = MARWILConfig()  
>>> config = config.training(beta=1.0, lr=0.00001, gamma=0.99)  
>>> config = config.offline_data(  
...     input_=["./rllib/tests/data/cartpole/large.json"])
>>> print(config.to_dict()) 
...
>>> # Build an Algorithm object from the config and run 1 training iteration.
>>> algo = config.build()  
>>> algo.train() 

Example

>>> from ray.rllib.algorithms.marwil import MARWILConfig
>>> from ray import tune
>>> config = MARWILConfig()
>>> # Print out some default values.
>>> print(config.beta)  
>>> # Update the config object.
>>> config.training(lr=tune.grid_search(  
...     [0.001, 0.0001]), beta=0.75)
>>> # Set the config object's data path.
>>> # Run this from the ray directory root.
>>> config.offline_data( 
...     input_=["./rllib/tests/data/cartpole/large.json"])
>>> # Set the config object's env, used for evaluation.
>>> config.environment(env="CartPole-v1")  
>>> # Use to_dict() to get the old-style python config dict
>>> # when running with tune.
>>> tune.Tuner(  
...     "MARWIL",
...     param_space=config.to_dict(),
... ).fit()
training(*, beta: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, bc_logstd_coeff: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, moving_average_sqd_adv_norm_update_rate: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, moving_average_sqd_adv_norm_start: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, use_gae: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, vf_coeff: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, grad_clip: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, **kwargs) ray.rllib.algorithms.marwil.marwil.MARWILConfig[source]

Sets the training related configuration.

Parameters
  • beta – Scaling of advantages in exponential terms. When beta is 0.0, MARWIL is reduced to behavior cloning (imitation learning); see bc.py algorithm in this same directory.

  • bc_logstd_coeff – A coefficient to encourage higher action distribution entropy for exploration.

  • moving_average_sqd_adv_norm_start – Starting value for the squared moving average advantage norm (c^2).

  • use_gae – If True, use the Generalized Advantage Estimator (GAE) with a value function, see https://arxiv.org/pdf/1506.02438.pdf in case an input line ends with a non-terminal timestep.

  • vf_coeff – Balancing value estimation loss and policy optimization loss. moving_average_sqd_adv_norm_update_rate: Update rate for the squared moving average advantage norm (c^2).

  • grad_clip – If specified, clip the global norm of gradients by this amount.

Returns

This updated AlgorithmConfig object.

Model-free On-policy RL

Asynchronous Proximal Policy Optimization (APPO)

pytorch tensorflow [paper] [implementation] We include an asynchronous variant of Proximal Policy Optimization (PPO) based on the IMPALA architecture. This is similar to IMPALA but using a surrogate policy loss with clipping. Compared to synchronous PPO, APPO is more efficient in wall-clock time due to its use of asynchronous sampling. Using a clipped loss also allows for multiple SGD passes, and therefore the potential for better sample efficiency compared to IMPALA. V-trace can also be enabled to correct for off-policy samples.

Tip

APPO is not always more efficient; it is often better to use standard PPO or IMPALA.

../_images/impala-arch.svg

APPO architecture (same as IMPALA)

Tuned examples: PongNoFrameskip-v4

APPO-specific configs (see also common configs):

class ray.rllib.algorithms.appo.appo.APPOConfig(algo_class=None)[source]

Defines a configuration class from which an APPO Algorithm can be built.

Example

>>> from ray.rllib.algorithms.appo import APPOConfig
>>> config = APPOConfig().training(lr=0.01, grad_clip=30.0)
>>> config = config.resources(num_gpus=1)
>>> config = config.rollouts(num_rollout_workers=16)
>>> config = config.environment("CartPole-v1")
>>> print(config.to_dict())  
>>> # Build an Algorithm object from the config and run 1 training iteration.
>>> algo = config.build()  
>>> algo.train()  

Example

>>> from ray.rllib.algorithms.appo import APPOConfig
>>> from ray import air
>>> from ray import tune
>>> config = APPOConfig()
>>> # Print out some default values.
>>> print(config.sample_async)   
>>> # Update the config object.
>>> config = config.training(lr=tune.grid_search([0.001, 0.0001]))
>>> # Set the config object's env.
>>> config = config.environment(env="CartPole-v1")
>>> # Use to_dict() to get the old-style python config dict
>>> # when running with tune.
>>> tune.Tuner(  
...     "APPO",
...     run_config=air.RunConfig(stop={"episode_reward_mean": 200}),
...     param_space=config.to_dict(),
... ).fit()
training(*, vtrace: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, use_critic: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, use_gae: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, lambda_: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, clip_param: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, use_kl_loss: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, kl_coeff: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, kl_target: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, **kwargs) ray.rllib.algorithms.appo.appo.APPOConfig[source]

Sets the training related configuration.

Parameters
  • vtrace – Whether to use V-trace weighted advantages. If false, PPO GAE advantages will be used instead.

  • use_critic – Should use a critic as a baseline (otherwise don’t use value baseline; required for using GAE). Only applies if vtrace=False.

  • use_gae – If true, use the Generalized Advantage Estimator (GAE) with a value function, see https://arxiv.org/pdf/1506.02438.pdf. Only applies if vtrace=False.

  • lambda – GAE (lambda) parameter.

  • clip_param – PPO surrogate slipping parameter.

  • use_kl_loss – Whether to use the KL-term in the loss function.

  • kl_coeff – Coefficient for weighting the KL-loss term.

  • kl_target – Target term for the KL-term to reach (via adjusting the kl_coeff automatically).

Returns

This updated AlgorithmConfig object.

Decentralized Distributed Proximal Policy Optimization (DD-PPO)

pytorch [paper] [implementation] Unlike APPO or PPO, with DD-PPO policy improvement is no longer done centralized in the algorithm process. Instead, gradients are computed remotely on each rollout worker and all-reduced at each mini-batch using torch distributed. This allows each worker’s GPU to be used both for sampling and for training.

Tip

DD-PPO is best for envs that require GPUs to function, or if you need to scale out SGD to multiple nodes. If you don’t meet these requirements, standard PPO will be more efficient.

../_images/ddppo-arch.svg

DD-PPO architecture (both sampling and learning are done on worker GPUs)

Tuned examples: CartPole-v1, BreakoutNoFrameskip-v4

DDPPO-specific configs (see also common configs):

class ray.rllib.algorithms.ddppo.ddppo.DDPPOConfig(algo_class=None)[source]

Defines a configuration class from which a DDPPO Algorithm can be built.

Note(jungong) : despite best efforts, DDPPO does not use fault tolerant and elastic features of WorkerSet, because of the way Torch DDP is set up.

Example

>>> from ray.rllib.algorithms.ddppo import DDPPOConfig
>>> config = DDPPOConfig().training(lr=0.003, keep_local_weights_in_sync=True)
>>> config = config.resources(num_gpus=1)
>>> config = config.rollouts(num_rollout_workers=10)
>>> print(config.to_dict())   
>>> # Build a Algorithm object from the config and run 1 training iteration.
>>> algo = config.build(env="CartPole-v1")  
>>> algo.train()  

Example

>>> from ray.rllib.algorithms.ddppo import DDPPOConfig
>>> from ray import air
>>> from ray import tune
>>> config = DDPPOConfig()
>>> # Print out some default values.
>>> print(config.kl_coeff)   
>>> # Update the config object.
>>> config.training( 
...     lr=tune.grid_search([0.001, 0.0001]), num_sgd_iter=15)
>>> # Set the config object's env.
>>> config.environment(env="CartPole-v1") 
>>> # Use to_dict() to get the old-style python config dict
>>> # when running with tune.
>>> tune.Tuner( 
...     "DDPPO",
...     run_config=air.RunConfig(stop={"episode_reward_mean": 200}),
...     param_space=config.to_dict(),
... ).fit()
training(*, keep_local_weights_in_sync: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, torch_distributed_backend: Optional[str] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, **kwargs) ray.rllib.algorithms.ddppo.ddppo.DDPPOConfig[source]

Sets the training related configuration.

Parameters
  • keep_local_weights_in_sync – Download weights between each training step. This adds a bit of overhead but allows the user to access the weights from the trainer.

  • torch_distributed_backend – The communication backend for PyTorch distributed.

Returns

This updated AlgorithmConfig object.

Proximal Policy Optimization (PPO)

pytorch tensorflow [paper] [implementation] PPO’s clipped objective supports multiple SGD passes over the same batch of experiences. RLlib’s multi-GPU optimizer pins that data in GPU memory to avoid unnecessary transfers from host memory, substantially improving performance over a naive implementation. PPO scales out using multiple workers for experience collection, and also to multiple GPUs for SGD.

Tip

If you need to scale out with GPUs on multiple nodes, consider using decentralized PPO.

../_images/ppo-arch.svg

PPO architecture

Tuned examples: Unity3D Soccer (multi-agent: Strikers vs Goalie), Humanoid-v1, Hopper-v1, Pendulum-v1, PongDeterministic-v4, Walker2d-v1, HalfCheetah-v2, {BeamRider,Breakout,Qbert,SpaceInvaders}NoFrameskip-v4

Atari results: more details

Atari env

RLlib PPO @10M

RLlib PPO @25M

Baselines PPO @10M

BeamRider

2807

4480

~1800

Breakout

104

201

~250

Qbert

11085

14247

~14000

SpaceInvaders

671

944

~800

Scalability: more details

MuJoCo env

RLlib PPO 16-workers @ 1h

Fan et al PPO 16-workers @ 1h

HalfCheetah

9664

~7700

../_images/ppo.png

RLlib’s multi-GPU PPO scales to multiple GPUs and hundreds of CPUs on solving the Humanoid-v1 task. Here we compare against a reference MPI-based implementation.

PPO-specific configs (see also common configs):

class ray.rllib.algorithms.ppo.ppo.PPOConfig(algo_class=None)[source]

Defines a configuration class from which a PPO Algorithm can be built.

Example

>>> from ray.rllib.algorithms.ppo import PPOConfig
>>> config = PPOConfig()  
>>> config = config.training(gamma=0.9, lr=0.01, kl_coeff=0.3)  
>>> config = config.resources(num_gpus=0)  
>>> config = config.rollouts(num_rollout_workers=4)  
>>> print(config.to_dict())  
>>> # Build a Algorithm object from the config and run 1 training iteration.
>>> algo = config.build(env="CartPole-v1")  
>>> algo.train()  

Example

>>> from ray.rllib.algorithms.ppo import PPOConfig
>>> from ray import air
>>> from ray import tune
>>> config = PPOConfig()
>>> # Print out some default values.
>>> print(config.clip_param)  
>>> # Update the config object.
>>> config.training(  
... lr=tune.grid_search([0.001, 0.0001]), clip_param=0.2
... )
>>> # Set the config object's env.
>>> config = config.environment(env="CartPole-v1")   
>>> # Use to_dict() to get the old-style python config dict
>>> # when running with tune.
>>> tune.Tuner(  
...     "PPO",
...     run_config=air.RunConfig(stop={"episode_reward_mean": 200}),
...     param_space=config.to_dict(),
... ).fit()
training(*, lr_schedule: Optional[List[List[Union[int, float]]]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, use_critic: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, use_gae: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, lambda_: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, kl_coeff: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, sgd_minibatch_size: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, num_sgd_iter: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, shuffle_sequences: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, vf_loss_coeff: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, entropy_coeff: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, entropy_coeff_schedule: Optional[List[List[Union[int, float]]]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, clip_param: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, vf_clip_param: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, grad_clip: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, kl_target: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, vf_share_layers=-1, **kwargs) ray.rllib.algorithms.ppo.ppo.PPOConfig[source]

Sets the training related configuration.

Parameters
  • lr_schedule – Learning rate schedule. In the format of [[timestep, lr-value], [timestep, lr-value], …] Intermediary timesteps will be assigned to interpolated learning rate values. A schedule should normally start from timestep 0.

  • use_critic – Should use a critic as a baseline (otherwise don’t use value baseline; required for using GAE).

  • use_gae – If true, use the Generalized Advantage Estimator (GAE) with a value function, see https://arxiv.org/pdf/1506.02438.pdf.

  • lambda – The GAE (lambda) parameter.

  • kl_coeff – Initial coefficient for KL divergence.

  • sgd_minibatch_size – Total SGD batch size across all devices for SGD. This defines the minibatch size within each epoch.

  • num_sgd_iter – Number of SGD iterations in each outer loop (i.e., number of epochs to execute per train batch).

  • shuffle_sequences – Whether to shuffle sequences in the batch when training (recommended).

  • vf_loss_coeff – Coefficient of the value function loss. IMPORTANT: you must tune this if you set vf_share_layers=True inside your model’s config.

  • entropy_coeff – Coefficient of the entropy regularizer.

  • entropy_coeff_schedule – Decay schedule for the entropy regularizer.

  • clip_param – PPO clip parameter.

  • vf_clip_param – Clip param for the value function. Note that this is sensitive to the scale of the rewards. If your expected V is large, increase this.

  • grad_clip – If specified, clip the global norm of gradients by this amount.

  • kl_target – Target value for KL divergence.

Returns

This updated AlgorithmConfig object.

Importance Weighted Actor-Learner Architecture (IMPALA)

pytorch tensorflow [paper] [implementation] In IMPALA, a central learner runs SGD in a tight loop while asynchronously pulling sample batches from many actor processes. RLlib’s IMPALA implementation uses DeepMind’s reference V-trace code. Note that we do not provide a deep residual network out of the box, but one can be plugged in as a custom model. Multiple learner GPUs and experience replay are also supported.

../_images/impala-arch.svg

IMPALA architecture

Tuned examples: PongNoFrameskip-v4, vectorized configuration, multi-gpu configuration, {BeamRider,Breakout,Qbert,SpaceInvaders}NoFrameskip-v4

Atari results @10M steps: more details

Atari env

RLlib IMPALA 32-workers

Mnih et al A3C 16-workers

BeamRider

2071

~3000

Breakout

385

~150

Qbert

4068

~1000

SpaceInvaders

719

~600

Scalability:

Atari env

RLlib IMPALA 32-workers @1 hour

Mnih et al A3C 16-workers @1 hour

BeamRider

3181

~1000

Breakout

538

~10

Qbert

10850

~500

SpaceInvaders

843

~300

../_images/impala.png

Multi-GPU IMPALA scales up to solve PongNoFrameskip-v4 in ~3 minutes using a pair of V100 GPUs and 128 CPU workers. The maximum training throughput reached is ~30k transitions per second (~120k environment frames per second).

IMPALA-specific configs (see also common configs):

class ray.rllib.algorithms.impala.impala.ImpalaConfig(algo_class=None)[source]

Defines a configuration class from which an Impala can be built.

Example

>>> from ray.rllib.algorithms.impala import ImpalaConfig
>>> config = ImpalaConfig()
>>> config = config.training(lr=0.0003, train_batch_size=512)  
>>> config = config.resources(num_gpus=4)  
>>> config = config.rollouts(num_rollout_workers=64)  
>>> print(config.to_dict())  
>>> # Build a Algorithm object from the config and run 1 training iteration.
>>> algo = config.build(env="CartPole-v1")  
>>> algo.train()  

Example

>>> from ray.rllib.algorithms.impala import ImpalaConfig
>>> from ray import air
>>> from ray import tune
>>> config = ImpalaConfig()
>>> # Print out some default values.
>>> print(config.vtrace)  
>>> # Update the config object.
>>> config = config.training(   
...     lr=tune.grid_search([0.0001, 0.0003]), grad_clip=20.0
... )
>>> # Set the config object's env.
>>> config = config.environment(env="CartPole-v1")  
>>> # Use to_dict() to get the old-style python config dict
>>> # when running with tune.
>>> tune.Tuner(  
...     "IMPALA",
...     run_config=air.RunConfig(stop={"episode_reward_mean": 200}),
...     param_space=config.to_dict(),
... ).fit()
training(*, vtrace: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, vtrace_clip_rho_threshold: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, vtrace_clip_pg_rho_threshold: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, vtrace_drop_last_ts: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, num_multi_gpu_tower_stacks: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, minibatch_buffer_size: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, num_sgd_iter: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, replay_proportion: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, replay_buffer_num_slots: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, learner_queue_size: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, learner_queue_timeout: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, max_requests_in_flight_per_aggregator_worker: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, timeout_s_sampler_manager: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, timeout_s_aggregator_manager: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, broadcast_interval: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, num_aggregation_workers: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, grad_clip: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, opt_type: Optional[str] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, lr_schedule: Optional[List[List[Union[int, float]]]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, decay: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, momentum: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, epsilon: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, vf_loss_coeff: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, entropy_coeff: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, entropy_coeff_schedule: Optional[List[List[Union[int, float]]]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, _separate_vf_optimizer: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, _lr_vf: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, after_train_step: Optional[Callable[[dict], None]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, **kwargs) ray.rllib.algorithms.impala.impala.ImpalaConfig[source]

Sets the training related configuration.

Parameters
  • vtrace – V-trace params (see vtrace_tf/torch.py).

  • vtrace_clip_rho_threshold

  • vtrace_clip_pg_rho_threshold

  • vtrace_drop_last_ts – If True, drop the last timestep for the vtrace calculations, such that all data goes into the calculations as [B x T-1] (+ the bootstrap value). This is the default and legacy RLlib behavior, however, could potentially have a destabilizing effect on learning, especially in sparse reward or reward-at-goal environments. False for not dropping the last timestep. System params.

  • num_multi_gpu_tower_stacks – For each stack of multi-GPU towers, how many slots should we reserve for parallel data loading? Set this to >1 to load data into GPUs in parallel. This will increase GPU memory usage proportionally with the number of stacks. Example: 2 GPUs and num_multi_gpu_tower_stacks=3: - One tower stack consists of 2 GPUs, each with a copy of the model/graph. - Each of the stacks will create 3 slots for batch data on each of its GPUs, increasing memory requirements on each GPU by 3x. - This enables us to preload data into these stacks while another stack is performing gradient calculations.

  • minibatch_buffer_size – How many train batches should be retained for minibatching. This conf only has an effect if num_sgd_iter > 1.

  • num_sgd_iter – Number of passes to make over each train batch.

  • replay_proportion – Set >0 to enable experience replay. Saved samples will be replayed with a p:1 proportion to new data samples.

  • replay_buffer_num_slots – Number of sample batches to store for replay. The number of transitions saved total will be (replay_buffer_num_slots * rollout_fragment_length).

  • learner_queue_size – Max queue size for train batches feeding into the learner.

  • learner_queue_timeout – Wait for train batches to be available in minibatch buffer queue this many seconds. This may need to be increased e.g. when training with a slow environment.

  • max_requests_in_flight_per_aggregator_worker – Level of queuing for replay aggregator operations (if using aggregator workers).

  • timeout_s_sampler_manager – The timeout for waiting for sampling results for workers – typically if this is too low, the manager won’t be able to retrieve ready sampling results.

  • timeout_s_aggregator_manager – The timeout for waiting for replay worker results – typically if this is too low, the manager won’t be able to retrieve ready replay requests.

  • broadcast_interval – Number of training step calls before weights are broadcasted to rollout workers that are sampled during any iteration.

  • num_aggregation_workers – Use n (num_aggregation_workers) extra Actors for multi-level aggregation of the data produced by the m RolloutWorkers (num_workers). Note that n should be much smaller than m. This can make sense if ingesting >2GB/s of samples, or if the data requires decompression.

  • grad_clip – If specified, clip the global norm of gradients by this amount.

  • opt_type – Either “adam” or “rmsprop”.

  • lr_schedule – Learning rate schedule. In the format of [[timestep, lr-value], [timestep, lr-value], …] Intermediary timesteps will be assigned to interpolated learning rate values. A schedule should normally start from timestep 0.

  • decay – Decay setting for the RMSProp optimizer, in case opt_type=rmsprop.

  • momentum – Momentum setting for the RMSProp optimizer, in case opt_type=rmsprop.

  • epsilon – Epsilon setting for the RMSProp optimizer, in case opt_type=rmsprop.

  • vf_loss_coeff – Coefficient for the value function term in the loss function.

  • entropy_coeff – Coefficient for the entropy regularizer term in the loss function.

  • entropy_coeff_schedule – Decay schedule for the entropy regularizer.

  • _separate_vf_optimizer – Set this to true to have two separate optimizers optimize the policy-and value networks.

  • _lr_vf – If _separate_vf_optimizer is True, define separate learning rate for the value network.

  • after_train_step – Callback for APPO to use to update KL, target network periodically. The input to the callback is the learner fetches dict.

Returns

This updated AlgorithmConfig object.

Advantage Actor-Critic (A2C)

pytorch tensorflow [paper] [implementation] A2C scales to 16-32+ worker processes depending on the environment and supports microbatching (i.e., gradient accumulation), which can be enabled by setting the microbatch_size config. Microbatching allows for training with a train_batch_size much larger than GPU memory.

../_images/a2c-arch.svg

A2C architecture

Tuned examples: Atari environments

Tip

Consider using IMPALA for faster training with similar timestep efficiency.

Atari results @10M steps: more details

Atari env

RLlib A2C 5-workers

Mnih et al A3C 16-workers

BeamRider

1401

~3000

Breakout

374

~150

Qbert

3620

~1000

SpaceInvaders

692

~600

A2C-specific configs (see also common configs):

class ray.rllib.algorithms.a2c.a2c.A2CConfig[source]

Defines a configuration class from which a new A2C can be built.

Example

>>> from ray import tune
>>> from ray.rllib.algorithms.a2c import A2CConfig
>>> config = A2CConfig()
>>> config = config.training(lr=0.01, grad_clip=30.0)  
>>> config = config.resources(num_gpus=0)  
>>> config = config.rollouts(num_rollout_workers=2)  
>>> config = config.environment("CartPole-v1")  
>>> print(config.to_dict())  
>>> # Build a Algorithm object from the config and run 1 training iteration.
>>> algo = config.build()  
>>> algo.train()  

Example

>>> import ray.air as air
>>> from ray import tune
>>> from ray.rllib.algorithms.a2c import A2CConfig
>>> config = A2CConfig()
>>> # Print out some default values.
>>> print(config.sample_async)   
>>> # Update the config object.
>>> config = config.training(lr=tune.grid_search(  
...     [0.001, 0.0001]), use_critic=False)
>>> # Set the config object's env.
>>> config = config.environment(env="CartPole-v1")  
>>> # Use to_dict() to get the old-style python config dict
>>> # when running with tune.
>>> tune.Tuner(  
...     "A2C",
...     run_config=air.RunConfig(stop={"episode_reward_mean": 200}),
...     param_space=config.to_dict(),
... ).fit()
training(*, microbatch_size: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, **kwargs) ray.rllib.algorithms.a2c.a2c.A2CConfig[source]

Sets the training related configuration.

Parameters

microbatch_size – A2C supports microbatching, in which we accumulate gradients over batch of this size until the train batch size is reached. This allows training with batch sizes much larger than can fit in GPU memory. To enable, set this to a value less than the train batch size.

Returns

This updated AlgorithmConfig object.

Asynchronous Advantage Actor-Critic (A3C)

pytorch tensorflow [paper] [implementation] A3C is the asynchronous version of A2C, where gradients are computed on the workers directly after trajectory rollouts, and only then shipped to a central learner to accumulate these gradients on the central model. After the central model update, parameters are broadcast back to all workers. Similar to A2C, A3C scales to 16-32+ worker processes depending on the environment.

Tuned examples: PongDeterministic-v4

Tip

Consider using IMPALA for faster training with similar timestep efficiency.

A3C-specific configs (see also common configs):

class ray.rllib.algorithms.a3c.a3c.A3CConfig(algo_class=None)[source]

Defines a configuration class from which a A3C Algorithm can be built.

Example

>>> from ray import tune
>>> from ray.rllib.algorithms.a3c import A3CConfig
>>> config = A3CConfig() 
>>> config = config.training(lr=0.01, grad_clip=30.0) 
>>> config = config.resources(num_gpus=0) 
>>> config = config.rollouts(num_rollout_workers=4) 
>>> config = config.environment("CartPole-v1") 
>>> print(config.to_dict())  
>>> # Build a Algorithm object from the config and run 1 training iteration.
>>> algo = config.build()  
>>> algo.train()  

Example

>>> from ray.rllib.algorithms.a3c import A3CConfig
>>> config = A3CConfig()
>>> # Print out some default values.
>>> print(config.sample_async)  
>>> # Update the config object.
>>> config = config.training( 
...     lr=tune.grid_search([0.001, 0.0001]), use_critic=False)
>>> # Set the config object's env.
>>> config = config.environment(env="CartPole-v1") 
>>> # Use to_dict() to get the old-style python config dict
>>> # when running with tune.
>>> tune.Tuner(  
...     "A3C",
...     stop={"episode_reward_mean": 200},
...     param_space=config.to_dict(),
... ).fit()
training(*, lr_schedule: Optional[List[List[Union[int, float]]]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, use_critic: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, use_gae: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, lambda_: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, grad_clip: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, vf_loss_coeff: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, entropy_coeff: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, entropy_coeff_schedule: Optional[List[List[Union[int, float]]]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, sample_async: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, **kwargs) ray.rllib.algorithms.a3c.a3c.A3CConfig[source]

Sets the training related configuration.

Parameters
  • lr_schedule – Learning rate schedule. In the format of [[timestep, lr-value], [timestep, lr-value], …] Intermediary timesteps will be assigned to interpolated learning rate values. A schedule should normally start from timestep 0.

  • use_critic – Should use a critic as a baseline (otherwise don’t use value baseline; required for using GAE).

  • use_gae – If true, use the Generalized Advantage Estimator (GAE) with a value function, see https://arxiv.org/pdf/1506.02438.pdf.

  • lambda – GAE(gamma) parameter.

  • grad_clip – Max global norm for each gradient calculated by worker.

  • vf_loss_coeff – Value Function Loss coefficient.

  • entropy_coeff – Coefficient of the entropy regularizer.

  • entropy_coeff_schedule – Decay schedule for the entropy regularizer.

  • sample_async – Whether workers should sample async. Note that this increases the effective rollout_fragment_length by up to 5x due to async buffering of batches.

Returns

This updated AlgorithmConfig object.

Policy Gradients (PG)

pytorch tensorflow [paper] [implementation] We include a vanilla policy gradients implementation as an example algorithm.

../_images/a2c-arch.svg

Policy gradients architecture (same as A2C)

Tuned examples: CartPole-v1

PG-specific configs (see also common configs):

class ray.rllib.algorithms.pg.pg.PGConfig(algo_class=None)[source]

Defines a configuration class from which a PG Algorithm can be built.

Example

>>> from ray.rllib.algorithms.pg import PGConfig
>>> config = PGConfig().training(lr=0.01).resources(num_gpus=1)
>>> print(config.to_dict())  
>>> # Build a Algorithm object from the config and run 1 training iteration.
>>> algo = config.build(env="CartPole-v1")  
>>> algo.train()  

Example

>>> from ray.rllib.algorithms.pg import PGConfig
>>> from ray import air
>>> from ray import tune
>>> config = PGConfig()
>>> # Print out some default values.
>>> print(config.lr) 
0.0004
>>> # Update the config object.
>>> config = config.training(lr=tune.grid_search([0.001, 0.0001]))
>>> # Set the config object's env.
>>> config = config.environment(env="CartPole-v1")
>>> # Use to_dict() to get the old-style python config dict
>>> # when running with tune.
>>> tune.Tuner(  
...     "PG",
...     run_config=air.RunConfig(stop={"episode_reward_mean": 200}),
...     param_space=config.to_dict(),
... ).fit()
training(*, lr_schedule: Optional[List[List[Union[int, float]]]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, **kwargs) ray.rllib.algorithms.pg.pg.PGConfig[source]

Sets the training related configuration.

Parameters
  • gamma – Float specifying the discount factor of the Markov Decision process.

  • lr – The default learning rate.

  • train_batch_size – Training batch size, if applicable.

  • model – Arguments passed into the policy model. See models/catalog.py for a full list of the available model options.

  • optimizer – Arguments to pass to the policy optimizer.

  • lr_schedule – Learning rate schedule. In the format of [[timestep, lr-value], [timestep, lr-value], …] Intermediary timesteps will be assigned to interpolated learning rate values. A schedule should normally start from timestep 0.

Returns

This updated AlgorithmConfig object.

Model-Agnostic Meta-Learning (MAML)

pytorch tensorflow [paper] [implementation]

RLlib’s MAML implementation is a meta-learning method for learning and quick adaptation across different tasks for continuous control. Code here is adapted from https://github.com/jonasrothfuss, which outperforms vanilla MAML and avoids computation of the higher order gradients during the meta-update step. MAML is evaluated on custom environments that are described in greater detail here.

MAML uses additional metrics to measure performance; episode_reward_mean measures the agent’s returns before adaptation, episode_reward_mean_adapt_N measures the agent’s returns after N gradient steps of inner adaptation, and adaptation_delta measures the difference in performance before and after adaptation. Examples can be seen here.

Tuned examples: HalfCheetahRandDirecEnv (Env, Config), AntRandGoalEnv (Env, Config), PendulumMassEnv (Env, Config)

MAML-specific configs (see also common configs):

class ray.rllib.algorithms.maml.maml.MAMLConfig(algo_class=None)[source]

Defines a configuration class from which a MAML Algorithm can be built.

Example

>>> from ray.rllib.algorithms.maml import MAMLConfig
>>> config = MAMLConfig().training(use_gae=False).resources(num_gpus=1)
>>> print(config.to_dict())  
>>> # Build a Algorithm object from the config and run 1 training iteration.
>>> algo = config.build(env="CartPole-v1")  
>>> algo.train()  

Example

>>> from ray.rllib.algorithms.maml import MAMLConfig
>>> from ray import air
>>> from ray import tune
>>> config = MAMLConfig()
>>> # Print out some default values.
>>> print(config.lr)  
>>> # Update the config object.
>>> config = config.training(  
...     grad_clip=tune.grid_search([10.0, 40.0]))
>>> # Set the config object's env.
>>> config = config.environment(env="CartPole-v1")
>>> # Use to_dict() to get the old-style python config dict
>>> # when running with tune.
>>> tune.Tuner(  
...     "MAML",
...     run_config=air.RunConfig(stop={"episode_reward_mean": 200}),
...     param_space=config.to_dict(),
... ).fit()
training(*, use_gae: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, lambda_: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, kl_coeff: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, vf_loss_coeff: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, entropy_coeff: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, clip_param: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, vf_clip_param: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, grad_clip: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, kl_target: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, inner_adaptation_steps: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, maml_optimizer_steps: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, inner_lr: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, use_meta_env: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, **kwargs) ray.rllib.algorithms.maml.maml.MAMLConfig[source]

Sets the training related configuration.

Parameters
  • use_gae – If true, use the Generalized Advantage Estimator (GAE) with a value function, see https://arxiv.org/pdf/1506.02438.pdf.

  • lambda – The GAE (lambda) parameter.

  • kl_coeff – Initial coefficient for KL divergence.

  • vf_loss_coeff – Coefficient of the value function loss.

  • entropy_coeff – Coefficient of the entropy regularizer.

  • clip_param – PPO clip parameter.

  • vf_clip_param – Clip param for the value function. Note that this is sensitive to the scale of the rewards. If your expected V is large, increase this.

  • grad_clip – If specified, clip the global norm of gradients by this amount.

  • kl_target – Target value for KL divergence.

  • inner_adaptation_steps – Number of Inner adaptation steps for the MAML algorithm.

  • maml_optimizer_steps – Number of MAML steps per meta-update iteration (PPO steps).

  • inner_lr – Inner Adaptation Step size.

  • use_meta_env – Use Meta Env Template.

Returns

This updated AlgorithmConfig object.

Model-free Off-policy RL

Distributed Prioritized Experience Replay (Ape-X)

pytorch tensorflow [paper] [implementation] Ape-X variations of DQN and DDPG (APEX_DQN, APEX_DDPG) use a single GPU learner and many CPU workers for experience collection. Experience collection can scale to hundreds of CPU workers due to the distributed prioritization of experience prior to storage in replay buffers.

../_images/apex-arch.svg

Ape-X architecture

Tuned examples: PongNoFrameskip-v4, Pendulum-v1, MountainCarContinuous-v0, {BeamRider,Breakout,Qbert,SpaceInvaders}NoFrameskip-v4.

Atari results @10M steps: more details

Atari env

RLlib Ape-X 8-workers

Mnih et al Async DQN 16-workers

BeamRider

6134

~6000

Breakout

123

~50

Qbert

15302

~1200

SpaceInvaders

686

~600

Scalability:

Atari env

RLlib Ape-X 8-workers @1 hour

Mnih et al Async DQN 16-workers @1 hour

BeamRider

4873

~1000

Breakout

77

~10

Qbert

4083

~500

SpaceInvaders

646

~300

../_images/apex.png

Ape-X using 32 workers in RLlib vs vanilla DQN (orange) and A3C (blue) on PongNoFrameskip-v4.

Ape-X specific configs (see also common configs):

class ray.rllib.algorithms.apex_dqn.apex_dqn.ApexDQNConfig(algo_class=None)[source]

Defines a configuration class from which an ApexDQN Algorithm can be built.

Example

>>> from ray.rllib.algorithms.apex_dqn.apex_dqn import ApexDQNConfig
>>> config = ApexDQNConfig()
>>> print(config.replay_buffer_config) 
>>> replay_config = config.replay_buffer_config.update( 
...     {
...         "capacity": 100000,
...         "prioritized_replay_alpha": 0.45,
...         "prioritized_replay_beta": 0.55,
...         "prioritized_replay_eps": 3e-6,
...     }
... )
>>> config = config.training(replay_buffer_config=replay_config) 
>>> config = config.resources(num_gpus=1)  
>>> config = config.rollouts(num_rollout_workers=30)  
>>> config = config.environment("CartPole-v1")  
>>> algo = config.build() 
>>> algo.train()  

Example

>>> from ray.rllib.algorithms.apex_dqn.apex_dqn import ApexDQNConfig
>>> from ray import air
>>> from ray import tune
>>> config = ApexDQNConfig()
>>> config.training(  
...     num_atoms=tune.grid_search(list(range(1, 11)))
>>> config.environment(env="CartPole-v1")  
>>> tune.Tuner( 
...     "APEX",
...     run_config=air.RunConfig(stop={"episode_reward_mean":200}),
...     param_space=config.to_dict()
... ).fit()

Example

>>> from ray.rllib.algorithms.apex_dqn.apex_dqn import ApexDQNConfig
>>> config = ApexDQNConfig()
>>> print(config.exploration_config)  
>>> explore_config = config.exploration_config.update(  
...     {
...         "type": "EpsilonGreedy",
...         "initial_epsilon": 0.96,
...         "final_epsilon": 0.01,
...         "epsilone_timesteps": 5000,
...     }
... )
>>> config = config.training(  
...     lr_schedule=[[1, 1e-3, [500, 5e-3]]
... )
>>> config = config.exploration(  
...     exploration_config=explore_config
... )

Example

>>> from ray.rllib.algorithms.apex_dqn.apex_dqn import ApexDQNConfig
>>> config = ApexDQNConfig()
>>> print(config.exploration_config)  
>>> explore_config = config.exploration_config.update(  
...     {
...         "type": "SoftQ",
...         "temperature": [1.0],
...     }
... )
>>> config = config.training(  
...     lr_schedule=[[1, 1e-3, [500, 5e-3]]
... )
>>> config = config.exploration(  
...     exploration_config=explore_config
... )
training(*, max_requests_in_flight_per_replay_worker: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, timeout_s_sampler_manager: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, timeout_s_replay_manager: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, **kwargs) ray.rllib.algorithms.apex_dqn.apex_dqn.ApexDQNConfig[source]

Sets the training related configuration.

Parameters
  • num_atoms – Number of atoms for representing the distribution of return. When this is greater than 1, distributional Q-learning is used.

  • v_min – Minimum value estimation

  • v_max – Maximum value estimation

  • noisy – Whether to use noisy network to aid exploration. This adds parametric noise to the model weights.

  • sigma0 – Control the initial parameter noise for noisy nets.

  • dueling – Whether to use dueling DQN.

  • hiddens – Dense-layer setup for each the advantage branch and the value branch

  • double_q – Whether to use double DQN.

  • n_step – N-step for Q-learning.

  • before_learn_on_batch – Callback to run before learning on a multi-agent batch of experiences.

  • training_intensity – The intensity with which to update the model (vs collecting samples from the env). If None, uses “natural” values of: train_batch_size / (rollout_fragment_length x num_workers x num_envs_per_worker). If not None, will make sure that the ratio between timesteps inserted into and sampled from the buffer matches the given values. Example: training_intensity=1000.0 train_batch_size=250 rollout_fragment_length=1 num_workers=1 (or 0) num_envs_per_worker=1 -> natural value = 250 / 1 = 250.0 -> will make sure that replay+train op will be executed 4x asoften as rollout+insert op (4 * 250 = 1000). See: rllib/algorithms/dqn/dqn.py::calculate_rr_weights for further details.

  • replay_buffer_config – Replay buffer config. Examples: { “_enable_replay_buffer_api”: True, “type”: “MultiAgentReplayBuffer”, “capacity”: 50000, “replay_sequence_length”: 1, } - OR - { “_enable_replay_buffer_api”: True, “type”: “MultiAgentPrioritizedReplayBuffer”, “capacity”: 50000, “prioritized_replay_alpha”: 0.6, “prioritized_replay_beta”: 0.4, “prioritized_replay_eps”: 1e-6, “replay_sequence_length”: 1, } - Where - prioritized_replay_alpha: Alpha parameter controls the degree of prioritization in the buffer. In other words, when a buffer sample has a higher temporal-difference error, with how much more probability should it drawn to use to update the parametrized Q-network. 0.0 corresponds to uniform probability. Setting much above 1.0 may quickly result as the sampling distribution could become heavily “pointy” with low entropy. prioritized_replay_beta: Beta parameter controls the degree of importance sampling which suppresses the influence of gradient updates from samples that have higher probability of being sampled via alpha parameter and the temporal-difference error. prioritized_replay_eps: Epsilon parameter sets the baseline probability for sampling so that when the temporal-difference error of a sample is zero, there is still a chance of drawing the sample.

  • max_requests_in_flight_per_replay_worker – Max number of inflight requests to each replay (shard) worker. See the FaultTolerantActorManager class for more details. Tuning these values is important when running experimens with large sample batches, where there is the risk that the object store may fill up, causing spilling of objects to disk. This can cause any asynchronous requests to become very slow, making your experiment run slow as well. You can inspect the object store during your experiment via a call to ray memory on your headnode, and by using the ray dashboard. If you’re seeing that the object store is filling up, turn down the number of remote requests in flight, or enable compression in your experiment of timesteps.

  • timeout_s_sampler_manager – The timeout for waiting for sampling results for workers – typically if this is too low, the manager won’t be able to retrieve ready sampling results.

  • timeout_s_replay_manager – The timeout for waiting for replay worker results – typically if this is too low, the manager won’t be able to retrieve ready replay requests.

Recurrent Replay Distributed DQN (R2D2)

pytorch tensorflow [paper] [implementation] R2D2 can be scaled by increasing the number of workers. All of the DQN improvements evaluated in Rainbow are available, though not all are enabled by default.

Tuned examples: Stateless CartPole-v1

Deep Q Networks (DQN, Rainbow, Parametric DQN)

pytorch tensorflow [paper] [implementation] DQN can be scaled by increasing the number of workers or using Ape-X. Memory usage is reduced by compressing samples in the replay buffer with LZ4. All of the DQN improvements evaluated in Rainbow are available, though not all are enabled by default. See also how to use parametric-actions in DQN.

../_images/dqn-arch.svg

DQN architecture

Tuned examples: PongDeterministic-v4, Rainbow configuration, {BeamRider,Breakout,Qbert,SpaceInvaders}NoFrameskip-v4, with Dueling and Double-Q, with Distributional DQN.

Tip

Consider using Ape-X for faster training with similar timestep efficiency.

Hint

For a complete rainbow setup, make the following changes to the default DQN config: "n_step": [between 1 and 10], "noisy": True, "num_atoms": [more than 1], "v_min": -10.0, "v_max": 10.0 (set v_min and v_max according to your expected range of returns).

Atari results @10M steps: more details

Atari env

RLlib DQN

RLlib Dueling DDQN

RLlib Dist. DQN

Hessel et al. DQN

BeamRider

2869

1910

4447

~2000

Breakout

287

312

410

~150

Qbert

3921

7968

15780

~4000

SpaceInvaders

650

1001

1025

~500

DQN-specific configs (see also common configs):

class ray.rllib.algorithms.dqn.dqn.DQNConfig(algo_class=None)[source]

Defines a configuration class from which a DQN Algorithm can be built.

Example

>>> from ray.rllib.algorithms.dqn.dqn import DQNConfig
>>> config = DQNConfig()
>>> print(config.replay_buffer_config)  
>>> replay_config = config.replay_buffer_config.update( 
...     {
...         "capacity": 60000,
...         "prioritized_replay_alpha": 0.5,
...         "prioritized_replay_beta": 0.5,
...         "prioritized_replay_eps": 3e-6,
...     }
... )
>>> config = config.training(replay_buffer_config=replay_config)
>>> config = config.resources(num_gpus=1)  
>>> config = config.rollouts(num_rollout_workers=3)  
>>> config = config.environment("CartPole-v1")  
>>> trainer = DQN(config=config)  
>>> trainer.train()  

Example

>>> from ray.rllib.algorithms.dqn.dqn import DQNConfig
>>> from ray import air
>>> from ray import tune
>>> config = DQNConfig()
>>> config = config.training( 
...     num_atoms=tune.grid_search(list(range(1,11))))
>>> config = config.environment(env="CartPole-v1") 
>>> tune.Tuner(  
...     "DQN",
...     run_config=air.RunConfig(stop={"episode_reward_mean":200}),
...     param_space=config.to_dict()
... ).fit()

Example

>>> from ray.rllib.algorithms.dqn.dqn import DQNConfig
>>> config = DQNConfig()
>>> print(config.exploration_config)  
>>> explore_config = config.exploration_config.update( 
...     {
...         "initial_epsilon": 1.5,
...         "final_epsilon": 0.01,
...         "epsilone_timesteps": 5000,
...     }
... )
>>> config.training(lr_schedule=[[1, 1e-3, [500, 5e-3]])\ 
...       .exploration(exploration_config=explore_config)

Example

>>> from ray.rllib.algorithms.dqn.dqn import DQNConfig
>>> config = DQNConfig()
>>> print(config.exploration_config)  
>>> explore_config = config.exploration_config.update( 
...     {
...         "type": "softq",
...         "temperature": [1.0],
...     }
... )
>>> config.training(lr_schedule=[[1, 1e-3, [500, 5e-3]])\ 
...       .exploration(exploration_config=explore_config)
training(*, num_atoms: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, v_min: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, v_max: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, noisy: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, sigma0: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, dueling: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, hiddens: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, double_q: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, n_step: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, before_learn_on_batch: Callable[[Type[ray.rllib.policy.sample_batch.MultiAgentBatch], List[Type[ray.rllib.policy.policy.Policy]], Type[int]], Type[ray.rllib.policy.sample_batch.MultiAgentBatch]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, training_intensity: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, td_error_loss_fn: Optional[str] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, categorical_distribution_temperature: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, **kwargs) ray.rllib.algorithms.dqn.dqn.DQNConfig[source]

Sets the training related configuration.

Parameters
  • num_atoms – Number of atoms for representing the distribution of return. When this is greater than 1, distributional Q-learning is used.

  • v_min – Minimum value estimation

  • v_max – Maximum value estimation

  • noisy – Whether to use noisy network to aid exploration. This adds parametric noise to the model weights.

  • sigma0 – Control the initial parameter noise for noisy nets.

  • dueling – Whether to use dueling DQN.

  • hiddens – Dense-layer setup for each the advantage branch and the value branch

  • double_q – Whether to use double DQN.

  • n_step – N-step for Q-learning.

  • before_learn_on_batch – Callback to run before learning on a multi-agent batch of experiences.

  • training_intensity – The intensity with which to update the model (vs collecting samples from the env). If None, uses “natural” values of: train_batch_size / (rollout_fragment_length x num_workers x num_envs_per_worker). If not None, will make sure that the ratio between timesteps inserted into and sampled from the buffer matches the given values. Example: training_intensity=1000.0 train_batch_size=250 rollout_fragment_length=1 num_workers=1 (or 0) num_envs_per_worker=1 -> natural value = 250 / 1 = 250.0 -> will make sure that replay+train op will be executed 4x asoften as rollout+insert op (4 * 250 = 1000). See: rllib/algorithms/dqn/dqn.py::calculate_rr_weights for further details.

  • replay_buffer_config – Replay buffer config. Examples: { “_enable_replay_buffer_api”: True, “type”: “MultiAgentReplayBuffer”, “capacity”: 50000, “replay_sequence_length”: 1, } - OR - { “_enable_replay_buffer_api”: True, “type”: “MultiAgentPrioritizedReplayBuffer”, “capacity”: 50000, “prioritized_replay_alpha”: 0.6, “prioritized_replay_beta”: 0.4, “prioritized_replay_eps”: 1e-6, “replay_sequence_length”: 1, } - Where - prioritized_replay_alpha: Alpha parameter controls the degree of prioritization in the buffer. In other words, when a buffer sample has a higher temporal-difference error, with how much more probability should it drawn to use to update the parametrized Q-network. 0.0 corresponds to uniform probability. Setting much above 1.0 may quickly result as the sampling distribution could become heavily “pointy” with low entropy. prioritized_replay_beta: Beta parameter controls the degree of importance sampling which suppresses the influence of gradient updates from samples that have higher probability of being sampled via alpha parameter and the temporal-difference error. prioritized_replay_eps: Epsilon parameter sets the baseline probability for sampling so that when the temporal-difference error of a sample is zero, there is still a chance of drawing the sample.

  • td_error_loss_fn – “huber” or “mse”. loss function for calculating TD error when num_atoms is 1. Note that if num_atoms is > 1, this parameter is simply ignored, and softmax cross entropy loss will be used.

  • categorical_distribution_temperature – Set the temperature parameter used by Categorical action distribution. A valid temperature is in the range of [0, 1]. Note that this mostly affects evaluation since TD error uses argmax for return calculation.

Returns

This updated AlgorithmConfig object.

Deep Deterministic Policy Gradients (DDPG)

pytorch tensorflow [paper] [implementation] DDPG is implemented similarly to DQN (below). The algorithm can be scaled by increasing the number of workers or using Ape-X. The improvements from TD3 are available as TD3.

../_images/dqn-arch.svg

DDPG architecture (same as DQN)

Tuned examples: Pendulum-v1, MountainCarContinuous-v0, HalfCheetah-v2.

DDPG-specific configs (see also common configs):

class ray.rllib.algorithms.ddpg.ddpg.DDPGConfig(algo_class=None)[source]

Defines a configuration class from which a DDPG Trainer can be built.

Example

>>> from ray.rllib.algorithms.ddpg.ddpg import DDPGConfig
>>> config = DDPGConfig().training(lr=0.01).resources(num_gpus=1)
>>> print(config.to_dict())  
>>> # Build a Trainer object from the config and run one training iteration.
>>> algo = config.build(env="Pendulum-v1") 
>>> algo.train()  

Example

>>> from ray.rllib.algorithms.ddpg.ddpg import DDPGConfig
>>> from ray import air
>>> from ray import tune
>>> config = DDPGConfig()
>>> # Print out some default values.
>>> print(config.lr) 
0.0004
>>> # Update the config object.
>>> config = config.training(lr=tune.grid_search([0.001, 0.0001]))
>>> # Set the config object's env.
>>> config = config.environment(env="Pendulum-v1")
>>> # Use to_dict() to get the old-style python config dict
>>> # when running with tune.
>>> tune.Tuner(  
...     "DDPG",
...     run_config=air.RunConfig(stop={"episode_reward_mean": 200}),
...     param_space=config.to_dict(),
... ).fit()
training(*, twin_q: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, policy_delay: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, smooth_target_policy: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, target_noise: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, target_noise_clip: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, use_state_preprocessor: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, actor_hiddens: Optional[List[int]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, actor_hidden_activation: Optional[str] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, critic_hiddens: Optional[List[int]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, critic_hidden_activation: Optional[str] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, n_step: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, critic_lr: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, actor_lr: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, tau: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, use_huber: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, huber_threshold: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, l2_reg: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, training_intensity: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, **kwargs) ray.rllib.algorithms.ddpg.ddpg.DDPGConfig[source]

Sets the training related configuration.

=== Twin Delayed DDPG (TD3) and Soft Actor-Critic (SAC) tricks === TD3: https://spinningup.openai.com/en/latest/algorithms/td3.html In addition to settings below, you can use “exploration_noise_type” and “exploration_gauss_act_noise” to get IID Gaussian exploration noise instead of OrnsteinUhlenbeck exploration noise.

Parameters
  • twin_q – Use twin Q-net.

  • policy_delay – Delayed policy update.

  • smooth_target_policy – Target policy smoothing (this also replaces OrnsteinUhlenbeck exploration noise with IID Gaussian exploration noise, for now).

  • target_noise – Gaussian stddev of target action noise for smoothing.

  • target_noise_clip – Target noise limit (bound).

  • use_state_preprocessor – Apply a state preprocessor with spec given by the “model” config option (like other RL algorithms). This is mostly useful if you have a weird observation shape, like an image. Disabled by default.

  • actor_hiddens – Postprocess the policy network model output with these hidden layers. If use_state_preprocessor is False, then these will be the only hidden layers in the network.

  • actor_hidden_activation – Hidden layers activation of the postprocessing stage of the policy network

  • critic_hiddens – Postprocess the critic network model output with these hidden layers; again, if use_state_preprocessor is True, then the state will be preprocessed by the model specified with the “model” config option first.

  • critic_hidden_activation – Hidden layers activation of the postprocessing state of the critic.

  • n_step – N-step Q learning

  • critic_lr – Learning rate for the critic (Q-function) optimizer.

  • actor_lr – Learning rate for the actor (policy) optimizer.

  • tau – Update the target by au * policy + (1- au) * target_policy

  • use_huber – Conventionally, no need to clip gradients if using a huber loss

  • huber_threshold – Threshold of a huber loss

  • l2_reg – Weights for L2 regularization

  • training_intensity

    The intensity with which to update the model (vs collecting samples from the env). If None, uses the “natural” value of: train_batch_size / (rollout_fragment_length x num_workers x num_envs_per_worker). If provided, will make sure that the ratio between ts inserted into and sampled from the buffer matches the given value. .. rubric:: Example

    training_intensity=1000.0 train_batch_size=250 rollout_fragment_length=1 num_workers=1 (or 0) num_envs_per_worker=1 -> natural value = 250 / 1 = 250.0 -> will make sure that replay+train op will be executed 4x as often as rollout+insert op (4 * 250 = 1000).

    See: rllib/algorithms/dqn/dqn.py::calculate_rr_weights for further details.

Returns

This updated DDPGConfig object.

Twin Delayed DDPG (TD3)

pytorch tensorflow [paper] [implementation] TD3 represents an improvement over DDPG. Its implementation is available in RLlib as TD3.

Tuned examples: TD3 Pendulum-v1, TD3 InvertedPendulum-v2, TD3 Mujoco suite (Ant-v2, HalfCheetah-v2, Hopper-v2, Walker2d-v2).

TD3-specific configs (see also common configs):

class ray.rllib.algorithms.td3.td3.TD3Config(algo_class=None)[source]

Defines a configuration class from which a TD3 Algorithm can be built.

Example

>>> from ray.rllib.algorithms.td3 import TD3Config
>>> config = TD3Config().training(lr=0.01).resources(num_gpus=1)
>>> print(config.to_dict())  
>>> # Build a Algorithm object from the config and run one training iteration.
>>> algo = config.build(env="Pendulum-v1")  
>>> algo.train()  

Example

>>> from ray.rllib.algorithms.td3 import TD3Config
>>> from ray import air
>>> from ray import tune
>>> config = TD3Config()
>>> # Print out some default values.
>>> print(config.lr)   
>>> # Update the config object.
>>> config = config.training(lr=tune.grid_search(  
...     [0.001, 0.0001]))  
>>> # Set the config object's env.
>>> config.environment(env="Pendulum-v1")  
>>> # Use to_dict() to get the old-style python config dict
>>> # when running with tune.
>>> tune.Tuner(  
...     "TD3",
...     run_config=air.RunConfig(stop={"episode_reward_mean": 200}),
...     param_space=config.to_dict(),
... ).fit()
training(*, twin_q: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, policy_delay: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, smooth_target_policy: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, target_noise: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, target_noise_clip: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, use_state_preprocessor: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, actor_hiddens: Optional[List[int]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, actor_hidden_activation: Optional[str] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, critic_hiddens: Optional[List[int]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, critic_hidden_activation: Optional[str] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, n_step: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, critic_lr: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, actor_lr: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, tau: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, use_huber: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, huber_threshold: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, l2_reg: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, training_intensity: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, **kwargs) ray.rllib.algorithms.ddpg.ddpg.DDPGConfig

Sets the training related configuration.

=== Twin Delayed DDPG (TD3) and Soft Actor-Critic (SAC) tricks === TD3: https://spinningup.openai.com/en/latest/algorithms/td3.html In addition to settings below, you can use “exploration_noise_type” and “exploration_gauss_act_noise” to get IID Gaussian exploration noise instead of OrnsteinUhlenbeck exploration noise.

Parameters
  • twin_q – Use twin Q-net.

  • policy_delay – Delayed policy update.

  • smooth_target_policy – Target policy smoothing (this also replaces OrnsteinUhlenbeck exploration noise with IID Gaussian exploration noise, for now).

  • target_noise – Gaussian stddev of target action noise for smoothing.

  • target_noise_clip – Target noise limit (bound).

  • use_state_preprocessor – Apply a state preprocessor with spec given by the “model” config option (like other RL algorithms). This is mostly useful if you have a weird observation shape, like an image. Disabled by default.

  • actor_hiddens – Postprocess the policy network model output with these hidden layers. If use_state_preprocessor is False, then these will be the only hidden layers in the network.

  • actor_hidden_activation – Hidden layers activation of the postprocessing stage of the policy network

  • critic_hiddens – Postprocess the critic network model output with these hidden layers; again, if use_state_preprocessor is True, then the state will be preprocessed by the model specified with the “model” config option first.

  • critic_hidden_activation – Hidden layers activation of the postprocessing state of the critic.

  • n_step – N-step Q learning

  • critic_lr – Learning rate for the critic (Q-function) optimizer.

  • actor_lr – Learning rate for the actor (policy) optimizer.

  • tau – Update the target by au * policy + (1- au) * target_policy

  • use_huber – Conventionally, no need to clip gradients if using a huber loss

  • huber_threshold – Threshold of a huber loss

  • l2_reg – Weights for L2 regularization

  • training_intensity

    The intensity with which to update the model (vs collecting samples from the env). If None, uses the “natural” value of: train_batch_size / (rollout_fragment_length x num_workers x num_envs_per_worker). If provided, will make sure that the ratio between ts inserted into and sampled from the buffer matches the given value. .. rubric:: Example

    training_intensity=1000.0 train_batch_size=250 rollout_fragment_length=1 num_workers=1 (or 0) num_envs_per_worker=1 -> natural value = 250 / 1 = 250.0 -> will make sure that replay+train op will be executed 4x as often as rollout+insert op (4 * 250 = 1000).

    See: rllib/algorithms/dqn/dqn.py::calculate_rr_weights for further details.

Returns

This updated DDPGConfig object.

Soft Actor Critic (SAC)

pytorch tensorflow [original paper], [follow up paper], [discrete actions paper] [implementation]

../_images/dqn-arch.svg

SAC architecture (same as DQN)

RLlib’s soft-actor critic implementation is ported from the official SAC repo to better integrate with RLlib APIs. Note that SAC has two fields to configure for custom models: policy_model_config and q_model_config, the model field of the config will be ignored.

Tuned examples (continuous actions): Pendulum-v1, HalfCheetah-v3, Tuned examples (discrete actions): CartPole-v1

MuJoCo results @3M steps: more details

MuJoCo env

RLlib SAC

Haarnoja et al SAC

HalfCheetah

13000

~15000

SAC-specific configs (see also common configs):

class ray.rllib.algorithms.sac.sac.SACConfig(algo_class=None)[source]

Defines a configuration class from which an SAC Algorithm can be built.

Example

>>> config = SACConfig().training(gamma=0.9, lr=0.01)  
>>> config = config.resources(num_gpus=0)  
>>> config = config.rollouts(num_rollout_workers=4)  
>>> print(config.to_dict())  
>>> # Build a Algorithm object from the config and run 1 training iteration.
>>> algo = config.build(env="CartPole-v1")  
>>> algo.train()  
training(*, twin_q: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, q_model_config: Optional[Dict[str, Any]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, policy_model_config: Optional[Dict[str, Any]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, tau: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, initial_alpha: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, target_entropy: Optional[Union[str, float]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, n_step: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, store_buffer_in_checkpoints: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, replay_buffer_config: Optional[Dict[str, Any]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, training_intensity: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, clip_actions: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, grad_clip: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, optimization_config: Optional[Dict[str, Any]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, target_network_update_freq: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, _deterministic_loss: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, _use_beta_distribution: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, num_steps_sampled_before_learning_starts: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, **kwargs) ray.rllib.algorithms.sac.sac.SACConfig[source]

Sets the training related configuration.

Parameters
  • twin_q – Use two Q-networks (instead of one) for action-value estimation. Note: Each Q-network will have its own target network.

  • q_model_config – Model configs for the Q network(s). These will override MODEL_DEFAULTS. This is treated just as the top-level model dict in setting up the Q-network(s) (2 if twin_q=True). That means, you can do for different observation spaces: obs=Box(1D) -> Tuple(Box(1D) + Action) -> concat -> post_fcnet obs=Box(3D) -> Tuple(Box(3D) + Action) -> vision-net -> concat w/ action -> post_fcnet obs=Tuple(Box(1D), Box(3D)) -> Tuple(Box(1D), Box(3D), Action) -> vision-net -> concat w/ Box(1D) and action -> post_fcnet You can also have SAC use your custom_model as Q-model(s), by simply specifying the custom_model sub-key in below dict (just like you would do in the top-level model dict.

  • policy_model_config – Model options for the policy function (see q_model_config above for details). The difference to q_model_config above is that no action concat’ing is performed before the post_fcnet stack.

  • tau – Update the target by au * policy + (1- au) * target_policy.

  • initial_alpha – Initial value to use for the entropy weight alpha.

  • target_entropy – Target entropy lower bound. If “auto”, will be set to -|A| (e.g. -2.0 for Discrete(2), -3.0 for Box(shape=(3,))). This is the inverse of reward scale, and will be optimized automatically.

  • n_step – N-step target updates. If >1, sars’ tuples in trajectories will be postprocessed to become sa[discounted sum of R][s t+n] tuples.

  • store_buffer_in_checkpoints – Set this to True, if you want the contents of your buffer(s) to be stored in any saved checkpoints as well. Warnings will be created if: - This is True AND restoring from a checkpoint that contains no buffer data. - This is False AND restoring from a checkpoint that does contain buffer data.

  • replay_buffer_config – Replay buffer config. Examples: { “_enable_replay_buffer_api”: True, “type”: “MultiAgentReplayBuffer”, “capacity”: 50000, “replay_batch_size”: 32, “replay_sequence_length”: 1, } - OR - { “_enable_replay_buffer_api”: True, “type”: “MultiAgentPrioritizedReplayBuffer”, “capacity”: 50000, “prioritized_replay_alpha”: 0.6, “prioritized_replay_beta”: 0.4, “prioritized_replay_eps”: 1e-6, “replay_sequence_length”: 1, } - Where - prioritized_replay_alpha: Alpha parameter controls the degree of prioritization in the buffer. In other words, when a buffer sample has a higher temporal-difference error, with how much more probability should it drawn to use to update the parametrized Q-network. 0.0 corresponds to uniform probability. Setting much above 1.0 may quickly result as the sampling distribution could become heavily “pointy” with low entropy. prioritized_replay_beta: Beta parameter controls the degree of importance sampling which suppresses the influence of gradient updates from samples that have higher probability of being sampled via alpha parameter and the temporal-difference error. prioritized_replay_eps: Epsilon parameter sets the baseline probability for sampling so that when the temporal-difference error of a sample is zero, there is still a chance of drawing the sample.

  • training_intensity – The intensity with which to update the model (vs collecting samples from the env). If None, uses “natural” values of: train_batch_size / (rollout_fragment_length x num_workers x num_envs_per_worker). If not None, will make sure that the ratio between timesteps inserted into and sampled from th buffer matches the given values. Example: training_intensity=1000.0 train_batch_size=250 rollout_fragment_length=1 num_workers=1 (or 0) num_envs_per_worker=1 -> natural value = 250 / 1 = 250.0 -> will make sure that replay+train op will be executed 4x asoften as rollout+insert op (4 * 250 = 1000). See: rllib/algorithms/dqn/dqn.py::calculate_rr_weights for further details.

  • clip_actions – Whether to clip actions. If actions are already normalized, this should be set to False.

  • grad_clip – If not None, clip gradients during optimization at this value.

  • optimization_config – Config dict for optimization. Set the supported keys actor_learning_rate, critic_learning_rate, and entropy_learning_rate in here.

  • target_network_update_freq – Update the target network every target_network_update_freq steps.

  • _deterministic_loss – Whether the loss should be calculated deterministically (w/o the stochastic action sampling step). True only useful for continuous actions and for debugging.

  • _use_beta_distribution – Use a Beta-distribution instead of a SquashedGaussian for bounded, continuous action spaces (not recommended; for debugging only).

Returns

This updated AlgorithmConfig object.

Model-based RL

Dreamer

pytorch [paper] [implementation]

Dreamer is an image-only model-based RL method that learns by imagining trajectories in the future and is evaluated on the DeepMind Control Suite environments. RLlib’s Dreamer is adapted from the official Google research repo.

To visualize learning, RLlib Dreamer’s imagined trajectories are logged as gifs in TensorBoard. Examples of such can be seen here.

Tuned examples: Deepmind Control Environments

Deepmind Control results @1M steps: more details

DMC env

RLlib Dreamer

Danijar et al Dreamer

Walker-Walk

920

~930

Cheetah-Run

640

~800

Dreamer-specific configs (see also common configs):

class ray.rllib.algorithms.dreamer.dreamer.DreamerConfig[source]

Defines a configuration class from which a Dreamer Algorithm can be built.

Example

>>> from ray.rllib.algorithms.dreamer import DreamerConfig
>>> config = DreamerConfig().training(gamma=0.9, lr=0.01)  
>>> config = config.resources(num_gpus=0)  
>>> config = config.rollouts(num_rollout_workers=4)  
>>> print(config.to_dict())  
>>> # Build a Algorithm object from the config and run 1 training iteration.
>>> algo = config.build(env="CartPole-v1")  
>>> algo.train()  

Example

>>> from ray import air
>>> from ray import tune
>>> from ray.rllib.algorithms.dreamer import DreamerConfig
>>> config = DreamerConfig()
>>> # Print out some default values.
>>> print(config.clip_param)  
>>> # Update the config object.
>>> config = config.training(  
...     lr=tune.grid_search([0.001, 0.0001]), clip_param=0.2)
>>> # Set the config object's env.
>>> config = config.environment(env="CartPole-v1")  
>>> # Use to_dict() to get the old-style python config dict
>>> # when running with tune.
>>> tune.Tuner(  
...     "Dreamer",
...     run_config=air.RunConfig(stop={"episode_reward_mean": 200}),
...     param_space=config.to_dict(),
... ).fit()
training(*, td_model_lr: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, actor_lr: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, critic_lr: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, grad_clip: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, lambda_: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, dreamer_train_iters: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, batch_size: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, batch_length: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, imagine_horizon: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, free_nats: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, kl_coeff: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, prefill_timesteps: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, explore_noise: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, dreamer_model: Optional[dict] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, num_steps_sampled_before_learning_starts: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, **kwargs) ray.rllib.algorithms.dreamer.dreamer.DreamerConfig[source]
Parameters
  • td_model_lr – PlaNET (transition dynamics) model learning rate.

  • actor_lr – Actor model learning rate.

  • critic_lr – Critic model learning rate.

  • grad_clip – If specified, clip the global norm of gradients by this amount.

  • lambda – The GAE (lambda) parameter.

  • dreamer_train_iters – Training iterations per data collection from real env.

  • batch_size – Number of episodes to sample for loss calculation.

  • batch_length – Length of each episode to sample for loss calculation.

  • imagine_horizon – Imagination horizon for training Actor and Critic.

  • free_nats – Free nats.

  • kl_coeff – KL coefficient for the model Loss.

  • prefill_timesteps – Prefill timesteps.

  • explore_noise – Exploration Gaussian noise.

  • dreamer_model – Custom model config.

  • num_steps_sampled_before_learning_starts – Number of timesteps to collect from rollout workers before we start sampling from replay buffers for learning. Whether we count this in agent steps or environment steps depends on config[“multiagent”][“count_steps_by”].

Returns:

Model-Based Meta-Policy-Optimization (MB-MPO)

pytorch [paper] [implementation]

RLlib’s MBMPO implementation is a Dyna-styled model-based RL method that learns based on the predictions of an ensemble of transition-dynamics models. Similar to MAML, MBMPO metalearns an optimal policy by treating each dynamics model as a different task. Code here is adapted from https://github.com/jonasrothfuss/model_ensemble_meta_learning. Similar to the original paper, MBMPO is evaluated on MuJoCo, with the horizon set to 200 instead of the default 1000.

Additional statistics are logged in MBMPO. Each MBMPO iteration corresponds to multiple MAML iterations, and MAMLIter$i$_DynaTrajInner_$j$_episode_reward_mean measures the agent’s returns across the dynamics models at iteration i of MAML and step j of inner adaptation. Examples can be seen here.

Tuned examples (continuous actions): Pendulum-v1, HalfCheetah, Hopper, Tuned examples (discrete actions): CartPole-v1

MuJoCo results @100K steps: more details

MuJoCo env

RLlib MBMPO

Clavera et al MBMPO

HalfCheetah

520

~550

Hopper

620

~650

MBMPO-specific configs (see also common configs):

class ray.rllib.algorithms.mbmpo.mbmpo.MBMPOConfig(algo_class=None)[source]

Defines a configuration class from which an MBMPO Algorithm can be built.

Example

>>> from ray.rllib.algorithms.mbmpo import MBMPOConfig
>>> config = MBMPOConfig()
>>> config = config.training(lr=0.0003, train_batch_size=512)  
>>> config = config.resources(num_gpus=4) 
>>> config = config.rollouts(num_rollout_workers=64)  
>>> print(config.to_dict())  
>>> # Build a Algorithm object from the config and run 1 training iteration.
>>> algo = config.build(env="CartPole-v1")  
>>> algo.train()  

Example

>>> from ray.rllib.algorithms.mbmpo import MBMPOConfig
>>> from ray import air
>>> from ray import tune
>>> config = MBMPOConfig()
>>> # Print out some default values.
>>> print(config.vtrace)  
>>> # Update the config object.
>>> config = config\  
...     .training(lr=tune.grid_search([0.0001, 0.0003]), grad_clip=20.0)
>>> # Set the config object's env.
>>> config = config.environment(env="CartPole-v1")  
>>> # Use to_dict() to get the old-style python config dict
>>> # when running with tune.
>>> tune.Tuner(  
...     "AlphaStar",
...     run_config=air.RunConfig(stop={"episode_reward_mean": 200}),
...     param_space=config.to_dict(),
... ).fit()
training(*, use_gae: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, lambda_: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, kl_coeff: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, vf_loss_coeff: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, entropy_coeff: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, clip_param: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, vf_clip_param: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, grad_clip: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, kl_target: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, inner_adaptation_steps: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, maml_optimizer_steps: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, inner_lr: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, dynamics_model: Optional[dict] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, custom_vector_env: Optional[type] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, num_maml_steps: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, **kwargs) ray.rllib.algorithms.mbmpo.mbmpo.MBMPOConfig[source]

Sets the training related configuration.

Parameters
  • use_gae – If true, use the Generalized Advantage Estimator (GAE) with a value function, see https://arxiv.org/pdf/1506.02438.pdf.

  • lambda – The GAE (lambda) parameter.

  • kl_coeff – Initial coefficient for KL divergence.

  • vf_loss_coeff – Coefficient of the value function loss.

  • entropy_coeff – Coefficient of the entropy regularizer.

  • clip_param – PPO clip parameter.

  • vf_clip_param – Clip param for the value function. Note that this is sensitive to the scale of the rewards. If your expected V is large, increase this.

  • grad_clip – If specified, clip the global norm of gradients by this amount.

  • kl_target – Target value for KL divergence.

  • inner_adaptation_steps – Number of Inner adaptation steps for the MAML algorithm.

  • maml_optimizer_steps – Number of MAML steps per meta-update iteration (PPO steps).

  • inner_lr – Inner adaptation step size.

  • dynamics_model – Dynamics ensemble hyperparameters.

  • custom_vector_env – Workers sample from dynamics models, not from actual envs.

  • num_maml_steps – How many iterations through MAML per MBMPO iteration.

Returns

This updated AlgorithmConfig object.

Derivative-free

Augmented Random Search (ARS)

pytorch tensorflow [paper] [implementation] ARS is a random search method for training linear policies for continuous control problems. Code here is adapted from https://github.com/modestyachts/ARS to integrate with RLlib APIs.

Tuned examples: CartPole-v1, Swimmer-v2

ARS-specific configs (see also common configs):

class ray.rllib.algorithms.ars.ars.ARSConfig[source]

Defines a configuration class from which an ARS Algorithm can be built.

Example

>>> from ray.rllib.algorithms.ars import ARSConfig
>>> config = ARSConfig()  
>>> config = config.training(report_length=20) 
>>> config = config.resources(num_gpus=0)  
>>> config = config.rollouts(num_rollout_workers=4)  
>>> config = config.environment("CartPole-v1")  
>>> print(config.to_dict())  
>>> # Build a Algorithm object from the config and run 1 training iteration.
>>> algo = config.build()  
>>> algo.train()  

Example

>>> from ray.rllib.algorithms.ars import ARSConfig
>>> from ray import air
>>> from ray import tune
>>> config = ARSConfig()
>>> # Print out some default values.
>>> print(config.action_noise_std)  
>>> # Update the config object.
>>> config = config.training(  
...     rollouts_used=tune.grid_search([32, 64]), eval_prob=0.5)
>>> # Set the config object's env.
>>> config = config.environment(env="CartPole-v1")  
>>> # Use to_dict() to get the old-style python config dict
>>> # when running with tune.
>>> tune.Tuner(  
...     "ARS",
...     run_config=air.RunConfig(stop={"episode_reward_mean": 200}),
...     param_space=config.to_dict(),
... ).fit()
training(*, action_noise_std: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, noise_stdev: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, num_rollouts: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, rollouts_used: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, sgd_stepsize: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, noise_size: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, eval_prob: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, report_length: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, offset: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, tf_single_threaded: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, **kwargs) ray.rllib.algorithms.ars.ars.ARSConfig[source]

Sets the training related configuration.

Parameters
  • action_noise_std – Std. deviation to be used when adding (standard normal) noise to computed actions. Action noise is only added, if compute_actions is called with the add_noise arg set to True.

  • noise_stdev – Std. deviation of parameter noise.

  • num_rollouts – Number of perturbs to try.

  • rollouts_used – Number of perturbs to keep in gradient estimate.

  • sgd_stepsize – SGD step-size used for the Adam optimizer.

  • noise_size – Number of rows in the noise table (shared across workers). Each row contains a gaussian noise value for each model parameter.

  • eval_prob – Probability of evaluating the parameter rewards.

  • report_length – How many of the last rewards we average over.

  • offset – Value to subtract from the reward (e.g. survival bonus from humanoid) during rollouts.

  • tf_single_threaded – Whether the tf-session should be generated without any parallelism options.

Returns

This updated AlgorithmConfig object.

Evolution Strategies (ES)

pytorch tensorflow [paper] [implementation] Code here is adapted from https://github.com/openai/evolution-strategies-starter to execute in the distributed setting with Ray.

Tuned examples: Humanoid-v1

Scalability:

../_images/es.png

RLlib’s ES implementation scales further and is faster than a reference Redis implementation on solving the Humanoid-v1 task.

ES-specific configs (see also common configs):

class ray.rllib.algorithms.es.es.ESConfig[source]

Defines a configuration class from which an ES Algorithm can be built.

Example

>>> from ray.rllib.algorithms.es import ESConfig
>>> config = ESConfig()  
>>> config = config.training(sgd_stepsize=0.02, report_length=20)
>>> config = config.resources(num_gpus=0)  
>>> config = config.rollouts(num_rollout_workers=4)  
>>> print(config.to_dict())  
>>> # Build a Algorithm object from the config and run 1 training iteration.
>>> algo = config.build(env="CartPole-v1")  
>>> algo.train()  

Example

>>> from ray.rllib.algorithms.es import ESConfig
>>> from ray import tune
>>> config = ESConfig()
>>> # Print out some default values.
>>> print(config.action_noise_std)  
>>> # Update the config object.
>>> config = config.training(  
...     rollouts_used=tune.grid_search([32, 64]), eval_prob=0.5)
>>> # Set the config object's env.
>>> config = config.environment(env="CartPole-v1")  
>>> # Use to_dict() to get the old-style python config dict
>>> # when running with tune.
>>> tune.Tuner(  
...     "ES",
...     run_config=ray.air.RunConfig(stop={"episode_reward_mean": 200}),
...     param_space=config.to_dict(),
... ).fit()
training(*, action_noise_std: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, l2_coeff: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, noise_stdev: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, episodes_per_batch: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, eval_prob: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, stepsize: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, noise_size: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, report_length: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, tf_single_threaded: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, **kwargs) ray.rllib.algorithms.es.es.ESConfig[source]

Sets the training related configuration.

Parameters
  • action_noise_std – Std. deviation to be used when adding (standard normal) noise to computed actions. Action noise is only added, if compute_actions is called with the add_noise arg set to True.

  • l2_coeff – Coefficient to multiply current weights with inside the globalg optimizer update term.

  • noise_stdev – Std. deviation of parameter noise.

  • episodes_per_batch – Minimum number of episodes to pack into the train batch.

  • eval_prob – Probability of evaluating the parameter rewards.

  • stepsize – SGD step-size used for the Adam optimizer.

  • noise_size – Number of rows in the noise table (shared across workers). Each row contains a gaussian noise value for each model parameter.

  • report_length – How many of the last rewards we average over.

  • tf_single_threaded – Whether the tf-session should be generated without any parallelism options.

Returns

This updated AlgorithmConfig object.

RL for recommender systems

SlateQ

pytorch [paper] [implementation]

SlateQ is a model-free RL method that builds on top of DQN and generates recommendation slates for recommender system environments. Since these types of environments come with large combinatorial action spaces, SlateQ mitigates this by decomposing the Q-value into single-item Q-values and solves the decomposed objective via mixing integer programming and deep learning optimization. SlateQ can be evaluated on Google’s RecSim environment. An RLlib wrapper for RecSim can be found here <.

RecSim environment wrapper: Google RecSim

SlateQ-specific configs (see also common configs):

class ray.rllib.algorithms.slateq.slateq.SlateQConfig[source]

Defines a configuration class from which a SlateQ Algorithm can be built.

Example

>>> from ray.rllib.algorithms.slateq import SlateQConfig
>>> config = SlateQConfig().training(lr=0.01).resources(num_gpus=1)
>>> print(config.to_dict())  
>>> # Build a Algorithm object from the config and run 1 training iteration.
>>> algo = config.build(env="CartPole-v1")  
>>> algo.train()  

Example

>>> from ray.rllib.algorithms.slateq import SlateQConfig
>>> from ray import air
>>> from ray import tune
>>> config = SlateQConfig()
>>> # Print out some default values.
>>> print(config.lr)  
>>> # Update the config object.
>>> config = config.training(  
...     lr=tune.grid_search([0.001, 0.0001]))
>>> # Set the config object's env.
>>> config = config.environment(env="CartPole-v1")  
>>> # Use to_dict() to get the old-style python config dict
>>> # when running with tune.
>>> tune.Tuner(  
...     "SlateQ",
...     run_config=air.RunConfig(stop={"episode_reward_mean": 160.0}),
...     param_space=config.to_dict(),
... ).fit()
training(*, replay_buffer_config: Optional[Dict[str, Any]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, fcnet_hiddens_per_candidate: Optional[List[int]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, target_network_update_freq: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, tau: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, use_huber: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, huber_threshold: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, training_intensity: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, lr_schedule: Optional[List[List[Union[int, float]]]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, lr_choice_model: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, rmsprop_epsilon: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, grad_clip: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, n_step: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, num_steps_sampled_before_learning_starts: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, **kwargs) ray.rllib.algorithms.slateq.slateq.SlateQConfig[source]

Sets the training related configuration.

Parameters
  • replay_buffer_config – The config dict to specify the replay buffer used. May contain a type key (default: MultiAgentPrioritizedReplayBuffer) indicating the class being used. All other keys specify the names and values of kwargs passed to to this class’ constructor.

  • fcnet_hiddens_per_candidate – Dense-layer setup for each the n (document) candidate Q-network stacks.

  • target_network_update_freq – Update the target network every target_network_update_freq sample steps.

  • tau – Update the target by au * policy + (1- au) * target_policy.

  • use_huber – If True, use huber loss instead of squared loss for critic network. Conventionally, no need to clip gradients if using a huber loss.

  • huber_threshold – The threshold for the Huber loss.

  • training_intensity – If set, this will fix the ratio of replayed from a buffer and learned on timesteps to sampled from an environment and stored in the replay buffer timesteps. Otherwise, the replay will proceed at the native ratio determined by (train_batch_size / rollout_fragment_length).

  • lr_schedule – Learning rate schedule. In the format of [[timestep, lr-value], [timestep, lr-value], …] Intermediary timesteps will be assigned to interpolated learning rate values. A schedule should normally start from timestep 0.

  • lr_choice_model – Learning rate for adam optimizer for the user choice model. So far, only relevant/supported for framework=torch.

  • rmsprop_epsilon – RMSProp epsilon hyperparameter.

  • grad_clip – If not None, clip gradients during optimization at this value.

  • n_step – N-step parameter for Q-learning.

Returns

This updated AlgorithmConfig object.

Contextual Bandits

The Multi-armed bandit (MAB) problem provides a simplified RL setting that involves learning to act under one situation only, i.e. the context (observation/state) and arms (actions/items-to-select) are both fixed. Contextual bandit is an extension of the MAB problem, where at each round the agent has access not only to a set of bandit arms/actions but also to a context (state) associated with this iteration. The context changes with each iteration, but, is not affected by the action that the agent takes. The objective of the agent is to maximize the cumulative rewards, by collecting enough information about how the context and the rewards of the arms are related to each other. The agent does this by balancing the trade-off between exploration and exploitation.

Contextual bandit algorithms typically consist of an action-value model (Q model) and an exploration strategy (epsilon-greedy, LinUCB, Thompson Sampling etc.)

RLlib supports the following online contextual bandit algorithms, named after the exploration strategies that they employ:

Linear Upper Confidence Bound (BanditLinUCB)

pytorch [paper] [implementation] LinUCB assumes a linear dependency between the expected reward of an action and its context. It estimates the Q value of each action using ridge regression. It constructs a confidence region around the weights of the linear regression model and uses this confidence ellipsoid to estimate the uncertainty of action values.

Tuned examples: SimpleContextualBandit, UCB Bandit on RecSim. ParametricItemRecoEnv.

LinUCB-specific configs (see also common configs):

class ray.rllib.algorithms.bandit.bandit.BanditLinUCBConfig[source]

Defines a config class from which an upper confidence bound bandit can be built.

Example

>>> from ray.rllib.algorithms.bandit import BanditLinUCBConfig
>>> from ray.rllib.examples.env.bandit_envs_discrete import WheelBanditEnv
>>> config = BanditLinUCBConfig()  
>>> config = config.rollouts(num_rollout_workers=4) 
>>> print(config.to_dict())  
>>> # Build a Algorithm object from the config and run 1 training iteration.
>>> algo = config.build(env=WheelBanditEnv)  
>>> algo.train()  
training(gamma: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, lr: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, train_batch_size: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, model: Optional[dict] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, optimizer: Optional[dict] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, max_requests_in_flight_per_sampler_worker: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>) ray.rllib.algorithms.algorithm_config.AlgorithmConfig

Sets the training related configuration.

Parameters
  • gamma – Float specifying the discount factor of the Markov Decision process.

  • lr – The default learning rate.

  • train_batch_size – Training batch size, if applicable.

  • model – Arguments passed into the policy model. See models/catalog.py for a full list of the available model options. TODO: Provide ModelConfig objects instead of dicts.

  • optimizer – Arguments to pass to the policy optimizer.

  • max_requests_in_flight_per_sampler_worker – Max number of inflight requests to each sampling worker. See the FaultTolerantActorManager class for more details. Tuning these values is important when running experimens with large sample batches, where there is the risk that the object store may fill up, causing spilling of objects to disk. This can cause any asynchronous requests to become very slow, making your experiment run slow as well. You can inspect the object store during your experiment via a call to ray memory on your headnode, and by using the ray dashboard. If you’re seeing that the object store is filling up, turn down the number of remote requests in flight, or enable compression in your experiment of timesteps.

Returns

This updated AlgorithmConfig object.

Linear Thompson Sampling (BanditLinTS)

pytorch [paper] [implementation] Like LinUCB, LinTS also assumes a linear dependency between the expected reward of an action and its context and uses online ridge regression to estimate the Q values of actions given the context. It assumes a Gaussian prior on the weights and a Gaussian likelihood function. For deciding which action to take, the agent samples weights for each arm, using the posterior distributions, and plays the arm that produces the highest reward.

Tuned examples: SimpleContextualBandit, WheelBandit.

LinTS-specific configs (see also common configs):

class ray.rllib.algorithms.bandit.bandit.BanditLinTSConfig[source]

Defines a configuration class from which a Thompson-sampling bandit can be built.

Example

>>> from ray.rllib.algorithms.bandit import BanditLinTSConfig 
>>> from ray.rllib.examples.env.bandit_envs_discrete import WheelBanditEnv
>>> config = BanditLinTSConfig().rollouts(num_rollout_workers=4)
>>> print(config.to_dict())  
>>> # Build a Algorithm object from the config and run 1 training iteration.
>>> algo = config.build(env=WheelBanditEnv)  
>>> algo.train()  
training(gamma: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, lr: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, train_batch_size: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, model: Optional[dict] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, optimizer: Optional[dict] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, max_requests_in_flight_per_sampler_worker: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>) ray.rllib.algorithms.algorithm_config.AlgorithmConfig

Sets the training related configuration.

Parameters
  • gamma – Float specifying the discount factor of the Markov Decision process.

  • lr – The default learning rate.

  • train_batch_size – Training batch size, if applicable.

  • model – Arguments passed into the policy model. See models/catalog.py for a full list of the available model options. TODO: Provide ModelConfig objects instead of dicts.

  • optimizer – Arguments to pass to the policy optimizer.

  • max_requests_in_flight_per_sampler_worker – Max number of inflight requests to each sampling worker. See the FaultTolerantActorManager class for more details. Tuning these values is important when running experimens with large sample batches, where there is the risk that the object store may fill up, causing spilling of objects to disk. This can cause any asynchronous requests to become very slow, making your experiment run slow as well. You can inspect the object store during your experiment via a call to ray memory on your headnode, and by using the ray dashboard. If you’re seeing that the object store is filling up, turn down the number of remote requests in flight, or enable compression in your experiment of timesteps.

Returns

This updated AlgorithmConfig object.

Multi-agent

Parameter Sharing

[paper], [paper] and [instructions]. Parameter sharing refers to a class of methods that take a base single agent method, and use it to learn a single policy for all agents. This simple approach has been shown to achieve state of the art performance in cooperative games, and is usually how you should start trying to learn a multi-agent problem.

Tuned examples: PettingZoo, waterworld, rock-paper-scissors, multi-agent cartpole

QMIX Monotonic Value Factorisation (QMIX, VDN, IQN)

pytorch [paper] [implementation] Q-Mix is a specialized multi-agent algorithm. Code here is adapted from https://github.com/oxwhirl/pymarl_alpha to integrate with RLlib multi-agent APIs. To use Q-Mix, you must specify an agent grouping in the environment (see the two-step game example). Currently, all agents in the group must be homogeneous. The algorithm can be scaled by increasing the number of workers or using Ape-X.

Tuned examples: Two-step game

QMIX-specific configs (see also common configs):

class ray.rllib.algorithms.qmix.qmix.QMixConfig[source]

Defines a configuration class from which QMix can be built.

Example

>>> from ray.rllib.examples.env.two_step_game import TwoStepGame
>>> from ray.rllib.algorithms.qmix import QMixConfig
>>> config = QMixConfig()  
>>> config = config.training(gamma=0.9, lr=0.01, kl_coeff=0.3)  
>>> config = config.resources(num_gpus=0)  
>>> config = config.rollouts(num_rollout_workers=4)  
>>> print(config.to_dict())  
>>> # Build an Algorithm object from the config and run 1 training iteration.
>>> algo = config.build(env=TwoStepGame)  
>>> algo.train()  

Example

>>> from ray.rllib.examples.env.two_step_game import TwoStepGame
>>> from ray.rllib.algorithms.qmix import QMixConfig
>>> from ray import air
>>> from ray import tune
>>> config = QMixConfig()
>>> # Print out some default values.
>>> print(config.optim_alpha)  
>>> # Update the config object.
>>> config.training(  
...     lr=tune.grid_search([0.001, 0.0001]), optim_alpha=0.97
... )
>>> # Set the config object's env.
>>> config.environment(env=TwoStepGame)  
>>> # Use to_dict() to get the old-style python config dict
>>> # when running with tune.
>>> tune.Tuner(  
...     "QMix",
...     run_config=air.RunConfig(stop={"episode_reward_mean": 200}),
...     param_space=config.to_dict(),
... ).fit()
training(*, mixer: Optional[str] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, mixing_embed_dim: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, double_q: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, target_network_update_freq: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, replay_buffer_config: Optional[dict] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, optim_alpha: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, optim_eps: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, grad_clip: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, grad_norm_clipping=-1, **kwargs) ray.rllib.algorithms.qmix.qmix.QMixConfig[source]

Sets the training related configuration.

Parameters
  • mixer – Mixing network. Either “qmix”, “vdn”, or None.

  • mixing_embed_dim – Size of the mixing network embedding.

  • double_q – Whether to use Double_Q learning.

  • target_network_update_freq – Update the target network every target_network_update_freq sample steps.

  • replay_buffer_config

  • optim_alpha – RMSProp alpha.

  • optim_eps – RMSProp epsilon.

  • grad_clip – If not None, clip gradients during optimization at this value.

  • grad_norm_clipping – Depcrecated in favor of grad_clip

Returns

This updated AlgorithmConfig object.

Multi-Agent Deep Deterministic Policy Gradient (MADDPG)

tensorflow [paper] [implementation] MADDPG is a DDPG centralized/shared critic algorithm. Code here is adapted from https://github.com/openai/maddpg to integrate with RLlib multi-agent APIs. Please check justinkterry/maddpg-rllib for examples and more information. Note that the implementation here is based on OpenAI’s, and is intended for use with the discrete MPE environments. Please also note that people typically find this method difficult to get to work, even with all applicable optimizations for their environment applied. This method should be viewed as for research purposes, and for reproducing the results of the paper introducing it.

MADDPG-specific configs (see also common configs):

Tuned examples: Multi-Agent Particle Environment, Two-step game

class ray.rllib.algorithms.maddpg.maddpg.MADDPGConfig(algo_class=None)[source]

Defines a configuration class from which a MADDPG Algorithm can be built.

Example

>>> from ray.rllib.algorithms.maddpg.maddpg import MADDPGConfig
>>> config = MADDPGConfig()
>>> print(config.replay_buffer_config)  
>>> replay_config = config.replay_buffer_config.update(  
...     {
...         "capacity": 100000,
...         "prioritized_replay_alpha": 0.8,
...         "prioritized_replay_beta": 0.45,
...         "prioritized_replay_eps": 2e-6,
...     }
... )
>>> config.training(replay_buffer_config=replay_config)   
>>> config = config.resources(num_gpus=0)   
>>> config = config.rollouts(num_rollout_workers=4)   
>>> config = config.environment("CartPole-v1")   
>>> algo = config.build()  
>>> algo.train()  

Example

>>> from ray.rllib.algorithms.maddpg.maddpg import MADDPGConfig
>>> from ray import air
>>> from ray import tune
>>> config = MADDPGConfig()
>>> config.training(n_step=tune.grid_search([3, 5]))  
>>> config.environment(env="CartPole-v1")  
>>> tune.Tuner(  
...     "MADDPG",
...     run_config=air.RunConfig(stop={"episode_reward_mean":200}),
...     param_space=config.to_dict()
... ).fit()
training(*, agent_id: Optional[str] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, use_local_critic: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, use_state_preprocessor: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, actor_hiddens: Optional[List[int]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, actor_hidden_activation: Optional[str] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, critic_hiddens: Optional[List[int]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, critic_hidden_activation: Optional[str] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, n_step: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, good_policy: Optional[str] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, adv_policy: Optional[str] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, replay_buffer_config: Optional[dict] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, training_intensity: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, num_steps_sampled_before_learning_starts: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, critic_lr: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, actor_lr: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, target_network_update_freq: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, tau: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, actor_feature_reg: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, grad_norm_clipping: Optional[float] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, **kwargs) ray.rllib.algorithms.maddpg.maddpg.MADDPGConfig[source]

Sets the training related configuration.

Parameters
  • agent_id – ID of the agent controlled by this policy.

  • use_local_critic – Use a local critic for this policy.

  • use_state_preprocessor – Apply a state preprocessor with spec given by the “model” config option (like other RL algorithms). This is mostly useful if you have a weird observation shape, like an image. Disabled by default.

  • actor_hiddens – Postprocess the policy network model output with these hidden layers. If use_state_preprocessor is False, then these will be the only hidden layers in the network.

  • actor_hidden_activation – Hidden layers activation of the postprocessing stage of the policy network.

  • critic_hiddens – Postprocess the critic network model output with these hidden layers; again, if use_state_preprocessor is True, then the state will be preprocessed by the model specified with the “model” config option first.

  • critic_hidden_activation – Hidden layers activation of the postprocessing state of the critic.

  • n_step – N-step for Q-learning.

  • good_policy – Algorithm for good policies.

  • adv_policy – Algorithm for adversary policies.

  • replay_buffer_config – Replay buffer config. Examples: { “_enable_replay_buffer_api”: True, “type”: “MultiAgentReplayBuffer”, “capacity”: 50000, “replay_sequence_length”: 1, } - OR - { “_enable_replay_buffer_api”: True, “type”: “MultiAgentPrioritizedReplayBuffer”, “capacity”: 50000, “prioritized_replay_alpha”: 0.6, “prioritized_replay_beta”: 0.4, “prioritized_replay_eps”: 1e-6, “replay_sequence_length”: 1, } - Where - prioritized_replay_alpha: Alpha parameter controls the degree of prioritization in the buffer. In other words, when a buffer sample has a higher temporal-difference error, with how much more probability should it drawn to use to update the parametrized Q-network. 0.0 corresponds to uniform probability. Setting much above 1.0 may quickly result as the sampling distribution could become heavily “pointy” with low entropy. prioritized_replay_beta: Beta parameter controls the degree of importance sampling which suppresses the influence of gradient updates from samples that have higher probability of being sampled via alpha parameter and the temporal-difference error. prioritized_replay_eps: Epsilon parameter sets the baseline probability for sampling so that when the temporal-difference error of a sample is zero, there is still a chance of drawing the sample.

  • training_intensity – If set, this will fix the ratio of replayed from a buffer and learned on timesteps to sampled from an environment and stored in the replay buffer timesteps. Otherwise, the replay will proceed at the native ratio determined by (train_batch_size / rollout_fragment_length).

  • num_steps_sampled_before_learning_starts – Number of timesteps to collect from rollout workers before we start sampling from replay buffers for learning. Whether we count this in agent steps or environment steps depends on config[“multiagent”][“count_steps_by”].

  • critic_lr – Learning rate for the critic (Q-function) optimizer.

  • actor_lr – Learning rate for the actor (policy) optimizer.

  • target_network_update_freq – Update the target network every target_network_update_freq sample steps.

  • tau – Update the target by au * policy + (1- au) * target_policy.

  • actor_feature_reg – Weights for feature regularization for the actor.

  • grad_norm_clipping – If not None, clip gradients during optimization at this value.

Returns

This updated AlgorithmConfig object.

Shared Critic Methods

[instructions] Shared critic methods are when all agents use a single parameter shared critic network (in some cases with access to more of the observation space than agents can see). Note that many specialized multi-agent algorithms such as MADDPG are mostly shared critic forms of their single-agent algorithm (DDPG in the case of MADDPG).

Tuned examples: TwoStepGame

Others

Single-Player Alpha Zero (AlphaZero)

pytorch [paper] [implementation] AlphaZero is an RL agent originally designed for two-player games. This version adapts it to handle single player games. The code can be scaled to any number of workers. It also implements the ranked rewards (R2) strategy to enable self-play even in the one-player setting. The code is mainly purposed to be used for combinatorial optimization.

Tuned examples: Sparse reward CartPole

AlphaZero-specific configs (see also common configs):

class ray.rllib.algorithms.alpha_zero.alpha_zero.AlphaZeroConfig(algo_class=None)[source]

Defines a configuration class from which an AlphaZero Algorithm can be built.

Example

>>> from ray.rllib.algorithms.alpha_zero import AlphaZeroConfig
>>> config = AlphaZeroConfig()   
>>> config = config.training(sgd_minibatch_size=256)   
>>> config = config..resources(num_gpus=0)   
>>> config = config..rollouts(num_rollout_workers=4)   
>>> print(config.to_dict()) 
>>> # Build a Algorithm object from the config and run 1 training iteration.
>>> algo = config.build(env="CartPole-v1")  
>>> algo.train() 

Example

>>> from ray.rllib.algorithms.alpha_zero import AlphaZeroConfig
>>> from ray import air
>>> from ray import tune
>>> config = AlphaZeroConfig()
>>> # Print out some default values.
>>> print(config.shuffle_sequences) 
>>> # Update the config object.
>>> config.training(lr=tune.grid_search([0.001, 0.0001]))  
>>> # Set the config object's env.
>>> config.environment(env="CartPole-v1")   
>>> # Use to_dict() to get the old-style python config dict
>>> # when running with tune.
>>> tune.Tuner( 
...     "AlphaZero",
...     run_config=air.RunConfig(stop={"episode_reward_mean": 200}),
...     param_space=config.to_dict(),
... ).fit()
training(*, sgd_minibatch_size: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, shuffle_sequences: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, num_sgd_iter: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, replay_buffer_config: Optional[dict] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, lr_schedule: Optional[List[List[Union[int, float]]]] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, vf_share_layers: Optional[bool] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, mcts_config: Optional[dict] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, ranked_rewards: Optional[dict] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, num_steps_sampled_before_learning_starts: Optional[int] = <ray.rllib.algorithms.algorithm_config._NotProvided object>, **kwargs) ray.rllib.algorithms.alpha_zero.alpha_zero.AlphaZeroConfig[source]

Sets the training related configuration.

Parameters
  • sgd_minibatch_size – Total SGD batch size across all devices for SGD.

  • shuffle_sequences – Whether to shuffle sequences in the batch when training (recommended).

  • num_sgd_iter – Number of SGD iterations in each outer loop.

  • replay_buffer_config – Replay buffer config. Examples: { “_enable_replay_buffer_api”: True, “type”: “MultiAgentReplayBuffer”, “learning_starts”: 1000, “capacity”: 50000, “replay_sequence_length”: 1, } - OR - { “_enable_replay_buffer_api”: True, “type”: “MultiAgentPrioritizedReplayBuffer”, “capacity”: 50000, “prioritized_replay_alpha”: 0.6, “prioritized_replay_beta”: 0.4, “prioritized_replay_eps”: 1e-6, “replay_sequence_length”: 1, } - Where - prioritized_replay_alpha: Alpha parameter controls the degree of prioritization in the buffer. In other words, when a buffer sample has a higher temporal-difference error, with how much more probability should it drawn to use to update the parametrized Q-network. 0.0 corresponds to uniform probability. Setting much above 1.0 may quickly result as the sampling distribution could become heavily “pointy” with low entropy. prioritized_replay_beta: Beta parameter controls the degree of importance sampling which suppresses the influence of gradient updates from samples that have higher probability of being sampled via alpha parameter and the temporal-difference error. prioritized_replay_eps: Epsilon parameter sets the baseline probability for sampling so that when the temporal-difference error of a sample is zero, there is still a chance of drawing the sample.

  • lr_schedule – Learning rate schedule. In the format of [[timestep, lr-value], [timestep, lr-value], …] Intermediary timesteps will be assigned to interpolated learning rate values. A schedule should normally start from timestep 0.

  • vf_share_layers – Share layers for value function. If you set this to True, it’s important to tune vf_loss_coeff.

  • mcts_config – MCTS specific settings.

  • ranked_rewards – Settings for the ranked reward (r2) algorithm from: https://arxiv.org/pdf/1807.01672.pdf

  • num_steps_sampled_before_learning_starts – Number of timesteps to collect from rollout workers before we start sampling from replay buffers for learning. Whether we count this in agent steps or environment steps depends on config[“multiagent”][“count_steps_by”].

Returns

This updated AlgorithmConfig object.

Curiosity (ICM: Intrinsic Curiosity Module)

pytorch [paper] [implementation]

Tuned examples: Pyramids (Unity3D) (use --env Pyramids command line option) Test case with MiniGrid example (UnitTest case: test_curiosity_on_partially_observable_domain)

Activating Curiosity The curiosity plugin can be easily activated by specifying it as the Exploration class to-be-used in the main Algorithm config. Most of its parameters usually do not have to be specified as the module uses the values from the paper by default. For example:

config = ppo.DEFAULT_CONFIG.copy()
config["num_workers"] = 0
config["exploration_config"] = {
    "type": "Curiosity",  # <- Use the Curiosity module for exploring.
    "eta": 1.0,  # Weight for intrinsic rewards before being added to extrinsic ones.
    "lr": 0.001,  # Learning rate of the curiosity (ICM) module.
    "feature_dim": 288,  # Dimensionality of the generated feature vectors.
    # Setup of the feature net (used to encode observations into feature (latent) vectors).
    "feature_net_config": {
        "fcnet_hiddens": [],
        "fcnet_activation": "relu",
    },
    "inverse_net_hiddens": [256],  # Hidden layers of the "inverse" model.
    "inverse_net_activation": "relu",  # Activation of the "inverse" model.
    "forward_net_hiddens": [256],  # Hidden layers of the "forward" model.
    "forward_net_activation": "relu",  # Activation of the "forward" model.
    "beta": 0.2,  # Weight for the "forward" loss (beta) over the "inverse" loss (1.0 - beta).
    # Specify, which exploration sub-type to use (usually, the algo's "default"
    # exploration, e.g. EpsilonGreedy for DQN, StochasticSampling for PG/SAC).
    "sub_exploration": {
        "type": "StochasticSampling",
    }
}

Functionality RLlib’s Curiosity is based on “ICM” (intrinsic curiosity module) described in this paper here. It allows agents to learn in sparse-reward- or even no-reward environments by calculating so-called “intrinsic rewards”, purely based on the information content that is incoming via the observation channel. Sparse-reward environments are envs where almost all reward signals are 0.0, such as these [MiniGrid env examples here]. In such environments, agents have to navigate (and change the underlying state of the environment) over long periods of time, without receiving much (or any) feedback. For example, the task could be to find a key in some room, pick it up, find a matching door (matching the color of the key), and eventually unlock this door with the key to reach a goal state, all the while not seeing any rewards. Such problems are impossible to solve with standard RL exploration methods like epsilon-greedy or stochastic sampling. The Curiosity module - when configured as the Exploration class to use via the Algorithm’s config (see above on how to do this) - automatically adds three simple models to the Policy’s self.model: a) a latent space learning (“feature”) model, taking an environment observation and outputting a latent vector, which represents this observation and b) a “forward” model, predicting the next latent vector, given the current observation vector and an action to take next. c) a so-called “inverse” net, only used to train the “feature” net. The inverse net tries to predict the action taken between two latent vectors (obs and next obs).

All the above extra Models are trained inside the modified Exploration.postprocess_trajectory() call.

Using the (ever changing) “forward” model, our Curiosity module calculates an artificial (intrinsic) reward signal, weights it via the eta parameter, and then adds it to the environment’s (extrinsic) reward. Intrinsic rewards for each env-step are calculated by taking the euclidian distance between the latent-space encoded next observation (“feature” model) and the predicted latent-space encoding for the next observation (“forward” model). This allows the agent to explore areas of the environment, where the “forward” model still performs poorly (are not “understood” yet), whereas exploration to these areas will taper down after the agent has visited them often: The “forward” model will eventually get better at predicting these next latent vectors, which in turn will diminish the intrinsic rewards (decrease the euclidian distance between predicted and actual vectors).

RE3 (Random Encoders for Efficient Exploration)

tensorflow [paper] [implementation]

Examples: LunarLanderContinuous-v2 (use --env LunarLanderContinuous-v2 command line option) Test case with Pendulum-v1 example

Activating RE3 The RE3 plugin can be easily activated by specifying it as the Exploration class to-be-used in the main Algorithm config and inheriting the RE3UpdateCallbacks as shown in this example. Most of its parameters usually do not have to be specified as the module uses the values from the paper by default. For example:

config = sac.DEFAULT_CONFIG.copy()
config["env"] = "Pendulum-v1"
config["seed"] = 12345
config["callbacks"] = RE3Callbacks
config["exploration_config"] = {
    "type": "RE3",
     # the dimensionality of the observation embedding vectors in latent space.
     "embeds_dim": 128,
     "rho": 0.1, # Beta decay factor, used for on-policy algorithm.
     "k_nn": 50, # Number of neighbours to set for K-NN entropy estimation.
     # Configuration for the encoder network, producing embedding vectors from observations.
     # This can be used to configure fcnet- or conv_net setups to properly process any
     # observation space. By default uses the Policy model configuration.
     "encoder_net_config": {
         "fcnet_hiddens": [],
         "fcnet_activation": "relu",
     },
     # Hyperparameter to choose between exploration and exploitation. A higher value of beta adds
     # more importance to the intrinsic reward, as per the following equation
     # `reward = r + beta * intrinsic_reward`
     "beta": 0.2,
     # Schedule to use for beta decay, one of constant" or "linear_decay".
     "beta_schedule": 'constant',
     # Specify, which exploration sub-type to use (usually, the algo's "default"
     # exploration, e.g. EpsilonGreedy for DQN, StochasticSampling for PG/SAC).
     "sub_exploration": {
         "type": "StochasticSampling",
     }
}

Functionality RLlib’s RE3 is based on “Random Encoders for Efficient Exploration” described in this paper here. RE3 quantifies exploration based on state entropy. The entropy of a state is calculated based on its distance from K nearest neighbor states present in the replay buffer in the latent space (With this implementation, KNN is implemented using training samples from the same batch). The state entropy is considered as an intrinsic reward and for policy optimization added to the extrinsic reward when available. If the extrinsic reward is not available then the state entropy is used as “intrinsic reward” for unsupervised pre-training of the RL agent. RE3 further allows agents to learn in sparse-reward or even no-reward environments by using the state entropy as “intrinsic rewards”.

This exploration objective can be used with both model-free and model-based RL algorithms. RE3 uses a randomly initialized encoder to get the state’s latent representation, thus taking away the complexity of training the representation learning method. The encoder weights are fixed during the entire duration of the training process.

Fully Independent Learning

[instructions] Fully independent learning involves a collection of agents learning independently of each other via single agent methods. This typically works, but can be less effective than dedicated multi-agent RL methods, since they do not account for the non-stationarity of the multi-agent environment.

Tuned examples: waterworld, multiagent-cartpole