Note

Ray 2.10.0 introduces the alpha stage of RLlib’s “new API stack”. The Ray Team plans to transition algorithms, example scripts, and documentation to the new code base thereby incrementally replacing the “old API stack” (e.g., ModelV2, Policy, RolloutWorker) throughout the subsequent minor releases leading up to Ray 3.0.

Note, however, that so far only PPO (single- and multi-agent) and SAC (single-agent only) support the “new API stack” and continue to run by default with the old APIs. You can continue to use the existing custom (old stack) classes.

See here for more details on how to use the new API stack.

RLlib Utilities#

Here is a list of all the utilities available in RLlib.

Exploration API#

Exploration is crucial in RL for enabling a learning agent to find new, potentially high-reward states by reaching unexplored areas of the environment.

RLlib has several built-in exploration components that the different algorithms use. You can also customize an algorithm’s exploration behavior by sub-classing the Exploration base class and implementing your own logic:

Built-in Exploration components#

Exploration

Implements an exploration strategy for Policies.

Random

A random action selector (deterministic/greedy for explore=False).

StochasticSampling

An exploration that simply samples from a distribution.

EpsilonGreedy

Epsilon-greedy Exploration class that produces exploration actions.

GaussianNoise

An exploration that adds white noise to continuous actions.

OrnsteinUhlenbeckNoise

An exploration that adds Ornstein-Uhlenbeck noise to continuous actions.

RE3

Random Encoder for Efficient Exploration.

Curiosity

Implementation of: [1] Curiosity-driven Exploration by Self-supervised Prediction Pathak, Agrawal, Efros, and Darrell - UC Berkeley - ICML 2017.

ParameterNoise

An exploration that changes a Model's parameters.

Inference#

get_exploration_action

Returns a (possibly) exploratory action and its log-likelihood.

Callback hooks#

before_compute_actions

Hook for preparations before policy.compute_actions() is called.

on_episode_start

Handles necessary exploration logic at the beginning of an episode.

on_episode_end

Handles necessary exploration logic at the end of an episode.

postprocess_trajectory

Handles post-processing of done episode trajectories.

Setting and getting states#

get_state

Returns the current exploration state.

set_state

Sets the Exploration object's state to the given values.

Scheduler API#

Use a scheduler to set scheduled values for variables (in Python, PyTorch, or TensorFlow) based on an (int64) timestep input. The computed values are usually float32 types.

Built-in Scheduler components#

Schedule

Schedule classes implement various time-dependent scheduling schemas.

ConstantSchedule

A Schedule where the value remains constant over time.

LinearSchedule

Linear interpolation between initial_p and final_p.

PiecewiseSchedule

Implements a Piecewise Scheduler.

ExponentialSchedule

Exponential decay schedule from initial_p to final_p.

PolynomialSchedule

Polynomial interpolation between initial_p and final_p.

Methods#

value

Generates the value given a timestep (based on schedule's logic).

__call__

Simply calls self.value(t).

Training Operations Utilities#

multi_gpu_train_one_step

Multi-GPU version of train_one_step.

train_one_step

Function that improves the all policies in train_batch on the local worker.

Framework Utilities#

Import utilities#

try_import_torch

Tries importing torch and returns the module (or None).

try_import_tf

Tries importing tf and returns the module (or None).

try_import_tfp

Tries importing tfp and returns the module (or None).

Tensorflow utilities#

explained_variance

Computes the explained variance for a pair of labels and predictions.

flatten_inputs_to_1d_tensor

Flattens arbitrary input structs according to the given spaces struct.

get_gpu_devices

Returns a list of GPU device names, e.g. ["/gpu:0", "/gpu:1"].

get_placeholder

Returns a tf1.placeholder object given optional hints, such as a space.

huber_loss

Computes the huber loss for a given term and delta parameter.

l2_loss

Computes half the L2 norm over a tensor's values without the sqrt.

make_tf_callable

Returns a function that can be executed in either graph or eager mode.

minimize_and_clip

Computes, then clips gradients using objective, optimizer and var list.

one_hot

Returns a one-hot tensor, given and int tensor and a space.

reduce_mean_ignore_inf

Same as tf.reduce_mean() but ignores -inf values.

scope_vars

Get variables inside a given scope.

warn_if_infinite_kl_divergence

zero_logps_from_actions

Helper function useful for returning dummy logp's (0) for some actions.

Torch utilities#

apply_grad_clipping

Applies gradient clipping to already computed grads inside optimizer.

concat_multi_gpu_td_errors

Concatenates multi-GPU (per-tower) TD error tensors given TorchPolicy.

convert_to_torch_tensor

Converts any struct to torch.Tensors.

explained_variance

Computes the explained variance for a pair of labels and predictions.

flatten_inputs_to_1d_tensor

Flattens arbitrary input structs according to the given spaces struct.

global_norm

Returns the global L2 norm over a list of tensors.

huber_loss

Computes the huber loss for a given term and delta parameter.

l2_loss

Computes half the L2 norm over a tensor's values without the sqrt.

minimize_and_clip

Clips grads found in optimizer.param_groups to given value in place.

one_hot

Returns a one-hot tensor, given and int tensor and a space.

reduce_mean_ignore_inf

Same as torch.mean() but ignores -inf values.

sequence_mask

Offers same behavior as tf.sequence_mask for torch.

warn_if_infinite_kl_divergence

set_torch_seed

Sets the torch random seed to the given value.

softmax_cross_entropy_with_logits

Same behavior as tf.nn.softmax_cross_entropy_with_logits.

Numpy utilities#

aligned_array

concat_aligned

convert_to_numpy

Converts values in stats to non-Tensor numpy or python types.

fc

Calculates FC (dense) layer outputs given weights/biases and input.

flatten_inputs_to_1d_tensor

Flattens arbitrary input structs according to the given spaces struct.

make_action_immutable

Flags actions immutable to notify users when trying to change them.

huber_loss

Reference: https://en.wikipedia.org/wiki/Huber_loss.

l2_loss

Computes half the L2 norm of a tensor (w/o the sqrt): sum(x**2) / 2.

lstm

Calculates LSTM layer output given weights/biases, states, and input.

one_hot

One-hot utility function for numpy.

relu

Implementation of the leaky ReLU function.

sigmoid

Returns the sigmoid function applied to x.

softmax

Returns the softmax values for x.

Checkpoint utilities#

Checkpointable

Abstract base class for a component of RLlib that can be checkpointed to disk.

convert_to_msgpack_checkpoint

Converts an Algorithm checkpoint (pickle based) to a msgpack based one.

convert_to_msgpack_policy_checkpoint

Converts a Policy checkpoint (pickle based) to a msgpack based one.

get_checkpoint_info

Returns a dict with information about an Algorithm/Policy checkpoint.

try_import_msgpack

Tries importing msgpack and msgpack_numpy and returns the patched msgpack module.

Policy utilities#

compute_log_likelihoods_from_input_dict

Returns log likelihood for actions in given batch for policy.

create_policy_for_framework

Framework-specific policy creation logics.

local_policy_inference

Run a connector enabled policy using environment observation.

parse_policy_specs_from_checkpoint

Read and parse policy specifications from a checkpoint file.

Other utilities#

utils.tensor_dtype.get_np_dtype

Returns the NumPy dtype of the given tensor or array.

common.CLIArguments

Dataclass for CLI arguments and options.

common.FrameworkEnum

Supported frameworks for RLlib, used for CLI argument validation.

common.SupportedFileType

Supported file types for RLlib, used for CLI argument validation.

core.rl_module.validate_module_id

Makes sure the given policy_id is valid.

train.load_experiments_from_file

Load experiments from a file.