Note

Ray 2.10.0 introduces the alpha stage of RLlib’s “new API stack”. The Ray Team plans to transition algorithms, example scripts, and documentation to the new code base thereby incrementally replacing the “old API stack” (e.g., ModelV2, Policy, RolloutWorker) throughout the subsequent minor releases leading up to Ray 3.0.

See here for more details on how to use the new API stack.

Replay Buffer API#

The following classes don’t take into account the separation of experiences from different policies, multi-agent replay buffers will be explained further below.

Replay Buffer Base Classes#

StorageUnit

Specifies how batches are structured in a ReplayBuffer.

ReplayBuffer

The lowest-level replay buffer interface used by RLlib.

PrioritizedReplayBuffer

This buffer implements Prioritized Experience Replay.

ReservoirReplayBuffer

This buffer implements reservoir sampling.

Public Methods#

sample

Samples num_items items from this buffer.

add

Adds a batch of experiences or other data to this buffer.

get_state

Returns all local state in a dict.

set_state

Restores all local state to the provided state.

Multi Agent Buffers#

The following classes use the above, “single-agent”, buffers as underlying buffers to facilitate splitting up experiences between the different agents’ policies. In multi-agent RL, more than one agent exists in the environment and not all of these agents may utilize the same policy (mapping M agents to N policies, where M <= N). This leads to the need for MultiAgentReplayBuffers that store the experiences of different policies separately.

MultiAgentReplayBuffer

A replay buffer shard for multiagent setups.

MultiAgentPrioritizedReplayBuffer

A prioritized replay buffer shard for multiagent setups.

Utility Methods#

update_priorities_in_replay_buffer

Updates the priorities in a prioritized replay buffer, given training results.

sample_min_n_steps_from_buffer

Samples a minimum of n timesteps from a given replay buffer.