ray.rllib.utils.replay_buffers.replay_buffer.ReplayBuffer

class ray.rllib.utils.replay_buffers.replay_buffer.ReplayBuffer(capacity: int = 10000, storage_unit: str | StorageUnit = 'timesteps', **kwargs)

Bases: ReplayBufferInterface, FaultAwareApply

The lowest-level replay buffer interface used by RLlib.

This class implements a basic ring-type of buffer with random sampling. ReplayBuffer is the base class for advanced types that add functionality while retaining compatibility through inheritance.

The following examples show how buffers behave with different storage_units and capacities. This behaviour is generally similar for other buffers, although they might not implement all storage_units.

Examples:

from ray.rllib.utils.replay_buffers.replay_buffer import ReplayBuffer
from ray.rllib.utils.replay_buffers.replay_buffer import StorageUnit
from ray.rllib.policy.sample_batch import SampleBatch

# Store any batch as a whole
buffer = ReplayBuffer(capacity=10, storage_unit=StorageUnit.FRAGMENTS)
buffer.add(SampleBatch({"a": [1], "b": [2, 3, 4]}))
buffer.sample(1)

# Store only complete episodes
buffer = ReplayBuffer(capacity=10,
                        storage_unit=StorageUnit.EPISODES)
buffer.add(SampleBatch({"c": [1, 2, 3, 4],
                        SampleBatch.T: [0, 1, 0, 1],
                        SampleBatch.TERMINATEDS: [False, True, False, True],
                        SampleBatch.EPS_ID: [0, 0, 1, 1]}))
buffer.sample(1)

# Store single timesteps
buffer = ReplayBuffer(capacity=2, storage_unit=StorageUnit.TIMESTEPS)
buffer.add(SampleBatch({"a": [1, 2], SampleBatch.T: [0, 1]}))
buffer.sample(1)

buffer.add(SampleBatch({"a": [3], SampleBatch.T: [2]}))
print(buffer._eviction_started)
buffer.sample(1)

buffer = ReplayBuffer(capacity=10, storage_unit=StorageUnit.SEQUENCES)
buffer.add(SampleBatch({"c": [1, 2, 3], SampleBatch.SEQ_LENS: [1, 2]}))
buffer.sample(1)

True

True is not the output of the above testcode, but an artifact of unexpected behaviour of sphinx doctests. (see ray-project/ray#32477)

DeveloperAPI: This API may change across minor Ray releases.

Methods

__init__	Initializes a (FIFO) ReplayBuffer instance.
add	Adds a batch of experiences or other data to this buffer.
apply	Calls the given function with this Actor instance.
get_host	Returns the computer's network name.
ping	Ping the actor.
sample	Samples num_items items from this buffer.
stats	Returns the stats of this buffer.