Offline RL

InputReaders

The InputReader API is used by an individual RolloutWorker to produce batches of experiences either from an simulator/environment or from an offline source (e.g. a file).

Here, we introduce the generic API and its child classes used for reading offline data (for offline RL). For details on RLlib’s Sampler implementations for collecting data from simulators/environments, see the Sampler docs here.

JsonReader (ray.rllib.offline.json_reader.JsonReader)

For reading data from offline files (for example when no simulator/environment is available), you can use the built-in JsonReader class.

You will have to change the input config value from “sampler” (default) to a JSON file name (str), a list of JSON files, or a path name (str) that contains JSON files. Alternatively, you can specify a callable that takes a IOContext object as only arg and returns a new InputReader instance, for example:

config = {
    "input": lambda io_ctx: MyReader([arg1], [arg2], [io_ctx]),
}

For details on the IOContext class, see below.

MixedInput (ray.rllib.offline.mixed_input.MixedInput)

In order to mix different input readers with each other in different custom ratios, you can use the MixedInput reader. This reader is chosen automatically by RLlib when you provide a dict under the input config key that maps input reader specifiers to probabilities, e.g.:

"input": {
   "sampler": 0.4,  # 40% of samples will come from environment
   "/tmp/experiences/*.json": 0.4,  # the rest from different JSON files
   "s3://bucket/expert.json": 0.2,
}

D4RLReader (ray.rllib.offline.d4rl_reader.D4RLReader)

IOContext

IOContext instances are used in every InputReader and OutputWriter class. They serve as simple containers for the properties: log_dir, config, and worker_index.