Offline RL
Contents
Offline RL#
InputReaders#
The InputReader API is used by an individual RolloutWorker
to produce batches of experiences either from an simulator/environment or from an
offline source (e.g. a file).
Here, we introduce the generic API and its child classes used for reading offline data (for offline RL).
For details on RLlib’s Sampler
implementations
for collecting data from simulators/environments, see the Sampler docs here.
JsonReader (ray.rllib.offline.json_reader.JsonReader)#
For reading data from offline files (for example when no simulator/environment is available),
you can use the built-in JsonReader
class.
You will have to change the input
config value from “sampler” (default) to a
JSON file name (str), a list of JSON files, or a path name (str) that contains JSON files.
Alternatively, you can specify a callable that takes a IOContext
object as only
arg and returns a new InputReader
instance, for example:
config = {
"input": lambda io_ctx: MyReader([arg1], [arg2], [io_ctx]),
}
For details on the IOContext
class, see below.
MixedInput (ray.rllib.offline.mixed_input.MixedInput)#
In order to mix different input readers with each other in different custom ratios, you can use
the MixedInput
reader. This reader is chosen
automatically by RLlib when you provide a dict under
the input
config key that maps input reader specifiers to probabilities, e.g.:
"input": {
"sampler": 0.4, # 40% of samples will come from environment
"/tmp/experiences/*.json": 0.4, # the rest from different JSON files
"s3://bucket/expert.json": 0.2,
}
D4RLReader (ray.rllib.offline.d4rl_reader.D4RLReader)#
IOContext#
IOContext instances are used in every InputReader and OutputWriter class. They serve as
simple containers for the properties: log_dir
, config
, and worker_index
.