OfflinePreLearner#

class ray.rllib.offline.offline_prelearner.OfflinePreLearner(*, config: AlgorithmConfig, spaces: Tuple[gymnasium.Space, gymnasium.Space] | None = None, module_state: Dict[str, Any] | None = None, **kwargs: Dict[str, Any])[source]#

Bases: object

Maps raw ingested data to episodes and runs the Learner pipeline.

OfflinePreLearner is meant to be used by RLlib to build a Ray Data pipeline using ray.data.Dataset.map_batches when ingesting data for offline training. Ray data is thereby used under the hood to parallelize the data transformation.

It’s basic function is to: (1) Convert the dataset into RLlib’s native episode format (SingleAgentEpisode, since MultiAgentEpisode is not supported yet). (2) Apply the learner connector pipeline to episodes to create batches that are ready to be trained on (can be passed in Learner.update methods).

OfflinePreLearner can be overridden to implement custom logic and passed into AlgorithmConfig.offline(prelearner_class).

Parameters:
  • config (AlgorithmConfig) – The AlgorithmConfig to build the RLModule and the Learner connector pipeline from.

  • spaces (Tuple[gymnasium.Space, gymnasium.Space] | None) – The observation and action space to build the Learner connector pipeline with.

  • module_state (Dict[str, Any] | None) – The RLModule state to set on the newly built RLModule.

  • **kwargs (Dict[str, Any]) – Deprecated. A module_spec keyword argument is still accepted, with a deprecation warning, in place of config.rl_module_spec.

PublicAPI (alpha): This API is in alpha and may change before becoming stable.

Methods