ray.rllib.core.learner.learner.Learner#

class ray.rllib.core.learner.learner.Learner(*, config: AlgorithmConfig, module_spec: SingleAgentRLModuleSpec | MultiAgentRLModuleSpec | None = None, module: RLModule | None = None, learner_group_scaling_config=None, learner_hyperparameters=None, framework_hyperparameters=None)[source]#

Base class for Learners.

This class will be used to train RLModules. It is responsible for defining the loss function, and updating the neural network weights that it owns. It also provides a way to add/remove modules to/from RLModules in a multi-agent scenario, in the middle of training (This is useful for league based training).

TF and Torch specific implementation of this class fills in the framework-specific implementation details for distributed training, and for computing and applying gradients. User should not need to sub-class this class, but instead inherit from the TF or Torch specific sub-classes to implement their algorithm-specific update logic.

Parameters:

config – The AlgorithmConfig object from which to derive most of the settings needed to build the Learner.
module_spec – The module specification for the RLModule that is being trained. If the module is a single agent module, after building the module it will be converted to a multi-agent module with a default key. Can be none if the module is provided directly via the module argument. Refer to ray.rllib.core.rl_module.SingleAgentRLModuleSpec or ray.rllib.core.rl_module.MultiAgentRLModuleSpec for more info.
module – If learner is being used stand-alone, the RLModule can be optionally passed in directly instead of the through the module_spec.

Note: We use PPO and torch as an example here because many of the showcased components need implementations to come together. However, the same pattern is generally applicable.

import gymnasium as gym

from ray.rllib.algorithms.ppo.ppo import PPOConfig
from ray.rllib.algorithms.ppo.ppo_catalog import PPOCatalog
from ray.rllib.algorithms.ppo.torch.ppo_torch_rl_module import (
    PPOTorchRLModule
)
from ray.rllib.core.rl_module.rl_module import SingleAgentRLModuleSpec

env = gym.make("CartPole-v1")

# Create a PPO config object first.
config = (
    PPOConfig()
    .framework("torch")
    .training(model={"fcnet_hiddens": [128, 128]})
)

# Create a learner instance directly from our config. All we need as
# extra information here is the env to be able to extract space information
# (needed to construct the RLModule inside the Learner).
learner = config.build_learner(env=env)

# Take one gradient update on the module and report the results.
# results = learner.update(...)

# Add a new module, perhaps for league based training.
learner.add_module(
    module_id="new_player",
    module_spec=SingleAgentRLModuleSpec(
        module_class=PPOTorchRLModule,
        observation_space=env.observation_space,
        action_space=env.action_space,
        model_config_dict={"fcnet_hiddens": [64, 64]},
        catalog_class=PPOCatalog,
    )
)

# Take another gradient update with both previous and new modules.
# results = learner.update(...)

# Remove a module.
learner.remove_module("new_player")

# Will train previous modules only.
# results = learner.update(...)

# Get the state of the learner.
state = learner.get_state()

# Set the state of the learner.
learner.set_state(state)

# Get the weights of the underly multi-agent RLModule.
weights = learner.get_module_state()

# Set the weights of the underly multi-agent RLModule.
learner.set_module_state(weights)

Extension pattern:

from ray.rllib.core.learner.torch.torch_learner import TorchLearner

class MyLearner(TorchLearner):

   def compute_loss(self, fwd_out, batch):
       # compute the loss based on batch and output of the forward pass
       # to access the learner hyper-parameters use `self._hps`
       return {ALL_MODULES: loss}

PublicAPI (alpha): This API is in alpha and may change before becoming stable.

Methods

`add_module`	Add a module to the underlying MultiAgentRLModule and the Learner.
`additional_update`	Apply additional non-gradient based updates to this Algorithm.
`additional_update_for_module`	Apply additional non-gradient based updates for a single module.
`apply_gradients`	Applies the gradients to the MultiAgentRLModule parameters.
`build`	Builds the Learner.
`compute_gradients`	Computes the gradients based on the given losses.
`compute_loss`	Computes the loss for the module being optimized.
`compute_loss_for_module`	Computes the loss for a single module.
`configure_optimizers`	Configures, creates, and registers the optimizers for this Learner.
`configure_optimizers_for_module`	Configures an optimizer for the given module_id.
`filter_param_dict_for_optimizer`	Reduces the given ParamDict to contain only parameters for given optimizer.
`get_module_state`	Returns the state of the underlying MultiAgentRLModule.
`get_optimizer`	Returns the optimizer object, configured under the given module_id and name.
`get_optimizer_state`	Returns the state of all optimizers currently registered in this Learner.
`get_optimizers_for_module`	Returns a list of (optimizer_name, optimizer instance)-tuples for module_id.
`get_param_ref`	Returns a hashable reference to a trainable parameter.
`get_parameters`	Returns the list of parameters of a module.
`get_state`	Get the state of the learner.
`load_state`	Load the state of the learner from path
`postprocess_gradients`	Applies potential postprocessing operations on the gradients.
`postprocess_gradients_for_module`	Applies postprocessing operations on the gradients of the given module.
`register_optimizer`	Registers an optimizer with a ModuleID, name, param list and lr-scheduler.
`remove_module`	Remove a module from the Learner.
`save_state`	Save the state of the learner to path
`set_module_state`	Sets the state of the underlying MultiAgentRLModule
`set_optimizer_state`	Sets the state of all optimizers currently registered in this Learner.
`set_state`	Set the state of the learner.
`should_module_be_updated`	Returns whether a module should be updated or not based on `self.config`.
`update_from_batch`	Do `num_iters` minibatch updates given a train batch.
`update_from_episodes`	Do `num_iters` minibatch updates given a list of episodes.

Attributes

`TOTAL_LOSS_KEY`
`distributed`	Whether the learner is running in distributed mode.
`framework`
`module`	The multi-agent RLModule that is being trained.