ray.rllib.utils.metrics.metrics_logger.MetricsLogger.reduce#

MetricsLogger.reduce(key: str | Tuple[str, ...] | None = None, *, return_stats_obj: bool = True) → Dict[source]#

Reduces all logged values based on their settings and returns a result dict.

DO NOT CALL THIS METHOD under normal circumstances! RLlib’s components call it right before a distinct step has been completed and the (MetricsLogger-based) results of that step need to be passed upstream to other components for further processing.

The returned result dict has the exact same structure as the logged keys (or nested key sequences) combined. At the leafs of the returned structure are either Stats objects (return_stats_obj=True, which is the default) or primitive (non-Stats) values (return_stats_obj=False). In case of return_stats_obj=True, the returned dict with Stats at the leafs can conveniently be re-used upstream for further logging and reduction operations.

For example, imagine component A (e.g. an Algorithm) containing a MetricsLogger and n remote components (e.g. n EnvRunners), each with their own MetricsLogger object. Component A calls its n remote components, each of which returns an equivalent, reduced dict with Stats as leafs. Component A can then further log these n result dicts through its own MetricsLogger through: logger.merge_and_log_n_dicts([n returned result dicts from n subcomponents]).

The returned result dict has the exact same structure as the logged keys (or nested key sequences) combined. At the leafs of the returned structure are either Stats objects (return_stats_obj=True, which is the default) or primitive (non-Stats) values (return_stats_obj=False). In case of return_stats_obj=True, the returned dict with Stats at the leafs can be reused conveniently downstream for further logging and reduction operations.

For example, imagine component A (e.g. an Algorithm) containing a MetricsLogger and n remote components (e.g. n EnvRunner workers), each with their own MetricsLogger object. Component A calls its n remote components, each of which returns an equivalent, reduced dict with Stats instances as leafs. Component A can now further log these n result dicts through its own MetricsLogger: logger.merge_and_log_n_dicts([n returned result dicts from the remote components]).

from ray.rllib.utils.metrics.metrics_logger import MetricsLogger
from ray.rllib.utils.test_utils import check

# Log some (EMA reduced) values.
logger = MetricsLogger()
logger.log_value("a", 2.0)
logger.log_value("a", 3.0)
expected_reduced = (1.0 - 0.01) * 2.0 + 0.01 * 3.0
# Reduce and return primitive values (not Stats objects).
results = logger.reduce(return_stats_obj=False)
check(results, {"a": expected_reduced})

# Log some values to be averaged with a sliding window.
logger = MetricsLogger()
logger.log_value("a", 2.0, window=2)
logger.log_value("a", 3.0)
logger.log_value("a", 4.0)
expected_reduced = (3.0 + 4.0) / 2  # <- win size is only 2; first logged
                                    # item not used
# Reduce and return primitive values (not Stats objects).
results = logger.reduce(return_stats_obj=False)
check(results, {"a": expected_reduced})

# Assume we have 2 remote components, each one returning an equivalent
# reduced dict when called. We can simply use these results and log them
# to our own MetricsLogger, then reduce over these 2 logged results.
comp1_logger = MetricsLogger()
comp1_logger.log_value("a", 1.0, window=10)
comp1_logger.log_value("a", 2.0)
result1 = comp1_logger.reduce()  # <- return Stats objects as leafs

comp2_logger = MetricsLogger()
comp2_logger.log_value("a", 3.0, window=10)
comp2_logger.log_value("a", 4.0)
result2 = comp2_logger.reduce()  # <- return Stats objects as leafs

# Now combine the 2 equivalent results into 1 end result dict.
downstream_logger = MetricsLogger()
downstream_logger.merge_and_log_n_dicts([result1, result2])
# What happens internally is that both values lists of the 2 components
# are merged (concat'd) and randomly shuffled, then clipped at 10 (window
# size). This is done such that no component has an "advantage" over the
# other as we don't know the exact time-order in which these parallelly
# running components logged their own "a"-values.
# We execute similarly useful merging strategies for other reduce settings,
# such as EMA, max/min/sum-reducing, etc..
end_result = downstream_logger.reduce(return_stats_obj=False)
check(end_result, {"a": 2.5})
Parameters:

  • key – Optional key or key sequence (for nested location within self.stats), limiting the reduce operation to that particular sub-structure of self. If None, will reduce all of self’s Stats.

  • return_stats_obj – Whether in the returned dict, the leafs should be Stats objects. This is the default as it enables users to continue using (and further logging) the results of this call inside another (downstream) MetricsLogger object.

Returns:

A (nested) dict matching the structure of self.stats (contains all ever logged keys to this MetricsLogger) with the leafs being (reduced) Stats objects if return_stats_obj=True or primitive values, carrying no reduction and history information, if return_stats_obj=False.