Monitor and log metrics#

ray.train.report(metrics, checkpoint) persists a checkpoint so that Ray Train can restore training after a failure. Ray Train attaches the metrics that you pass to ray.train.report to that checkpoint. Ray Train doesn’t persist metrics reported without a checkpoint.

Ray Train uses the metrics attached to checkpoints for the following:

To track metrics such as loss and accuracy over the course of training, log them from the workers to an experiment tracking tool such as MLflow or W&B. See Experiment tracking.

Ray Train attaches only the metrics that the rank 0 worker reports to the checkpoint. However, ray.train.report acts as a barrier to ensure consistency, so you must call it on each worker. To aggregate results from multiple workers, see Obtain and aggregate results from different workers.

Obtain and aggregate results from different workers#

In real applications, you might need optimization metrics beyond accuracy and loss, such as recall, precision, and F-beta score. You might also need metrics from multiple workers. Ray Train currently reports metrics only from the rank 0 worker. To report metrics from multiple workers, use third-party libraries or the distributed primitives of your machine learning framework.

Ray Train natively supports TorchMetrics, which provides machine learning metrics for distributed, scalable PyTorch models.

The following example reports both the aggregated R2 score and the mean training and validation loss from all workers:


# First, pip install torchmetrics
# This code is tested with torchmetrics==0.7.3 and torch==1.12.1

import os
import tempfile

import ray.train.torch
from ray import train
from ray.train import ScalingConfig
from ray.train.torch import TorchTrainer

import torch
import torch.nn as nn
import torchmetrics
from torch.optim import Adam
import numpy as np


def train_func(config):
    n = 100
    # create a toy dataset
    X = torch.Tensor(np.random.normal(0, 1, size=(n, 4)))
    X_valid = torch.Tensor(np.random.normal(0, 1, size=(n, 4)))
    Y = torch.Tensor(np.random.uniform(0, 1, size=(n, 1)))
    Y_valid = torch.Tensor(np.random.uniform(0, 1, size=(n, 1)))
    # toy neural network : 1-layer
    # wrap the model in DDP
    model = ray.train.torch.prepare_model(nn.Linear(4, 1))
    criterion = nn.MSELoss()

    mape = torchmetrics.MeanAbsolutePercentageError()
    # for averaging loss
    mean_valid_loss = torchmetrics.MeanMetric()

    optimizer = Adam(model.parameters(), lr=3e-4)
    for epoch in range(config["num_epochs"]):
        model.train()
        y = model.forward(X)

        # compute loss
        loss = criterion(y, Y)

        # back-propagate loss
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # evaluate
        model.eval()
        with torch.no_grad():
            pred = model(X_valid)
            valid_loss = criterion(pred, Y_valid)
            # save loss in aggregator
            mean_valid_loss(valid_loss)
            mape(pred, Y_valid)

        # collect all metrics
        # use .item() to obtain a value that can be reported
        valid_loss = valid_loss.item()
        mape_collected = mape.compute().item()
        mean_valid_loss_collected = mean_valid_loss.compute().item()

        with tempfile.TemporaryDirectory() as temp_checkpoint_dir:
            torch.save(
                model.state_dict(), os.path.join(temp_checkpoint_dir, "model.pt")
            )

            train.report(
                {
                    "mape_collected": mape_collected,
                    "valid_loss": valid_loss,
                    "mean_valid_loss_collected": mean_valid_loss_collected,
                },
                checkpoint=train.Checkpoint.from_directory(temp_checkpoint_dir),
            )

        # reset for next epoch
        mape.reset()
        mean_valid_loss.reset()


trainer = TorchTrainer(
    train_func,
    train_loop_config={"num_epochs": 5},
    scaling_config=ScalingConfig(num_workers=2),
)
result = trainer.fit()
print(result.metrics["valid_loss"], result.metrics["mean_valid_loss_collected"])
# 0.5109779238700867 0.5512474775314331

Deprecated: Reporting free-floating metrics#

Reporting metrics with ray.train.report(metrics, checkpoint=None) from every worker writes the metrics to the Ray Tune log files progress.csv and result.json. Access them through Result.metrics_dataframe on the Result that trainer.fit() returns.

As of Ray 2.43, this behavior is deprecated. Ray Train V2, an overhaul of Ray Train’s implementation and select APIs, doesn’t support it.

Ray Train V2 keeps only the slim set of experiment tracking features that fault tolerance needs, so it doesn’t support reporting free-floating metrics that aren’t attached to checkpoints. To track metrics, report them directly from the workers to experiment tracking tools such as MLflow and W&B. See Experiment tracking for examples.

In Ray Train V2, reporting only metrics from all workers is a no-op. However, you can still access the results that all workers report and use them to implement custom metric-handling logic, as the following example shows:

import os

assert os.environ["RAY_TRAIN_V2_ENABLED"] == "1"

from typing import Any, Dict, List, Optional

import ray.train
import ray.train.torch


def train_fn_per_worker(config):
    # Free-floating metrics can be accessed from the callback below.
    ray.train.report({"rank": ray.train.get_context().get_world_rank()})


class CustomMetricsCallback(ray.train.UserCallback):
    def after_report(
        self,
        run_context,
        metrics: List[Dict[str, Any]],
        checkpoint: Optional[ray.train.Checkpoint],
    ):
        rank_0_metrics = metrics[0]
        print(rank_0_metrics)
        # Ex: Write metrics to a file...


trainer = ray.train.torch.TorchTrainer(
    train_fn_per_worker,
    scaling_config=ray.train.ScalingConfig(num_workers=2),
    run_config=ray.train.RunConfig(callbacks=[CustomMetricsCallback()]),
)
trainer.fit()

To use Ray Tune Callbacks that depend on free-floating metrics that workers report, run Ray Train as a single Ray Tune trial.

For details, see the following resources: