Inspect training results#

trainer.fit() returns a Result object.

Among other information, the Result object contains the following:

  • The checkpoints that you save with ray.train.report in your training function, and the metrics attached to each one.

  • The storage location of the training run.

  • Any data that worker 0’s training function returns.

Retrieve checkpoints#

Checkpoints contain all the information needed to restore the training state, which usually includes the trained model.

You can use checkpoints for common downstream tasks such as offline batch inference with Ray Data or online model serving with Ray Serve.

Last saved checkpoint#

Use Result.checkpoint to retrieve the last checkpoint.

print("Last checkpoint:", result.checkpoint)

with result.checkpoint.as_directory() as tmpdir:
    # Load model from directory
    ...

Best checkpoint#

Sometimes you want an earlier checkpoint. For example, if your loss increases with more training because of overfitting, you might want to retrieve the checkpoint with the lowest loss.

Use Result.get_best_checkpoint to retrieve the checkpoint that ranks best by a metric that you attached to it. This method is an alpha API.

# Get the checkpoint with the lowest loss
best_checkpoint = result.get_best_checkpoint(metric="loss", mode="min")

with best_checkpoint.as_directory() as tmpdir:
    # Load model from directory
    ...

All saved checkpoints#

Use Result.best_checkpoints to retrieve a list of all saved checkpoints and their metrics. Ray Train keeps the checkpoints that the checkpoint configuration selects. By default, it keeps all of them.

# Print all saved checkpoints and their metrics
for checkpoint, metrics in result.best_checkpoints:
    print("Loss", metrics["loss"], "checkpoint", checkpoint)

View the metrics attached to checkpoints#

The Result object contains the metrics that you attach to checkpoints when you call ray.train.report in your training function. They typically include the metric that you rank checkpoints by, such as the validation loss or accuracy. For a full history of metrics over the course of training, use an experiment tracking tool.

Note

Persisting free-floating metrics that you report through ray.train.report(metrics, checkpoint=None) is deprecated, and so is retrieving them from the Result object. Ray Train persists only metrics attached to checkpoints. For details, see Deprecated: Reporting free-floating metrics.

Metrics of the last checkpoint#

Use Result.metrics to retrieve the metrics attached to the last saved checkpoint.

result = trainer.fit()

print("Observed metrics:", result.metrics)

DataFrame of all checkpoint metrics#

Use Result.metrics_dataframe to retrieve a pandas DataFrame of the metrics attached to every saved checkpoint. Each row corresponds to a checkpoint.

df = result.metrics_dataframe
print("Minimum loss", min(df["loss"]))

Retrieve data returned from the training function#

Use Result.return_value to retrieve any data that worker 0’s training function returns.

print("Returned data", result.return_value)

Access the storage location#

Ray Train stores checkpoints and the metrics attached to them on the persistent storage that you configure with the storage_path argument of RunConfig.

Use Result.path to retrieve the storage location of the training run. It’s a subdirectory of the storage_path, named after the name argument of the RunConfig. If you don’t set name, Ray Train uses ray_train_run-<date>.

The result also contains a pyarrow.fs.FileSystem that you can use to access the storage location. The file system is useful when the path is on cloud storage.

result_path: str = result.path
result_filesystem: pyarrow.fs.FileSystem = result.filesystem

print(f"Results location (fs, path) = ({result_filesystem}, {result_path})")

Restore a result from storage#

To retrieve the checkpoints and metrics of a training run later, restore the result with Result.from_path:

from ray.train import Result

restored_result = Result.from_path(result_path)
print("Restored loss", restored_result.metrics["loss"])

Catch errors#

If training fails, trainer.fit() raises a TrainingFailedError instead of returning a Result object. When a worker fails, the error is a WorkerGroupError. Its worker_failures attribute maps each failed worker’s rank to the exception that the worker raised.

try:
    result = trainer.fit()
except ray.train.TrainingFailedError as e:
    if isinstance(e, ray.train.WorkerGroupError):
        print(e.worker_failures)

The checkpoints that you saved before the failure remain on persistent storage. Because trainer.fit() doesn’t return a Result, build the storage location from the storage_path and name arguments of your RunConfig, <storage_path>/<name>, and pass it to Result.from_path. To make the location predictable, set name explicitly.