Logging and outputs in Tune

By default, Ray Tune logs results in TensorBoard, CSV, and JSON formats. To log lower-level data such as model weights or gradients, see trainable logging. For more on logging and how to customize it, see Tune Loggers (tune.logger).

How to configure logging in Tune?

Tune logs the results of each trial to a subdirectory of a specified local directory, which defaults to ~/ray_results.

# This logs to two different trial folders:
# ~/ray_results/trainable_name/trial_name_1 and ~/ray_results/trainable_name/trial_name_2
# trainable_name and trial_name are autogenerated.
tuner = tune.Tuner(trainable, tune_config=tune.TuneConfig(num_samples=2))
results = tuner.fit()

You can specify the storage_path and the experiment name:

# This logs to 2 different trial folders:
# ./results/test_experiment/trial_name_1 and ./results/test_experiment/trial_name_2
# Only trial_name is autogenerated.
tuner = tune.Tuner(trainable,
    tune_config=tune.TuneConfig(num_samples=2),
    run_config=RunConfig(storage_path="./results", name="test_experiment"))
results = tuner.fit()

For the trial API reference, see Trial.

How to log your Tune runs to TensorBoard?

Tune automatically outputs TensorBoard files during Tuner.fit(). To visualize learning in TensorBoard, install tensorboardX:

pip install tensorboardX

After you run an experiment, visualize it in TensorBoard by specifying the output directory of your results:

tensorboard --logdir=~/ray_results/my_experiment

If you’re running Ray on a remote multi-user cluster where you don’t have sudo access, you can run the following commands so that TensorBoard can write to the temporary directory:

export TMPDIR=/tmp/$USER; mkdir -p $TMPDIR; tensorboard --logdir=~/ray_results

If you use TensorFlow 2.x, Tune also generates TensorBoard HParams output automatically, as the following example shows:

tuner = tune.Tuner(
    ...,
    param_space={
        "lr": tune.grid_search([1e-5, 1e-4]),
        "momentum": tune.grid_search([0, 0.9])
    }
)
results = tuner.fit()

How to control console output with Tune?

Tune outputs the fields you provide automatically, on a best-effort basis. To customize the console output, you can use a Reporter object.

== Status ==
Memory usage on this node: 11.4/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 4/12 CPUs, 0/0 GPUs, 0.0/3.17 GiB heap, 0.0/1.07 GiB objects
Result logdir: /Users/foo/ray_results/myexp
Number of trials: 4 (4 RUNNING)
+----------------------+----------+---------------------+-----------+--------+--------+----------------+-------+
| Trial name           | status   | loc                 |    param1 | param2 |    acc | total time (s) |  iter |
|----------------------+----------+---------------------+-----------+--------+--------+----------------+-------|
| MyTrainable_a826033a | RUNNING  | 10.234.98.164:31115 | 0.303706  | 0.0761 | 0.1289 |        7.54952 |    15 |
| MyTrainable_a8263fc6 | RUNNING  | 10.234.98.164:31117 | 0.929276  | 0.158  | 0.4865 |        7.0501  |    14 |
| MyTrainable_a8267914 | RUNNING  | 10.234.98.164:31111 | 0.068426  | 0.0319 | 0.9585 |        7.0477  |    14 |
| MyTrainable_a826b7bc | RUNNING  | 10.234.98.164:31112 | 0.729127  | 0.0748 | 0.1797 |        7.05715 |    14 |
+----------------------+----------+---------------------+-----------+--------+--------+----------------+-------+

How to redirect trainable logs to files in a Tune run?

Tune runs trainables as remote actors. By default, Ray collects the stdout and stderr of actors and prints them to the head process. For more information, see worker logs. Logging inside trainables follows this default handling. To collect trainable logs in files for analysis instead, use the log_to_file option. It applies to print statements, warnings.warn, logger.info, and similar output.

To log stdout and stderr to trial_logdir/stdout and trial_logdir/stderr, respectively, pass log_to_file=True to the RunConfig that you pass to Tuner:

tuner = tune.Tuner(
    trainable,
    run_config=RunConfig(log_to_file=True)
)
results = tuner.fit()

To specify the output files, you can pass either one filename for the combined output, or two filenames for stdout and stderr, respectively:

tuner = tune.Tuner(
    trainable,
    run_config=RunConfig(log_to_file="std_combined.log")
)
tuner.fit()

tuner = tune.Tuner(
    trainable,
    run_config=RunConfig(log_to_file=("my_stdout.log", "my_stderr.log")))
results = tuner.fit()

The filenames are relative to the trial directory. You can also pass absolute paths.

Note

If you use Ray Tune with Ray Train, the log_to_file configuration doesn’t cover logging in the distributed training workers.

Where to find log_to_file files?

If you configure your Tune workload to sync to the head node, the log_to_file outputs are in each trial directory. If you configure it to sync to the cloud instead, the log_to_file outputs aren’t synced to the cloud, and you can find them only on the worker nodes where the corresponding trial ran.

Note

This behavior can cause problems when a trainable moves between nodes during its lifetime, which can happen with some schedulers or with node failures. The Ray project may prioritize enabling this feature if enough users request it. If this affects your workflow, consider commenting on GitHub issue #32142.

Leave feedback on this feature

To share how you prefer to interact with logging in trainables, leave a comment on GitHub issue #32142.

How do you log arbitrary files from a Tune trainable?

By default, Tune logs only the training result dictionaries and checkpoints from your trainable. You may also want to save a file that visualizes the model weights or model graph, or use a custom logging library that requires multi-process logging, for example to log images to TensorBoard. These saved files are trial artifacts.

Note

If you set SyncConfig(sync_artifacts=True), trial artifacts are uploaded periodically from each trial to the RunConfig(storage_path). With Ray Train, they’re uploaded from each remote training worker.

See the SyncConfig API reference for artifact syncing configuration options.

Tip

Keep any logging calls or objects within the scope of the trainable. Otherwise, you may see pickling or other serialization errors, or inconsistent logs.

You can save trial artifacts directly in the trainable, as the following examples show:

Function API

import logging_library  # ex: mlflow, wandb
from ray import tune

def trainable(config):
    logging_library.init(
        name=trial_id,
        id=trial_id,
        resume=trial_id,
        reinit=True,
        allow_val_change=True)
    logging_library.set_log_path(os.getcwd())

    for step in range(100):
        logging_library.log_model(...)
        logging_library.log(results, step=step)

        # You can also just write to a file directly.
        # The working directory is set to the trial directory, so
        # you don't need to worry about multiple workers saving
        # to the same location.
        with open(f"./artifact_{step}.txt", "w") as f:
            f.write("Artifact Data")

        tune.report(results)

Class API

import logging_library  # ex: mlflow, wandb
from ray import tune

class CustomLogging(tune.Trainable)
    def setup(self, config):
        trial_id = self.trial_id
        logging_library.init(
            name=trial_id,
            id=trial_id,
            resume=trial_id,
            reinit=True,
            allow_val_change=True
        )
        logging_library.set_log_path(os.getcwd())

    def step(self):
        logging_library.log_model(...)

        # You can also write to a file directly.
        # The working directory is set to the trial directory, so
        # you don't need to worry about multiple workers saving
        # to the same location.
        with open(f"./artifact_{self.iteration}.txt", "w") as f:
            f.write("Artifact Data")

    def log_result(self, result):
        res_dict = {
            str(k): v
            for k, v in result.items()
            if (v and "config" not in k and not isinstance(v, str))
        }
        step = result["training_iteration"]
        logging_library.log(res_dict, step=step)

In the preceding examples, logging_library stands for whatever third-party logging library you use. logging_library.set_log_path(os.getcwd()) is an imaginary API for demonstration purposes. It shows that you should configure the third-party library to log to the trainable’s working directory. By default, the working directory of a function trainable or class trainable is the corresponding trial directory once the trainable launches as a remote Ray actor.

How to build custom Tune loggers?

To create a custom logger, subclass the LoggerCallback interface, described in LoggerCallback Interface (tune.logger.LoggerCallback):

from typing import Dict, List

import json
import os

from ray.tune.logger import LoggerCallback


class CustomLoggerCallback(LoggerCallback):
    """Custom logger interface"""

    def __init__(self, filename: str = "log.txt"):
        self._trial_files = {}
        self._filename = filename

    def log_trial_start(self, trial: "Trial"):
        trial_logfile = os.path.join(trial.logdir, self._filename)
        self._trial_files[trial] = open(trial_logfile, "at")

    def log_trial_result(self, iteration: int, trial: "Trial", result: Dict):
        if trial in self._trial_files:
            self._trial_files[trial].write(json.dumps(result))

    def on_trial_complete(self, iteration: int, trials: List["Trial"],
                          trial: "Trial", **info):
        if trial in self._trial_files:
            self._trial_files[trial].close()
            del self._trial_files[trial]

Then pass your logger to RunConfig as a callback:

from ray import tune

tuner = tune.Tuner(
    MyTrainableClass,
    run_config=tune.RunConfig(
        name="experiment_name", callbacks=[CustomLoggerCallback("log_test.txt")]
    )
)
results = tuner.fit()

By default, Ray Tune creates JSON, CSV, and TensorBoardX logger callbacks if you don’t pass them yourself. To disable this behavior, set the TUNE_DISABLE_AUTO_CALLBACK_LOGGERS environment variable to "1".

For an example of a custom logger, see Logging Example.