Console Output (Reporters)

By default, Tune reports experiment progress periodically to the command-line as follows.

== Status ==
Memory usage on this node: 11.4/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 4/12 CPUs, 0/0 GPUs, 0.0/3.17 GiB heap, 0.0/1.07 GiB objects
Result logdir: /Users/foo/ray_results/myexp
Number of trials: 4 (4 RUNNING)
+----------------------+----------+---------------------+-----------+--------+--------+--------+--------+------------------+-------+
| Trial name           | status   | loc                 |    param1 | param2 | param3 |    acc |   loss |   total time (s) |  iter |
|----------------------+----------+---------------------+-----------+--------+--------+--------+--------+------------------+-------|
| MyTrainable_a826033a | RUNNING  | 10.234.98.164:31115 | 0.303706  | 0.0761 | 0.4328 | 0.1289 | 1.8572 |          7.54952 |    15 |
| MyTrainable_a8263fc6 | RUNNING  | 10.234.98.164:31117 | 0.929276  | 0.158  | 0.3417 | 0.4865 | 1.6307 |          7.0501  |    14 |
| MyTrainable_a8267914 | RUNNING  | 10.234.98.164:31111 | 0.068426  | 0.0319 | 0.1147 | 0.9585 | 1.9603 |          7.0477  |    14 |
| MyTrainable_a826b7bc | RUNNING  | 10.234.98.164:31112 | 0.729127  | 0.0748 | 0.1784 | 0.1797 | 1.7161 |          7.05715 |    14 |
+----------------------+----------+---------------------+-----------+--------+--------+--------+--------+------------------+-------+

Note that columns will be hidden if they are completely empty. The output can be configured in various ways by instantiating a CLIReporter instance (or JupyterNotebookReporter if you’re using jupyter notebook). Here’s an example:

from ray.tune import CLIReporter

# Limit the number of rows.
reporter = CLIReporter(max_progress_rows=10)
# Add a custom metric column, in addition to the default metrics.
# Note that this must be a metric that is returned in your training results.
reporter.add_metric_column("custom_metric")
tune.run(my_trainable, progress_reporter=reporter)

Extending CLIReporter lets you control reporting frequency. For example:

class ExperimentTerminationReporter(CLIReporter):
    def should_report(self, trials, done=False):
        """Reports only on experiment termination."""
        return done

tune.run(my_trainable, progress_reporter=ExperimentTerminationReporter())

class TrialTerminationReporter(CLIReporter):
    def __init__(self):
        self.num_terminated = 0

    def should_report(self, trials, done=False):
        """Reports only on trial termination events."""
        old_num_terminated = self.num_terminated
        self.num_terminated = len([t for t in trials if t.status == Trial.TERMINATED])
        return self.num_terminated > old_num_terminated

tune.run(my_trainable, progress_reporter=TrialTerminationReporter())

The default reporting style can also be overridden more broadly by extending the ProgressReporter interface directly. Note that you can print to any output stream, file etc.

from ray.tune import ProgressReporter

class CustomReporter(ProgressReporter):

    def should_report(self, trials, done=False):
        return True

    def report(self, trials, *sys_info):
        print(*sys_info)
        print("\n".join([str(trial) for trial in trials]))

tune.run(my_trainable, progress_reporter=CustomReporter())

CLIReporter

class ray.tune.CLIReporter(metric_columns=None, parameter_columns=None, total_samples=None, max_progress_rows=20, max_error_rows=20, max_report_frequency=5, infer_limit=3, metric=None, mode=None)[source]

Command-line reporter

Parameters
  • metric_columns (dict[str, str]|list[str]) – Names of metrics to include in progress table. If this is a dict, the keys should be metric names and the values should be the displayed names. If this is a list, the metric name is used directly.

  • parameter_columns (dict[str, str]|list[str]) – Names of parameters to include in progress table. If this is a dict, the keys should be parameter names and the values should be the displayed names. If this is a list, the parameter name is used directly. If empty, defaults to all available parameters.

  • max_progress_rows (int) – Maximum number of rows to print in the progress table. The progress table describes the progress of each trial. Defaults to 20.

  • max_error_rows (int) – Maximum number of rows to print in the error table. The error table lists the error file, if any, corresponding to each trial. Defaults to 20.

  • max_report_frequency (int) – Maximum report frequency in seconds. Defaults to 5s.

add_metric_column(metric, representation=None)

Adds a metric to the existing columns.

Parameters
  • metric (str) – Metric to add. This must be a metric being returned in training step results.

  • representation (str) – Representation to use in table. Defaults to metric.

JupyterNotebookReporter

class ray.tune.JupyterNotebookReporter(overwrite, metric_columns=None, parameter_columns=None, total_samples=None, max_progress_rows=20, max_error_rows=20, max_report_frequency=5, infer_limit=3, metric=None, mode=None)[source]

Jupyter notebook-friendly Reporter that can update display in-place.

Parameters
  • overwrite (bool) – Flag for overwriting the last reported progress.

  • metric_columns (dict[str, str]|list[str]) – Names of metrics to include in progress table. If this is a dict, the keys should be metric names and the values should be the displayed names. If this is a list, the metric name is used directly.

  • parameter_columns (dict[str, str]|list[str]) – Names of parameters to include in progress table. If this is a dict, the keys should be parameter names and the values should be the displayed names. If this is a list, the parameter name is used directly. If empty, defaults to all available parameters.

  • max_progress_rows (int) – Maximum number of rows to print in the progress table. The progress table describes the progress of each trial. Defaults to 20.

  • max_error_rows (int) – Maximum number of rows to print in the error table. The error table lists the error file, if any, corresponding to each trial. Defaults to 20.

  • max_report_frequency (int) – Maximum report frequency in seconds. Defaults to 5s.

add_metric_column(metric, representation=None)

Adds a metric to the existing columns.

Parameters
  • metric (str) – Metric to add. This must be a metric being returned in training step results.

  • representation (str) – Representation to use in table. Defaults to metric.

ProgressReporter

class ray.tune.ProgressReporter[source]

Abstract class for experiment progress reporting.

should_report() is called to determine whether or not report() should be called. Tune will call these functions after trial state transitions, receiving training results, and so on.

should_report(trials, done=False)[source]

Returns whether or not progress should be reported.

Parameters
  • trials (list[Trial]) – Trials to report on.

  • done (bool) – Whether this is the last progress report attempt.

report(trials, done, *sys_info)[source]

Reports progress across trials.

Parameters
  • trials (list[Trial]) – Trials to report on.

  • done (bool) – Whether this is the last progress report attempt.

  • sys_info – System info.