How to define stopping criteria for a Ray Tune experiment#

When you run a Ray Tune experiment, it can be hard to know in advance how long training should last. Use stopping criteria to end training when specific conditions hold.

For example, you might want to stop in any of the following cases:

  • End each trial after N epochs or when the reported evaluation score passes a threshold, whichever comes first.

  • Stop the experiment after T seconds.

  • Stop the experiment when trials encounter runtime errors.

  • Stop underperforming trials early with Tune’s early-stopping schedulers.

This guide shows how to set up each of these stopping criteria in a Tune experiment.

All the code examples use the following training function:

from ray import tune
import time

def my_trainable(config):
    i = 1
    while True:
        # Do some training...
        time.sleep(1)

        # Report some metrics for demonstration...
        tune.report({"mean_accuracy": min(i / 10, 1.0)})
        i += 1

Stop a Tune experiment manually#

To stop an experiment manually, send a SIGINT signal to the process running Tuner.fit(). Pressing Ctrl+C in the terminal usually does this. Ray Tune then shuts down training gracefully and saves the final experiment state.

Note

If you force a Tune experiment to stop, for example by pressing Ctrl+C several times, Tune can’t snapshot the experiment state one last time. If you later resume the experiment, it might resume from stale state.

Ray Tune also accepts the SIGUSR1 signal to interrupt training gracefully. Use it when you run Ray Tune in a remote Ray task, because Ray filters out SIGINT and SIGTERM signals by default.

Stop with metric-based criteria#

Besides manual stopping, Tune provides several ways to stop experiments programmatically. The simplest is metric-based criteria, a fixed set of thresholds that determine when to stop.

You can define the stopping criteria with a dictionary, a function, or a custom Stopper.

If you pass a dictionary, its keys can be any field in the results that you report with tune.report in the function API or return from step() in the class API.

Note

This includes auto-filled metrics such as training_iteration.

In the following example, each trial stops when it completes 10 iterations or when it reaches a mean accuracy of 0.8 or more, whichever comes first.

Tune assumes these metrics are increasing, so a trial stops once the reported metric reaches or exceeds the threshold in the dictionary.

from ray import tune

tuner = tune.Tuner(
    my_trainable,
    run_config=tune.RunConfig(stop={"training_iteration": 10, "mean_accuracy": 0.8}),
)
result_grid = tuner.fit()

For more flexibility, you can pass a function instead. The function must take (trial_id: str, result: dict) as arguments and return True if the trial should stop or False otherwise.

In the following example, each trial stops when it completes 10 iterations or when it reaches a mean accuracy of 0.8 or more, whichever comes first.

from ray import tune


def stop_fn(trial_id: str, result: dict) -> bool:
    return result["mean_accuracy"] >= 0.8 or result["training_iteration"] >= 10


tuner = tune.Tuner(my_trainable, run_config=tune.RunConfig(stop=stop_fn))
result_grid = tuner.fit()

You can implement the Stopper interface to stop individual trials or entire experiments based on custom stopping criteria. For example, the following stopper stops all trials once any individual trial meets the criteria, and prevents new trials from starting:

from ray import tune
from ray.tune import Stopper


class CustomStopper(Stopper):
    def __init__(self):
        self.should_stop = False

    def __call__(self, trial_id: str, result: dict) -> bool:
        if not self.should_stop and result["mean_accuracy"] >= 0.8:
            self.should_stop = True
        return self.should_stop

    def stop_all(self) -> bool:
        """Returns whether to stop trials and prevent new ones from starting."""
        return self.should_stop


stopper = CustomStopper()
tuner = tune.Tuner(
    my_trainable,
    run_config=tune.RunConfig(stop=stopper),
    tune_config=tune.TuneConfig(num_samples=2),
)
result_grid = tuner.fit()

In this example, once any trial reaches a mean_accuracy of 0.8 or more, all trials stop.

Note

When stop_all returns True, running trials don’t stop immediately. Each one stops at the end of its ongoing training iteration, after its next tune.report call or step.

Ray Tune includes a set of built-in stopper classes. See the Stopper documentation.

Stop after a time limit#

To stop based on time, you can either stop each trial individually after a timeout or stop the whole experiment after a set amount of time.

Stop trials individually with a timeout#

You can use a dictionary of stopping criteria, as described earlier, with the time_total_s metric that Tune fills in automatically.

from ray import tune

tuner = tune.Tuner(
    my_trainable,
    # Stop a trial after it's run for more than 5 seconds.
    run_config=tune.RunConfig(stop={"time_total_s": 5}),
)
result_grid = tuner.fit()

Note

If you use the function trainable API, include some intermediate reporting through tune.report. Each report automatically records the trial’s time_total_s, so Tune can use time as a stopping metric.

If the training loop hangs, Tune can’t intercept the training and stop the trial for you. In that case, you can implement timeout logic explicitly in the training loop.

Stop the experiment with a timeout#

To stop the experiment after time_budget_s seconds, set time_budget_s in tune.TuneConfig.

from ray import tune

# Stop the entire experiment after ANY trial has run for more than 5 seconds.
tuner = tune.Tuner(my_trainable, tune_config=tune.TuneConfig(time_budget_s=5.0))
result_grid = tuner.fit()

Note

If you use the function trainable API, include some intermediate reporting through tune.report, for the same reason as in the preceding section.

Stop on trial failures#

Besides stopping trials based on their performance, you can stop the entire experiment if any trial encounters a runtime error. To do this, use the ray.tune.FailureConfig class.

With FailureConfig(fail_fast=True), the entire experiment stops immediately if any trial encounters an error.

from ray import tune
import time


def my_failing_trainable(config):
    if config["should_fail"]:
        raise RuntimeError("Failing (on purpose)!")
    # Do some training...
    time.sleep(10)
    tune.report({"mean_accuracy": 0.9})


tuner = tune.Tuner(
    my_failing_trainable,
    param_space={"should_fail": tune.grid_search([True, False])},
    run_config=tune.RunConfig(failure_config=tune.FailureConfig(fail_fast=True)),
)
result_grid = tuner.fit()

This is useful when you’re debugging a Tune experiment with many trials.

Stop trials early with Tune schedulers#

You can also stop trials with early-stopping schedulers. These schedulers monitor trial performance and stop trials early if they aren’t making sufficient progress.

AsyncHyperBandScheduler and HyperBandForBOHB are examples of early-stopping schedulers built into Tune. For a full list and more realistic examples, see the Tune scheduler API reference.

The following example combines a dictionary of stopping criteria with an early-stopping scheduler:

from ray import tune
from ray.tune.schedulers import AsyncHyperBandScheduler


scheduler = AsyncHyperBandScheduler(time_attr="training_iteration")

tuner = tune.Tuner(
    my_trainable,
    run_config=tune.RunConfig(stop={"training_iteration": 10}),
    tune_config=tune.TuneConfig(
        scheduler=scheduler, num_samples=2, metric="mean_accuracy", mode="max"
    ),
)
result_grid = tuner.fit()

Summary#

This guide covered how to stop Tune experiments manually, with metrics, after a time limit, on trial errors, and with early-stopping schedulers.

See the following resources for more information: