Tune Stopping mechanisms (tune.stopper)#

In addition to Trial Schedulers like ASHA, where a number of trials are stopped if they perform subpar, Ray Tune also supports custom stopping mechanisms to stop trials early. They can also stop the entire experiment after a condition is met. For instance, stopping mechanisms can specify to stop trials when they reached a plateau and the metric doesn’t change anymore.

Ray Tune comes with several stopping mechanisms out of the box. For custom stopping behavior, you can inherit from the Stopper class.

Other stopping behaviors are described in the user guide.

Stopper (tune.Stopper)#

class ray.tune.Stopper[source]#

Base class for implementing a Tune experiment stopper.

Allows users to implement experiment-level stopping via stop_all. By default, this class does not stop any trials. Subclasses need to implement __call__ and stop_all.

Examples

>>> import time
>>> from ray import air, tune
>>> from ray.tune import Stopper
>>>
>>> class TimeStopper(Stopper):
...     def __init__(self):
...         self._start = time.time()
...         self._deadline = 5
...
...     def __call__(self, trial_id, result):
...         return False
...
...     def stop_all(self):
...         return time.time() - self._start > self._deadline
>>>
>>> tuner = tune.Tuner(
...     tune.Trainable,
...     tune_config=tune.TuneConfig(num_samples=200),
...     run_config=air.RunConfig(stop=TimeStopper())
... )
>>> tuner.fit()
== Status ==...

PublicAPI: This API is stable across Ray releases.

__call__(trial_id, result)[source]#

Returns true if the trial should be terminated given the result.

stop_all()[source]#

Returns true if the experiment should be terminated.

MaximumIterationStopper (tune.stopper.MaximumIterationStopper)#

class ray.tune.stopper.MaximumIterationStopper(max_iter: int)[source]#

Stop trials after reaching a maximum number of iterations

Parameters

max_iter – Number of iterations before stopping a trial.

PublicAPI: This API is stable across Ray releases.

ExperimentPlateauStopper (tune.stopper.ExperimentPlateauStopper)#

class ray.tune.stopper.ExperimentPlateauStopper(metric: str, std: float = 0.001, top: int = 10, mode: str = 'min', patience: int = 0)[source]#

Early stop the experiment when a metric plateaued across trials.

Stops the entire experiment when the metric has plateaued for more than the given amount of iterations specified in the patience parameter.

Parameters

  • metric – The metric to be monitored.

  • std – The minimal standard deviation after which the tuning process has to stop.

  • top – The number of best models to consider.

  • mode – The mode to select the top results. Can either be β€œmin” or β€œmax”.

  • patience – Number of epochs to wait for a change in the top models.

Raises

  • ValueError – If the mode parameter is not β€œmin” nor β€œmax”.

  • ValueError – If the top parameter is not an integer greater than 1.

  • ValueError – If the standard deviation parameter is not a strictly positive float.

  • ValueError – If the patience parameter is not a strictly positive integer.

PublicAPI: This API is stable across Ray releases.

TrialPlateauStopper (tune.stopper.TrialPlateauStopper)#

class ray.tune.stopper.TrialPlateauStopper(metric: str, std: float = 0.01, num_results: int = 4, grace_period: int = 4, metric_threshold: Optional[float] = None, mode: Optional[str] = None)[source]#

Early stop single trials when they reached a plateau.

When the standard deviation of the metric result of a trial is below a threshold std, the trial plateaued and will be stopped early.

Parameters

  • metric – Metric to check for convergence.

  • std – Maximum metric standard deviation to decide if a trial plateaued. Defaults to 0.01.

  • num_results – Number of results to consider for stdev calculation.

  • grace_period – Minimum number of timesteps before a trial can be early stopped

  • metric_threshold (Optional[float]) – Minimum or maximum value the result has to exceed before it can be stopped early.

  • mode – If a metric_threshold argument has been passed, this must be one of [min, max]. Specifies if we optimize for a large metric (max) or a small metric (min). If max, the metric_threshold has to be exceeded, if min the value has to be lower than metric_threshold in order to early stop.

PublicAPI: This API is stable across Ray releases.

TimeoutStopper (tune.stopper.TimeoutStopper)#

class ray.tune.stopper.TimeoutStopper(timeout: Union[int, float, datetime.timedelta])[source]#

Stops all trials after a certain timeout.

This stopper is automatically created when the time_budget_s argument is passed to air.RunConfig().

Parameters

timeout – Either a number specifying the timeout in seconds, or a datetime.timedelta object.

PublicAPI: This API is stable across Ray releases.

CombinedStopper (tune.stopper.CombinedStopper)#

class ray.tune.stopper.CombinedStopper(*stoppers: ray.tune.stopper.stopper.Stopper)[source]#

Combine several stoppers via β€˜OR’.

Parameters

*stoppers – Stoppers to be combined.

Examples

>>> from ray.tune.stopper import (CombinedStopper,
...     MaximumIterationStopper, TrialPlateauStopper)
>>>
>>> stopper = CombinedStopper(
...     MaximumIterationStopper(max_iter=20),
...     TrialPlateauStopper(metric="my_metric")
... )
>>>
>>> tuner = tune.Tuner(
...     tune.Trainable,
...     run_config=air.RunConfig(stop=stopper)
... )
>>> tuner.fit()
== Status ==...

PublicAPI: This API is stable across Ray releases.