Tune Stopping mechanisms (tune.stopper)
Contents
Tune Stopping mechanisms (tune.stopper)#
In addition to Trial Schedulers like ASHA, where a number of trials are stopped if they perform subpar, Ray Tune also supports custom stopping mechanisms to stop trials early. They can also stop the entire experiment after a condition is met. For instance, stopping mechanisms can specify to stop trials when they reached a plateau and the metric doesnβt change anymore.
Ray Tune comes with several stopping mechanisms out of the box. For custom stopping behavior, you can
inherit from the Stopper
class.
Other stopping behaviors are described in the user guide.
Stopper (tune.Stopper)#
- class ray.tune.Stopper[source]#
Base class for implementing a Tune experiment stopper.
Allows users to implement experiment-level stopping via
stop_all
. By default, this class does not stop any trials. Subclasses need to implement__call__
andstop_all
.Examples
>>> import time >>> from ray import air, tune >>> from ray.tune import Stopper >>> >>> class TimeStopper(Stopper): ... def __init__(self): ... self._start = time.time() ... self._deadline = 5 ... ... def __call__(self, trial_id, result): ... return False ... ... def stop_all(self): ... return time.time() - self._start > self._deadline >>> >>> tuner = tune.Tuner( ... tune.Trainable, ... tune_config=tune.TuneConfig(num_samples=200), ... run_config=air.RunConfig(stop=TimeStopper()) ... ) >>> tuner.fit() == Status ==...
PublicAPI: This API is stable across Ray releases.
MaximumIterationStopper (tune.stopper.MaximumIterationStopper)#
ExperimentPlateauStopper (tune.stopper.ExperimentPlateauStopper)#
- class ray.tune.stopper.ExperimentPlateauStopper(metric: str, std: float = 0.001, top: int = 10, mode: str = 'min', patience: int = 0)[source]#
Early stop the experiment when a metric plateaued across trials.
Stops the entire experiment when the metric has plateaued for more than the given amount of iterations specified in the patience parameter.
- Parameters
metric β The metric to be monitored.
std β The minimal standard deviation after which the tuning process has to stop.
top β The number of best models to consider.
mode β The mode to select the top results. Can either be βminβ or βmaxβ.
patience β Number of epochs to wait for a change in the top models.
- Raises
ValueError β If the mode parameter is not βminβ nor βmaxβ.
ValueError β If the top parameter is not an integer greater than 1.
ValueError β If the standard deviation parameter is not a strictly positive float.
ValueError β If the patience parameter is not a strictly positive integer.
PublicAPI: This API is stable across Ray releases.
TrialPlateauStopper (tune.stopper.TrialPlateauStopper)#
- class ray.tune.stopper.TrialPlateauStopper(metric: str, std: float = 0.01, num_results: int = 4, grace_period: int = 4, metric_threshold: Optional[float] = None, mode: Optional[str] = None)[source]#
Early stop single trials when they reached a plateau.
When the standard deviation of the
metric
result of a trial is below a thresholdstd
, the trial plateaued and will be stopped early.- Parameters
metric β Metric to check for convergence.
std β Maximum metric standard deviation to decide if a trial plateaued. Defaults to 0.01.
num_results β Number of results to consider for stdev calculation.
grace_period β Minimum number of timesteps before a trial can be early stopped
metric_threshold (Optional[float]) β Minimum or maximum value the result has to exceed before it can be stopped early.
mode β If a
metric_threshold
argument has been passed, this must be one of [min, max]. Specifies if we optimize for a large metric (max) or a small metric (min). If max, themetric_threshold
has to be exceeded, if min the value has to be lower thanmetric_threshold
in order to early stop.
PublicAPI: This API is stable across Ray releases.
TimeoutStopper (tune.stopper.TimeoutStopper)#
- class ray.tune.stopper.TimeoutStopper(timeout: Union[int, float, datetime.timedelta])[source]#
Stops all trials after a certain timeout.
This stopper is automatically created when the
time_budget_s
argument is passed toair.RunConfig()
.- Parameters
timeout β Either a number specifying the timeout in seconds, or a
datetime.timedelta
object.
PublicAPI: This API is stable across Ray releases.
CombinedStopper (tune.stopper.CombinedStopper)#
- class ray.tune.stopper.CombinedStopper(*stoppers: ray.tune.stopper.stopper.Stopper)[source]#
Combine several stoppers via βORβ.
- Parameters
*stoppers β Stoppers to be combined.
Examples
>>> from ray.tune.stopper import (CombinedStopper, ... MaximumIterationStopper, TrialPlateauStopper) >>> >>> stopper = CombinedStopper( ... MaximumIterationStopper(max_iter=20), ... TrialPlateauStopper(metric="my_metric") ... ) >>> >>> tuner = tune.Tuner( ... tune.Trainable, ... run_config=air.RunConfig(stop=stopper) ... ) >>> tuner.fit() == Status ==...
PublicAPI: This API is stable across Ray releases.