Search Algorithms (tune.suggest)¶
Tune’s Search Algorithms are wrappers around open-source optimization libraries for efficient hyperparameter selection. Each library has a specific way of defining the search space - please refer to their documentation for more details.
You can utilize these search algorithms as follows:
from ray.tune.suggest.hyperopt import HyperOptSearch
tune.run(my_function, search_alg=HyperOptSearch(...))
Summary¶
SearchAlgorithm |
Summary |
Website |
Code Example |
---|---|---|---|
Random search/grid search |
|||
Bayesian/Bandit Optimization |
[Ax] |
||
Scalable Bayesian Optimization |
|||
Bayesian Optimization |
|||
Tree-Parzen Estimators |
[HyperOpt] |
||
Bayesian Optimization |
|||
Bayesian Opt/HyperBand |
[BOHB] |
||
Gradient-free Optimization |
|||
Optuna search algorithms |
[Optuna] |
||
Zeroth-order Optimization |
[ZOOpt] |
||
Closed source |
[SigOpt] |
Note
Unlike Tune’s Trial Schedulers, Tune SearchAlgorithms cannot affect or stop training processes. However, you can use them together to early stop the evaluation of bad trials.
Want to use your own algorithm? The interface is easy to implement. Read instructions here.
Tune also provides helpful utilities to use with Search Algorithms:
Repeated Evaluations (tune.suggest.Repeater): Support for running each sampled hyperparameter with multiple random seeds.
ConcurrencyLimiter (tune.suggest.ConcurrencyLimiter): Limits the amount of concurrent trials when running optimization.
Shim Instantiation (tune.create_searcher): Allows creation of the search algorithm object given a string.
Saving and Restoring¶
Certain search algorithms have save/restore
implemented,
allowing reuse of learnings across multiple tuning runs.
search_alg = HyperOptSearch()
experiment_1 = tune.run(
trainable,
search_alg=search_alg)
search_alg.save("./my-checkpoint.pkl")
# Restore the saved state onto another search algorithm
search_alg2 = HyperOptSearch()
search_alg2.restore("./my-checkpoint.pkl")
experiment_2 = tune.run(
trainable,
search_alg=search_alg2)
Further, Tune automatically saves its state inside the current experiment folder (“Result Dir”) during tuning.
Note that if you have two Tune runs with the same experiment folder,
the previous state checkpoint will be overwritten. You can
avoid this by making sure tune.run(name=...)
is set to a unique
identifier.
search_alg = HyperOptSearch()
experiment_1 = tune.run(
cost,
num_samples=5,
search_alg=search_alg,
verbose=0,
name="my-experiment-1",
local_dir="~/my_results")
search_alg2 = HyperOptSearch()
search_alg2.restore_from_dir(
os.path.join("~/my_results", "my-experiment-1"))
Note
This is currently not implemented for: AxSearch, TuneBOHB, SigOptSearch, and DragonflySearch.
Random search and grid search (tune.suggest.basic_variant.BasicVariantGenerator)¶
The default and most basic way to do hyperparameter search is via random and grid search.
Ray Tune does this through the BasicVariantGenerator
class that generates trial variants given a search space definition.
The BasicVariantGenerator
is used per
default if no search algorithm is passed to
tune.run()
.
-
class
ray.tune.suggest.basic_variant.
BasicVariantGenerator
(points_to_evaluate: Optional[List[Dict]] = None)[source]¶ Uses Tune’s variant generation for resolving variables.
This is the default search algorithm used if no other search algorithm is specified.
- Parameters
points_to_evaluate (list) – Initial parameter suggestions to be run first. This is for when you already have some good parameters you want to run first to help the algorithm make better suggestions for future parameters. Needs to be a list of dicts containing the configurations.
Example:
from ray import tune # This will automatically use the `BasicVariantGenerator` tune.run( lambda config: config["a"] + config["b"], config={ "a": tune.grid_search([1, 2]), "b": tune.randint(0, 3) }, num_samples=4)
In the example above, 8 trials will be generated: For each sample (
4
), each of the grid search variants fora
will be sampled once. Theb
parameter will be sampled randomly.The generator accepts a pre-set list of points that should be evaluated. The points will replace the first samples of each experiment passed to the
BasicVariantGenerator
.Each point will replace one sample of the specified
num_samples
. If grid search variables are overwritten with the values specified in the presets, the number of samples will thus be reduced.Example:
from ray import tune from ray.tune.suggest.basic_variant import BasicVariantGenerator tune.run( lambda config: config["a"] + config["b"], config={ "a": tune.grid_search([1, 2]), "b": tune.randint(0, 3) }, search_alg=BasicVariantGenerator(points_to_evaluate=[ {"a": 2, "b": 2}, {"a": 1}, {"b": 2} ]), num_samples=4)
The example above will produce six trials via four samples:
The first sample will produce one trial with
a=2
andb=2
.The second sample will produce one trial with
a=1
andb
sampled randomlyThe third sample will produce two trials, one for each grid search value of
a
. It will beb=2
for both of these trials.The fourth sample will produce two trials, one for each grid search value of
a
.b
will be sampled randomly and independently for both of these trials.
Ax (tune.suggest.ax.AxSearch)¶
-
class
ray.tune.suggest.ax.
AxSearch
(space: Union[Dict, List[Dict], None] = None, metric: Optional[str] = None, mode: Optional[str] = None, points_to_evaluate: Optional[List[Dict]] = None, parameter_constraints: Optional[List] = None, outcome_constraints: Optional[List] = None, ax_client: Optional[<Mock name='mock.AxClient' id='140667056466384'>] = None, use_early_stopped_trials: Optional[bool] = None, max_concurrent: Optional[int] = None)[source]¶ Uses Ax to optimize hyperparameters.
Ax is a platform for understanding, managing, deploying, and automating adaptive experiments. Ax provides an easy to use interface with BoTorch, a flexible, modern library for Bayesian optimization in PyTorch. More information can be found in https://ax.dev/.
To use this search algorithm, you must install Ax and sqlalchemy:
$ pip install ax-platform sqlalchemy
- Parameters
space (list[dict]) – Parameters in the experiment search space. Required elements in the dictionaries are: “name” (name of this parameter, string), “type” (type of the parameter: “range”, “fixed”, or “choice”, string), “bounds” for range parameters (list of two values, lower bound first), “values” for choice parameters (list of values), and “value” for fixed parameters (single value).
metric (str) – Name of the metric used as objective in this experiment. This metric must be present in raw_data argument to log_data. This metric must also be present in the dict reported/returned by the Trainable. If None but a mode was passed, the ray.tune.result.DEFAULT_METRIC will be used per default.
mode (str) – One of {min, max}. Determines whether objective is minimizing or maximizing the metric attribute. Defaults to “max”.
points_to_evaluate (list) – Initial parameter suggestions to be run first. This is for when you already have some good parameters you want to run first to help the algorithm make better suggestions for future parameters. Needs to be a list of dicts containing the configurations.
parameter_constraints (list[str]) – Parameter constraints, such as “x3 >= x4” or “x3 + x4 >= 2”.
outcome_constraints (list[str]) – Outcome constraints of form “metric_name >= bound”, like “m1 <= 3.”
ax_client (AxClient) – Optional AxClient instance. If this is set, do not pass any values to these parameters: space, metric, parameter_constraints, outcome_constraints.
use_early_stopped_trials – Deprecated.
max_concurrent (int) – Deprecated.
Tune automatically converts search spaces to Ax’s format:
from ray import tune from ray.tune.suggest.ax import AxSearch config = { "x1": tune.uniform(0.0, 1.0), "x2": tune.uniform(0.0, 1.0) } def easy_objective(config): for i in range(100): intermediate_result = config["x1"] + config["x2"] * i tune.report(score=intermediate_result) ax_search = AxSearch(metric="score") tune.run( config=config, easy_objective, search_alg=ax_search)
If you would like to pass the search space manually, the code would look like this:
from ray import tune from ray.tune.suggest.ax import AxSearch parameters = [ {"name": "x1", "type": "range", "bounds": [0.0, 1.0]}, {"name": "x2", "type": "range", "bounds": [0.0, 1.0]}, ] def easy_objective(config): for i in range(100): intermediate_result = config["x1"] + config["x2"] * i tune.report(score=intermediate_result) ax_search = AxSearch(space=parameters, metric="score") tune.run(easy_objective, search_alg=ax_search)
Bayesian Optimization (tune.suggest.bayesopt.BayesOptSearch)¶
-
class
ray.tune.suggest.bayesopt.
BayesOptSearch
(space: Optional[Dict] = None, metric: Optional[str] = None, mode: Optional[str] = None, points_to_evaluate: Optional[List[Dict]] = None, utility_kwargs: Optional[Dict] = None, random_state: int = 42, random_search_steps: int = 10, verbose: int = 0, patience: int = 5, skip_duplicate: bool = True, analysis: Optional[ray.tune.analysis.experiment_analysis.ExperimentAnalysis] = None, max_concurrent: Optional[int] = None, use_early_stopped_trials: Optional[bool] = None)[source]¶ Uses fmfn/BayesianOptimization to optimize hyperparameters.
fmfn/BayesianOptimization is a library for Bayesian Optimization. More info can be found here: https://github.com/fmfn/BayesianOptimization.
This searcher will automatically filter out any NaN, inf or -inf results.
You will need to install fmfn/BayesianOptimization via the following:
pip install bayesian-optimization
This algorithm requires setting a search space using the BayesianOptimization search space specification.
- Parameters
space (dict) – Continuous search space. Parameters will be sampled from this space which will be used to run trials.
metric (str) – The training result objective value attribute. If None but a mode was passed, the anonymous metric _metric will be used per default.
mode (str) – One of {min, max}. Determines whether objective is minimizing or maximizing the metric attribute.
points_to_evaluate (list) – Initial parameter suggestions to be run first. This is for when you already have some good parameters you want to run first to help the algorithm make better suggestions for future parameters. Needs to be a list of dicts containing the configurations.
utility_kwargs (dict) – Parameters to define the utility function. The default value is a dictionary with three keys: - kind: ucb (Upper Confidence Bound) - kappa: 2.576 - xi: 0.0
random_state (int) – Used to initialize BayesOpt.
random_search_steps (int) – Number of initial random searches. This is necessary to avoid initial local overfitting of the Bayesian process.
analysis (ExperimentAnalysis) – Optionally, the previous analysis to integrate.
verbose (int) – Sets verbosity level for BayesOpt packages.
max_concurrent – Deprecated.
use_early_stopped_trials – Deprecated.
Tune automatically converts search spaces to BayesOptSearch’s format:
from ray import tune from ray.tune.suggest.bayesopt import BayesOptSearch config = { "width": tune.uniform(0, 20), "height": tune.uniform(-100, 100) } bayesopt = BayesOptSearch(metric="mean_loss", mode="min") tune.run(my_func, config=config, search_alg=bayesopt)
If you would like to pass the search space manually, the code would look like this:
from ray import tune from ray.tune.suggest.bayesopt import BayesOptSearch space = { 'width': (0, 20), 'height': (-100, 100), } bayesopt = BayesOptSearch(space, metric="mean_loss", mode="min") tune.run(my_func, search_alg=bayesopt)
BOHB (tune.suggest.bohb.TuneBOHB)¶
BOHB (Bayesian Optimization HyperBand) is an algorithm that both terminates bad trials and also uses Bayesian Optimization to improve the hyperparameter search. It is backed by the HpBandSter library.
Importantly, BOHB is intended to be paired with a specific scheduler class: HyperBandForBOHB.
This algorithm requires using the ConfigSpace search space specification. In order to use this search algorithm, you will need to install HpBandSter
and ConfigSpace
:
$ pip install hpbandster ConfigSpace
See the BOHB paper for more details.
-
class
ray.tune.suggest.bohb.
TuneBOHB
(space: Union[Dict, <Mock name='mock.ConfigurationSpace' id='140667056149200'>, None] = None, bohb_config: Optional[Dict] = None, max_concurrent: int = 10, metric: Optional[str] = None, mode: Optional[str] = None, points_to_evaluate: Optional[List[Dict]] = None, seed: Optional[int] = None)[source]¶ BOHB suggestion component.
Requires HpBandSter and ConfigSpace to be installed. You can install HpBandSter and ConfigSpace with:
pip install hpbandster ConfigSpace
.This should be used in conjunction with HyperBandForBOHB.
- Parameters
space (ConfigurationSpace) – Continuous ConfigSpace search space. Parameters will be sampled from this space which will be used to run trials.
bohb_config (dict) – configuration for HpBandSter BOHB algorithm
max_concurrent (int) – Number of maximum concurrent trials. Defaults to 10.
metric (str) – The training result objective value attribute. If None but a mode was passed, the anonymous metric _metric will be used per default.
mode (str) – One of {min, max}. Determines whether objective is minimizing or maximizing the metric attribute.
points_to_evaluate (list) – Initial parameter suggestions to be run first. This is for when you already have some good parameters you want to run first to help the algorithm make better suggestions for future parameters. Needs to be a list of dicts containing the configurations.
seed (int) – Optional random seed to initialize the random number generator. Setting this should lead to identical initial configurations at each run.
Tune automatically converts search spaces to TuneBOHB’s format:
config = { "width": tune.uniform(0, 20), "height": tune.uniform(-100, 100), "activation": tune.choice(["relu", "tanh"]) } algo = TuneBOHB(max_concurrent=4, metric="mean_loss", mode="min") bohb = HyperBandForBOHB( time_attr="training_iteration", metric="mean_loss", mode="min", max_t=100) run(my_trainable, config=config, scheduler=bohb, search_alg=algo)
If you would like to pass the search space manually, the code would look like this:
import ConfigSpace as CS config_space = CS.ConfigurationSpace() config_space.add_hyperparameter( CS.UniformFloatHyperparameter("width", lower=0, upper=20)) config_space.add_hyperparameter( CS.UniformFloatHyperparameter("height", lower=-100, upper=100)) config_space.add_hyperparameter( CS.CategoricalHyperparameter( name="activation", choices=["relu", "tanh"])) algo = TuneBOHB( config_space, max_concurrent=4, metric="mean_loss", mode="min") bohb = HyperBandForBOHB( time_attr="training_iteration", metric="mean_loss", mode="min", max_t=100) run(my_trainable, scheduler=bohb, search_alg=algo)
Dragonfly (tune.suggest.dragonfly.DragonflySearch)¶
-
class
ray.tune.suggest.dragonfly.
DragonflySearch
(optimizer: Optional[str] = None, domain: Optional[str] = None, space: Union[Dict, List[Dict], None] = None, metric: Optional[str] = None, mode: Optional[str] = None, points_to_evaluate: Optional[List[Dict]] = None, evaluated_rewards: Optional[List] = None, **kwargs)[source]¶ Uses Dragonfly to optimize hyperparameters.
Dragonfly provides an array of tools to scale up Bayesian optimisation to expensive large scale problems, including high dimensional optimisation. parallel evaluations in synchronous or asynchronous settings, multi-fidelity optimisation (using cheap approximations to speed up the optimisation process), and multi-objective optimisation. For more info:
Dragonfly Website: https://github.com/dragonfly/dragonfly
Dragonfly Documentation: https://dragonfly-opt.readthedocs.io/
To use this search algorithm, install Dragonfly:
$ pip install dragonfly-opt
This interface requires using FunctionCallers and optimizers provided by Dragonfly.
This searcher will automatically filter out any NaN, inf or -inf results.
- Parameters
optimizer (dragonfly.opt.BlackboxOptimiser|str) – Optimizer provided from dragonfly. Choose an optimiser that extends BlackboxOptimiser. If this is a string, domain must be set and optimizer must be one of [random, bandit, genetic].
domain (str) – Optional domain. Should only be set if you don’t pass an optimizer as the optimizer argument. Has to be one of [cartesian, euclidean].
space (list|dict) – Search space. Should only be set if you don’t pass an optimizer as the optimizer argument. Defines the search space and requires a domain to be set. Can be automatically converted from the config dict passed to tune.run().
metric (str) – The training result objective value attribute. If None but a mode was passed, the anonymous metric _metric will be used per default.
mode (str) – One of {min, max}. Determines whether objective is minimizing or maximizing the metric attribute.
points_to_evaluate (list) – Initial parameter suggestions to be run first. This is for when you already have some good parameters you want to run first to help the algorithm make better suggestions for future parameters. Needs to be a list of dicts containing the configurations.
evaluated_rewards (list) – If you have previously evaluated the parameters passed in as points_to_evaluate you can avoid re-running those trials by passing in the reward attributes as a list so the optimiser can be told the results without needing to re-compute the trial. Must be the same length as points_to_evaluate.
Tune automatically converts search spaces to Dragonfly’s format:
from ray import tune config = { "LiNO3_vol": tune.uniform(0, 7), "Li2SO4_vol": tune.uniform(0, 7), "NaClO4_vol": tune.uniform(0, 7) } df_search = DragonflySearch( optimizer="bandit", domain="euclidean", metric="objective", mode="max") tune.run(my_func, config=config, search_alg=df_search)
If you would like to pass the search space/optimizer manually, the code would look like this:
from ray import tune space = [{ "name": "LiNO3_vol", "type": "float", "min": 0, "max": 7 }, { "name": "Li2SO4_vol", "type": "float", "min": 0, "max": 7 }, { "name": "NaClO4_vol", "type": "float", "min": 0, "max": 7 }] df_search = DragonflySearch( optimizer="bandit", domain="euclidean", space=space, metric="objective", mode="max") tune.run(my_func, search_alg=df_search)
-
save
(checkpoint_path: str)[source]¶ Save state to path for this search algorithm.
- Parameters
checkpoint_path (str) – File where the search algorithm state is saved. This path should be used later when restoring from file.
Example:
search_alg = Searcher(...) analysis = tune.run( cost, num_samples=5, search_alg=search_alg, name=self.experiment_name, local_dir=self.tmpdir) search_alg.save("./my_favorite_path.pkl")
Changed in version 0.8.7: Save is automatically called by tune.run. You can use restore_from_dir to restore from an experiment directory such as ~/ray_results/trainable.
-
restore
(checkpoint_dir: str)[source]¶ Restore state for this search algorithm
- Parameters
checkpoint_path (str) – File where the search algorithm state is saved. This path should be the same as the one provided to “save”.
Example:
search_alg.save("./my_favorite_path.pkl") search_alg2 = Searcher(...) search_alg2 = ConcurrencyLimiter(search_alg2, 1) search_alg2.restore(checkpoint_path) tune.run(cost, num_samples=5, search_alg=search_alg2)
HyperOpt (tune.suggest.hyperopt.HyperOptSearch)¶
-
class
ray.tune.suggest.hyperopt.
HyperOptSearch
(space: Optional[Dict] = None, metric: Optional[str] = None, mode: Optional[str] = None, points_to_evaluate: Optional[List[Dict]] = None, n_initial_points: int = 20, random_state_seed: Optional[int] = None, gamma: float = 0.25, max_concurrent: Optional[int] = None, use_early_stopped_trials: Optional[bool] = None)[source]¶ A wrapper around HyperOpt to provide trial suggestions.
HyperOpt a Python library for serial and parallel optimization over awkward search spaces, which may include real-valued, discrete, and conditional dimensions. More info can be found at http://hyperopt.github.io/hyperopt.
HyperOptSearch uses the Tree-structured Parzen Estimators algorithm, though it can be trivially extended to support any algorithm HyperOpt supports.
To use this search algorithm, you will need to install HyperOpt:
pip install -U hyperopt
- Parameters
space (dict) – HyperOpt configuration. Parameters will be sampled from this configuration and will be used to override parameters generated in the variant generation process.
metric (str) – The training result objective value attribute. If None but a mode was passed, the anonymous metric _metric will be used per default.
mode (str) – One of {min, max}. Determines whether objective is minimizing or maximizing the metric attribute.
points_to_evaluate (list) – Initial parameter suggestions to be run first. This is for when you already have some good parameters you want to run first to help the algorithm make better suggestions for future parameters. Needs to be a list of dicts containing the configurations.
n_initial_points (int) – number of random evaluations of the objective function before starting to aproximate it with tree parzen estimators. Defaults to 20.
random_state_seed (int, array_like, None) – seed for reproducible results. Defaults to None.
gamma (float in range (0,1)) – parameter governing the tree parzen estimators suggestion algorithm. Defaults to 0.25.
max_concurrent – Deprecated.
use_early_stopped_trials – Deprecated.
Tune automatically converts search spaces to HyperOpt’s format:
config = { 'width': tune.uniform(0, 20), 'height': tune.uniform(-100, 100), 'activation': tune.choice(["relu", "tanh"]) } current_best_params = [{ 'width': 10, 'height': 0, 'activation': "relu", }] hyperopt_search = HyperOptSearch( metric="mean_loss", mode="min", points_to_evaluate=current_best_params) tune.run(trainable, config=config, search_alg=hyperopt_search)
If you would like to pass the search space manually, the code would look like this:
space = { 'width': hp.uniform('width', 0, 20), 'height': hp.uniform('height', -100, 100), 'activation': hp.choice("activation", ["relu", "tanh"]) } current_best_params = [{ 'width': 10, 'height': 0, 'activation': "relu", }] hyperopt_search = HyperOptSearch( space, metric="mean_loss", mode="min", points_to_evaluate=current_best_params) tune.run(trainable, search_alg=hyperopt_search)
-
save
(checkpoint_path: str)[source]¶ Save state to path for this search algorithm.
- Parameters
checkpoint_path (str) – File where the search algorithm state is saved. This path should be used later when restoring from file.
Example:
search_alg = Searcher(...) analysis = tune.run( cost, num_samples=5, search_alg=search_alg, name=self.experiment_name, local_dir=self.tmpdir) search_alg.save("./my_favorite_path.pkl")
Changed in version 0.8.7: Save is automatically called by tune.run. You can use restore_from_dir to restore from an experiment directory such as ~/ray_results/trainable.
-
restore
(checkpoint_path: str)[source]¶ Restore state for this search algorithm
- Parameters
checkpoint_path (str) – File where the search algorithm state is saved. This path should be the same as the one provided to “save”.
Example:
search_alg.save("./my_favorite_path.pkl") search_alg2 = Searcher(...) search_alg2 = ConcurrencyLimiter(search_alg2, 1) search_alg2.restore(checkpoint_path) tune.run(cost, num_samples=5, search_alg=search_alg2)
Nevergrad (tune.suggest.nevergrad.NevergradSearch)¶
-
class
ray.tune.suggest.nevergrad.
NevergradSearch
(optimizer: None = None, space: Optional[Dict] = None, metric: Optional[str] = None, mode: Optional[str] = None, points_to_evaluate: Optional[List[Dict]] = None, max_concurrent: Optional[int] = None, **kwargs)[source]¶ Uses Nevergrad to optimize hyperparameters.
Nevergrad is an open source tool from Facebook for derivative free optimization. More info can be found at: https://github.com/facebookresearch/nevergrad.
You will need to install Nevergrad via the following command:
$ pip install nevergrad
- Parameters
optimizer (nevergrad.optimization.Optimizer|class) – Optimizer provided from Nevergrad. Alter
space (list|nevergrad.parameter.Parameter) – Nevergrad parametrization to be passed to optimizer on instantiation, or list of parameter names if you passed an optimizer object.
metric (str) – The training result objective value attribute. If None but a mode was passed, the anonymous metric _metric will be used per default.
mode (str) – One of {min, max}. Determines whether objective is minimizing or maximizing the metric attribute.
points_to_evaluate (list) – Initial parameter suggestions to be run first. This is for when you already have some good parameters you want to run first to help the algorithm make better suggestions for future parameters. Needs to be a list of dicts containing the configurations.
use_early_stopped_trials – Deprecated.
max_concurrent – Deprecated.
Tune automatically converts search spaces to Nevergrad’s format:
import nevergrad as ng config = { "width": tune.uniform(0, 20), "height": tune.uniform(-100, 100), "activation": tune.choice(["relu", "tanh"]) } current_best_params = [{ "width": 10, "height": 0, "activation": relu", }] ng_search = NevergradSearch( optimizer=ng.optimizers.OnePlusOne, metric="mean_loss", mode="min", points_to_evaluate=current_best_params) run(my_trainable, config=config, search_alg=ng_search)
If you would like to pass the search space manually, the code would look like this:
import nevergrad as ng space = ng.p.Dict( width=ng.p.Scalar(lower=0, upper=20), height=ng.p.Scalar(lower=-100, upper=100), activation=ng.p.Choice(choices=["relu", "tanh"]) ) ng_search = NevergradSearch( optimizer=ng.optimizers.OnePlusOne, space=space, metric="mean_loss", mode="min") run(my_trainable, search_alg=ng_search)
-
save
(checkpoint_path: str)[source]¶ Save state to path for this search algorithm.
- Parameters
checkpoint_path (str) – File where the search algorithm state is saved. This path should be used later when restoring from file.
Example:
search_alg = Searcher(...) analysis = tune.run( cost, num_samples=5, search_alg=search_alg, name=self.experiment_name, local_dir=self.tmpdir) search_alg.save("./my_favorite_path.pkl")
Changed in version 0.8.7: Save is automatically called by tune.run. You can use restore_from_dir to restore from an experiment directory such as ~/ray_results/trainable.
-
restore
(checkpoint_path: str)[source]¶ Restore state for this search algorithm
- Parameters
checkpoint_path (str) – File where the search algorithm state is saved. This path should be the same as the one provided to “save”.
Example:
search_alg.save("./my_favorite_path.pkl") search_alg2 = Searcher(...) search_alg2 = ConcurrencyLimiter(search_alg2, 1) search_alg2.restore(checkpoint_path) tune.run(cost, num_samples=5, search_alg=search_alg2)
Optuna (tune.suggest.optuna.OptunaSearch)¶
-
class
ray.tune.suggest.optuna.
OptunaSearch
(space: Union[Dict, List[Tuple], None] = None, metric: Optional[str] = None, mode: Optional[str] = None, points_to_evaluate: Optional[List[Dict]] = None, sampler: None = None)[source]¶ A wrapper around Optuna to provide trial suggestions.
Optuna is a hyperparameter optimization library. In contrast to other libraries, it employs define-by-run style hyperparameter definitions.
This Searcher is a thin wrapper around Optuna’s search algorithms. You can pass any Optuna sampler, which will be used to generate hyperparameter suggestions.
Please note that this wrapper does not support define-by-run, so the search space will be configured before running the optimization. You will also need to use a Tune trainable (e.g. using the function API) with this wrapper.
For defining the search space, use
ray.tune.suggest.optuna.param
(see example).- Parameters
space (list) – Hyperparameter search space definition for Optuna’s sampler. This is a list, and samples for the parameters will be obtained in order.
metric (str) – The training result objective value attribute. If None but a mode was passed, the anonymous metric _metric will be used per default.
mode (str) – One of {min, max}. Determines whether objective is minimizing or maximizing the metric attribute.
points_to_evaluate (list) – Initial parameter suggestions to be run first. This is for when you already have some good parameters you want to run first to help the algorithm make better suggestions for future parameters. Needs to be a list of dicts containing the configurations.
sampler (optuna.samplers.BaseSampler) – Optuna sampler used to draw hyperparameter configurations. Defaults to
TPESampler
.
Tune automatically converts search spaces to Optuna’s format:
from ray.tune.suggest.optuna import OptunaSearch config = { "a": tune.uniform(6, 8) "b": tune.uniform(10, 20) } optuna_search = OptunaSearch( metric="loss", mode="min") tune.run(trainable, config=config, search_alg=optuna_search)
If you would like to pass the search space manually, the code would look like this:
from ray.tune.suggest.optuna import OptunaSearch, param space = [ param.suggest_uniform("a", 6, 8), param.suggest_uniform("b", 10, 20) ] algo = OptunaSearch( space, metric="loss", mode="min") tune.run(trainable, search_alg=optuna_search)
New in version 0.8.8.
SigOpt (tune.suggest.sigopt.SigOptSearch)¶
You will need to use the SigOpt experiment and space specification to specify your search space.
-
class
ray.tune.suggest.sigopt.
SigOptSearch
(space: List[Dict] = None, name: str = 'Default Tune Experiment', max_concurrent: int = 1, reward_attr: Optional[str] = None, connection: None = None, experiment_id: Optional[str] = None, observation_budget: Optional[int] = None, project: Optional[str] = None, metric: Union[None, str, List[str]] = 'episode_reward_mean', mode: Union[None, str, List[str]] = 'max', points_to_evaluate: Optional[List[Dict]] = None, **kwargs)[source]¶ A wrapper around SigOpt to provide trial suggestions.
You must install SigOpt and have a SigOpt API key to use this module. Store the API token as an environment variable
SIGOPT_KEY
as follows:pip install -U sigopt export SIGOPT_KEY= ...
You will need to use the SigOpt experiment and space specification.
This module manages its own concurrency.
- Parameters
space (list of dict) – SigOpt configuration. Parameters will be sampled from this configuration and will be used to override parameters generated in the variant generation process. Not used if existing experiment_id is given
name (str) – Name of experiment. Required by SigOpt.
max_concurrent (int) – Number of maximum concurrent trials supported based on the user’s SigOpt plan. Defaults to 1.
connection (Connection) – An existing connection to SigOpt.
experiment_id (str) – Optional, if given will connect to an existing experiment. This allows for a more interactive experience with SigOpt, such as prior beliefs and constraints.
observation_budget (int) – Optional, can improve SigOpt performance.
project (str) – Optional, Project name to assign this experiment to. SigOpt can group experiments by project
metric (str or list(str)) – If str then the training result objective value attribute. If list(str) then a list of metrics that can be optimized together. SigOpt currently supports up to 2 metrics.
mode (str or list(str)) – If experiment_id is given then this field is ignored, If str then must be one of {min, max}. If list then must be comprised of {min, max, obs}. Determines whether objective is minimizing or maximizing the metric attribute. If metrics is a list then mode must be a list of the same length as metric.
Example:
space = [ { 'name': 'width', 'type': 'int', 'bounds': { 'min': 0, 'max': 20 }, }, { 'name': 'height', 'type': 'int', 'bounds': { 'min': -100, 'max': 100 }, }, ] algo = SigOptSearch( space, name="SigOpt Example Experiment", max_concurrent=1, metric="mean_loss", mode="min") Example:
space = [ { 'name': 'width', 'type': 'int', 'bounds': { 'min': 0, 'max': 20 }, }, { 'name': 'height', 'type': 'int', 'bounds': { 'min': -100, 'max': 100 }, }, ] algo = SigOptSearch( space, name="SigOpt Multi Objective Example Experiment", max_concurrent=1, metric=["average", "std"], mode=["max", "min"])
Scikit-Optimize (tune.suggest.skopt.SkOptSearch)¶
-
class
ray.tune.suggest.skopt.
SkOptSearch
(optimizer: Optional[sko.optimizer.Optimizer] = None, space: Union[List[str], Dict[str, Union[Tuple, List]]] = None, metric: Optional[str] = None, mode: Optional[str] = None, points_to_evaluate: Optional[List[Dict]] = None, evaluated_rewards: Optional[List] = None, max_concurrent: Optional[int] = None, use_early_stopped_trials: Optional[bool] = None)[source]¶ Uses Scikit Optimize (skopt) to optimize hyperparameters.
Scikit-optimize is a black-box optimization library. Read more here: https://scikit-optimize.github.io.
You will need to install Scikit-Optimize to use this module.
pip install scikit-optimize
This Search Algorithm requires you to pass in a skopt Optimizer object.
This searcher will automatically filter out any NaN, inf or -inf results.
- Parameters
optimizer (skopt.optimizer.Optimizer) – Optimizer provided from skopt.
space (dict|list) – A dict mapping parameter names to valid parameters, i.e. tuples for numerical parameters and lists for categorical parameters. If you passed an optimizer instance as the optimizer argument, this should be a list of parameter names instead.
metric (str) – The training result objective value attribute. If None but a mode was passed, the anonymous metric _metric will be used per default.
mode (str) – One of {min, max}. Determines whether objective is minimizing or maximizing the metric attribute.
points_to_evaluate (list) – Initial parameter suggestions to be run first. This is for when you already have some good parameters you want to run first to help the algorithm make better suggestions for future parameters. Needs to be a list of dicts containing the configurations.
evaluated_rewards (list) – If you have previously evaluated the parameters passed in as points_to_evaluate you can avoid re-running those trials by passing in the reward attributes as a list so the optimiser can be told the results without needing to re-compute the trial. Must be the same length as points_to_evaluate. (See tune/examples/skopt_example.py)
max_concurrent – Deprecated.
use_early_stopped_trials – Deprecated.
Tune automatically converts search spaces to SkOpt’s format:
config = { "width": tune.uniform(0, 20), "height": tune.uniform(-100, 100) } current_best_params = [ { "width": 10, "height": 0, }, { "width": 15, "height": -20, } ] skopt_search = SkOptSearch( metric="mean_loss", mode="min", points_to_evaluate=current_best_params) tune.run(my_trainable, config=config, search_alg=skopt_search)
If you would like to pass the search space/optimizer manually, the code would look like this:
parameter_names = ["width", "height"] parameter_ranges = [(0,20),(-100,100)] current_best_params = [[10, 0], [15, -20]] skopt_search = SkOptSearch( parameter_names=parameter_names, parameter_ranges=parameter_ranges, metric="mean_loss", mode="min", points_to_evaluate=current_best_params) tune.run(my_trainable, search_alg=skopt_search)
-
save
(checkpoint_path: str)[source]¶ Save state to path for this search algorithm.
- Parameters
checkpoint_path (str) – File where the search algorithm state is saved. This path should be used later when restoring from file.
Example:
search_alg = Searcher(...) analysis = tune.run( cost, num_samples=5, search_alg=search_alg, name=self.experiment_name, local_dir=self.tmpdir) search_alg.save("./my_favorite_path.pkl")
Changed in version 0.8.7: Save is automatically called by tune.run. You can use restore_from_dir to restore from an experiment directory such as ~/ray_results/trainable.
-
restore
(checkpoint_path: str)[source]¶ Restore state for this search algorithm
- Parameters
checkpoint_path (str) – File where the search algorithm state is saved. This path should be the same as the one provided to “save”.
Example:
search_alg.save("./my_favorite_path.pkl") search_alg2 = Searcher(...) search_alg2 = ConcurrencyLimiter(search_alg2, 1) search_alg2.restore(checkpoint_path) tune.run(cost, num_samples=5, search_alg=search_alg2)
ZOOpt (tune.suggest.zoopt.ZOOptSearch)¶
-
class
ray.tune.suggest.zoopt.
ZOOptSearch
(algo: str = 'asracos', budget: Optional[int] = None, dim_dict: Optional[Dict] = None, metric: Optional[str] = None, mode: Optional[str] = None, points_to_evaluate: Optional[List[Dict]] = None, parallel_num: int = 1, **kwargs)[source]¶ A wrapper around ZOOpt to provide trial suggestions.
ZOOptSearch is a library for derivative-free optimization. It is backed by the ZOOpt package. Currently, Asynchronous Sequential RAndomized COordinate Shrinking (ASRacos) is implemented in Tune.
To use ZOOptSearch, install zoopt (>=0.4.1):
pip install -U zoopt
.Tune automatically converts search spaces to ZOOpt”s format:
from ray import tune from ray.tune.suggest.zoopt import ZOOptSearch "config": { "iterations": 10, # evaluation times "width": tune.uniform(-10, 10), "height": tune.uniform(-10, 10) } zoopt_search_config = { "parallel_num": 8, # how many workers to parallel } zoopt_search = ZOOptSearch( algo="Asracos", # only support Asracos currently budget=20, # must match `num_samples` in `tune.run()`. dim_dict=dim_dict, metric="mean_loss", mode="min", **zoopt_search_config ) tune.run(my_objective, config=config, search_alg=zoopt_search, name="zoopt_search", num_samples=20, stop={"timesteps_total": 10})
If you would like to pass the search space manually, the code would look like this:
from ray import tune from ray.tune.suggest.zoopt import ZOOptSearch from zoopt import ValueType dim_dict = { "height": (ValueType.CONTINUOUS, [-10, 10], 1e-2), "width": (ValueType.DISCRETE, [-10, 10], False), "layers": (ValueType.GRID, [4, 8, 16]) } "config": { "iterations": 10, # evaluation times } zoopt_search_config = { "parallel_num": 8, # how many workers to parallel } zoopt_search = ZOOptSearch( algo="Asracos", # only support Asracos currently budget=20, # must match `num_samples` in `tune.run()`. dim_dict=dim_dict, metric="mean_loss", mode="min", **zoopt_search_config ) tune.run(my_objective, config=config, search_alg=zoopt_search, name="zoopt_search", num_samples=20, stop={"timesteps_total": 10})
- Parameters
algo (str) – To specify an algorithm in zoopt you want to use. Only support ASRacos currently.
budget (int) – Number of samples.
dim_dict (dict) – Dimension dictionary. For continuous dimensions: (continuous, search_range, precision); For discrete dimensions: (discrete, search_range, has_order); For grid dimensions: (grid, grid_list). More details can be found in zoopt package.
metric (str) – The training result objective value attribute. If None but a mode was passed, the anonymous metric _metric will be used per default.
mode (str) – One of {min, max}. Determines whether objective is minimizing or maximizing the metric attribute.
points_to_evaluate (list) – Initial parameter suggestions to be run first. This is for when you already have some good parameters you want to run first to help the algorithm make better suggestions for future parameters. Needs to be a list of dicts containing the configurations.
parallel_num (int) – How many workers to parallel. Note that initial phase may start less workers than this number. More details can be found in zoopt package.
-
save
(checkpoint_path: str)[source]¶ Save state to path for this search algorithm.
- Parameters
checkpoint_path (str) – File where the search algorithm state is saved. This path should be used later when restoring from file.
Example:
search_alg = Searcher(...) analysis = tune.run( cost, num_samples=5, search_alg=search_alg, name=self.experiment_name, local_dir=self.tmpdir) search_alg.save("./my_favorite_path.pkl")
Changed in version 0.8.7: Save is automatically called by tune.run. You can use restore_from_dir to restore from an experiment directory such as ~/ray_results/trainable.
-
restore
(checkpoint_path: str)[source]¶ Restore state for this search algorithm
- Parameters
checkpoint_path (str) – File where the search algorithm state is saved. This path should be the same as the one provided to “save”.
Example:
search_alg.save("./my_favorite_path.pkl") search_alg2 = Searcher(...) search_alg2 = ConcurrencyLimiter(search_alg2, 1) search_alg2.restore(checkpoint_path) tune.run(cost, num_samples=5, search_alg=search_alg2)
Repeated Evaluations (tune.suggest.Repeater)¶
Use ray.tune.suggest.Repeater
to average over multiple evaluations of the same
hyperparameter configurations. This is useful in cases where the evaluated
training procedure has high variance (i.e., in reinforcement learning).
By default, Repeater
will take in a repeat
parameter and a search_alg
.
The search_alg
will suggest new configurations to try, and the Repeater
will run repeat
trials of the configuration. It will then average the
search_alg.metric
from the final results of each repeated trial.
Warning
It is recommended to not use Repeater
with a TrialScheduler.
Early termination can negatively affect the average reported metric.
-
class
ray.tune.suggest.
Repeater
(searcher: ray.tune.suggest.suggestion.Searcher, repeat: int = 1, set_index: bool = True)[source]¶ A wrapper algorithm for repeating trials of same parameters.
Set tune.run(num_samples=…) to be a multiple of repeat. For example, set num_samples=15 if you intend to obtain 3 search algorithm suggestions and repeat each suggestion 5 times. Any leftover trials (num_samples mod repeat) will be ignored.
It is recommended that you do not run an early-stopping TrialScheduler simultaneously.
- Parameters
searcher (Searcher) – Searcher object that the Repeater will optimize. Note that the Searcher will only see 1 trial among multiple repeated trials. The result/metric passed to the Searcher upon trial completion will be averaged among all repeats.
repeat (int) – Number of times to generate a trial with a repeated configuration. Defaults to 1.
set_index (bool) – Sets a tune.suggest.repeater.TRIAL_INDEX in Trainable/Function config which corresponds to the index of the repeated trial. This can be used for seeds. Defaults to True.
Example:
from ray.tune.suggest import Repeater search_alg = BayesOptSearch(...) re_search_alg = Repeater(search_alg, repeat=10) # Repeat 2 samples 10 times each. tune.run(trainable, num_samples=20, search_alg=re_search_alg)
ConcurrencyLimiter (tune.suggest.ConcurrencyLimiter)¶
Use ray.tune.suggest.ConcurrencyLimiter
to limit the amount of concurrency when using a search algorithm. This is useful when a given optimization algorithm does not parallelize very well (like a naive Bayesian Optimization).
-
class
ray.tune.suggest.
ConcurrencyLimiter
(searcher: ray.tune.suggest.suggestion.Searcher, max_concurrent: int, batch: bool = False)[source]¶ A wrapper algorithm for limiting the number of concurrent trials.
- Parameters
searcher (Searcher) – Searcher object that the ConcurrencyLimiter will manage.
max_concurrent (int) – Maximum concurrent samples from the underlying searcher.
batch (bool) – Whether to wait for all concurrent samples to finish before updating the underlying searcher.
Example:
from ray.tune.suggest import ConcurrencyLimiter search_alg = HyperOptSearch(metric="accuracy") search_alg = ConcurrencyLimiter(search_alg, max_concurrent=2) tune.run(trainable, search_alg=search_alg)
Custom Search Algorithms (tune.suggest.Searcher)¶
If you are interested in implementing or contributing a new Search Algorithm, provide the following interface:
-
class
ray.tune.suggest.
Searcher
(metric: Optional[str] = None, mode: Optional[str] = None, max_concurrent: Optional[int] = None, use_early_stopped_trials: Optional[bool] = None)[source]¶ Bases:
object
Abstract class for wrapping suggesting algorithms.
Custom algorithms can extend this class easily by overriding the suggest method provide generated parameters for the trials.
Any subclass that implements
__init__
must also call the constructor of this class:super(Subclass, self).__init__(...)
.To track suggestions and their corresponding evaluations, the method suggest will be passed a trial_id, which will be used in subsequent notifications.
Not all implementations support multi objectives.
- Parameters
metric (str or list) – The training result objective value attribute. If list then list of training result objective value attributes
mode (str or list) – If string One of {min, max}. If list then list of max and min, determines whether objective is minimizing or maximizing the metric attribute. Must match type of metric.
class ExampleSearch(Searcher): def __init__(self, metric="mean_loss", mode="min", **kwargs): super(ExampleSearch, self).__init__( metric=metric, mode=mode, **kwargs) self.optimizer = Optimizer() self.configurations = {} def suggest(self, trial_id): configuration = self.optimizer.query() self.configurations[trial_id] = configuration def on_trial_complete(self, trial_id, result, **kwargs): configuration = self.configurations[trial_id] if result and self.metric in result: self.optimizer.update(configuration, result[self.metric]) tune.run(trainable_function, search_alg=ExampleSearch())
-
set_search_properties
(metric: Optional[str], mode: Optional[str], config: Dict) → bool[source]¶ Pass search properties to searcher.
This method acts as an alternative to instantiating search algorithms with their own specific search spaces. Instead they can accept a Tune config through this method. A searcher should return
True
if setting the config was successful, orFalse
if it was unsuccessful, e.g. when the search space has already been set.- Parameters
metric (str) – Metric to optimize
mode (str) – One of [“min”, “max”]. Direction to optimize.
config (dict) – Tune config dict.
-
on_trial_result
(trial_id: str, result: Dict)[source]¶ Optional notification for result during training.
Note that by default, the result dict may include NaNs or may not include the optimization metric. It is up to the subclass implementation to preprocess the result to avoid breaking the optimization process.
- Parameters
trial_id (str) – A unique string ID for the trial.
result (dict) – Dictionary of metrics for current training progress. Note that the result dict may include NaNs or may not include the optimization metric. It is up to the subclass implementation to preprocess the result to avoid breaking the optimization process.
-
on_trial_complete
(trial_id: str, result: Optional[Dict] = None, error: bool = False)[source]¶ Notification for the completion of trial.
Typically, this method is used for notifying the underlying optimizer of the result.
- Parameters
trial_id (str) – A unique string ID for the trial.
result (dict) – Dictionary of metrics for current training progress. Note that the result dict may include NaNs or may not include the optimization metric. It is up to the subclass implementation to preprocess the result to avoid breaking the optimization process. Upon errors, this may also be None.
error (bool) – True if the training process raised an error.
-
suggest
(trial_id: str) → Optional[Dict][source]¶ Queries the algorithm to retrieve the next set of parameters.
- Parameters
trial_id (str) – Trial ID used for subsequent notifications.
- Returns
- Configuration for a trial, if possible.
If FINISHED is returned, Tune will be notified that no more suggestions/configurations will be provided. If None is returned, Tune will skip the querying of the searcher for this step.
- Return type
dict | FINISHED | None
-
save
(checkpoint_path: str)[source]¶ Save state to path for this search algorithm.
- Parameters
checkpoint_path (str) – File where the search algorithm state is saved. This path should be used later when restoring from file.
Example:
search_alg = Searcher(...) analysis = tune.run( cost, num_samples=5, search_alg=search_alg, name=self.experiment_name, local_dir=self.tmpdir) search_alg.save("./my_favorite_path.pkl")
Changed in version 0.8.7: Save is automatically called by tune.run. You can use restore_from_dir to restore from an experiment directory such as ~/ray_results/trainable.
-
restore
(checkpoint_path: str)[source]¶ Restore state for this search algorithm
- Parameters
checkpoint_path (str) – File where the search algorithm state is saved. This path should be the same as the one provided to “save”.
Example:
search_alg.save("./my_favorite_path.pkl") search_alg2 = Searcher(...) search_alg2 = ConcurrencyLimiter(search_alg2, 1) search_alg2.restore(checkpoint_path) tune.run(cost, num_samples=5, search_alg=search_alg2)
-
save_to_dir
(checkpoint_dir: str, session_str: str = 'default')[source]¶ Automatically saves the given searcher to the checkpoint_dir.
This is automatically used by tune.run during a Tune job.
- Parameters
checkpoint_dir (str) – Filepath to experiment dir.
session_str (str) – Unique identifier of the current run session.
-
restore_from_dir
(checkpoint_dir: str)[source]¶ Restores the state of a searcher from a given checkpoint_dir.
Typically, you should use this function to restore from an experiment directory such as ~/ray_results/trainable.
experiment_1 = tune.run( cost, num_samples=5, search_alg=search_alg, verbose=0, name=self.experiment_name, local_dir="~/my_results") search_alg2 = Searcher() search_alg2.restore_from_dir( os.path.join("~/my_results", self.experiment_name)
-
property
metric
¶ The training result objective value attribute.
-
property
mode
¶ Specifies if minimizing or maximizing the metric.
Shim Instantiation (tune.create_searcher)¶
There is also a shim function that constructs the search algorithm based on the provided string. This can be useful if the search algorithm you want to use changes often (e.g., specifying the search algorithm via a CLI option or config file).
-
tune.
create_searcher
(**kwargs)¶ Instantiate a search algorithm based on the given string.
This is useful for swapping between different search algorithms.
- Parameters
search_alg (str) – The search algorithm to use.
metric (str) – The training result objective value attribute. Stopping procedures will use this attribute.
mode (str) – One of {min, max}. Determines whether objective is minimizing or maximizing the metric attribute.
**kwargs – Additional parameters. These keyword arguments will be passed to the initialization function of the chosen class.
- Returns
The search algorithm.
- Return type
Example
>>> search_alg = tune.create_searcher('ax')