Key concepts of Ray Tune

This page walks through the key concepts you need to know to use Ray Tune. For practical tutorials, see the user guides. Tune has six components that you need to understand.

First, you define the hyperparameters you want to tune in a search space and pass them into a trainable that specifies the objective you want to tune. Then you select a search algorithm to optimize your parameters, and optionally use a scheduler to stop searches early and speed up your experiments. You pass your trainable, search algorithm, and scheduler, together with other configuration, into the Tuner, which runs your experiment and creates trials. The Tuner returns a ResultGrid that you use to inspect your experiment results. The following figure shows an overview of these components. The next sections cover each one in detail.

Ray Tune trainables

A trainable is an object that you can pass into a Tune run. You can define a trainable with either the function API or the class API. Both are valid, but the function API is the generally recommended choice, and the rest of this guide uses it.

Consider an example that optimizes the simple objective function a * (x ** 2) + b, where a and b are the hyperparameters you tune to minimize the objective. Because the objective also has a variable x, you need to test different values of x. Given concrete values for a, b, and x, you can evaluate the objective function and get a score to minimize.

Function API

With the function API, you create a function that takes a dictionary of hyperparameters. The following example names the function trainable. It computes a score in a “training loop” and reports the score back to Tune:

from ray import tune


def objective(x, a, b):  # Define an objective function.
    return a * (x**2) + b


def trainable(config):  # Pass a "config" dictionary into your trainable.

    for x in range(20):  # "Train" for 20 iterations and compute intermediate scores.
        score = objective(x, config["a"], config["b"])

        tune.report({"score": score})  # Send the score to Tune.

This example uses tune.report(...) to report the intermediate score in the training loop, which can be useful in many machine learning tasks. To report only the final score outside of this loop, return the score at the end of the trainable function with return {"score": score}. You can also use yield {"score": score} instead of tune.report().

Class API

The following example specifies the objective function with the class API:

from ray import tune


def objective(x, a, b):
    return a * (x**2) + b


class Trainable(tune.Trainable):
    def setup(self, config):
        # config (dict): A dict of hyperparameters
        self.x = 0
        self.a = config["a"]
        self.b = config["b"]

    def step(self):  # This is called iteratively.
        score = objective(self.x, self.a, self.b)
        self.x += 1
        return {"score": score}

Note

You can’t use tune.report within a Trainable class.

For more details, see the trainable documentation and the other Tune examples. The next section looks at the config dictionary that you pass into your trainables.

Tune search spaces

To optimize your hyperparameters, you have to define a search space. A search space defines valid values for your hyperparameters and can specify how these values are sampled, such as from a uniform distribution or a normal distribution.

Tune offers various functions to define search spaces and sampling methods. For the full explanation of each function, see the Tune search space API.

The following example covers all the search space functions:

config = {
    "uniform": tune.uniform(-5, -1),  # Uniform float between -5 and -1
    "quniform": tune.quniform(3.2, 5.4, 0.2),  # Round to multiples of 0.2
    "loguniform": tune.loguniform(1e-4, 1e-1),  # Uniform float in log space
    "qloguniform": tune.qloguniform(1e-4, 1e-1, 5e-5),  # Round to multiples of 0.00005
    "randn": tune.randn(10, 2),  # Normal distribution with mean 10 and sd 2
    "qrandn": tune.qrandn(10, 2, 0.2),  # Round to multiples of 0.2
    "randint": tune.randint(-9, 15),  # Random integer between -9 and 15
    "qrandint": tune.qrandint(-21, 12, 3),  # Round to multiples of 3 (includes 12)
    "lograndint": tune.lograndint(1, 10),  # Random integer in log space
    "qlograndint": tune.qlograndint(1, 10, 2),  # Round to multiples of 2
    "choice": tune.choice(["a", "b", "c"]),  # Choose one of these options uniformly
    "func": tune.sample_from(
        lambda config: config["uniform"] * 0.01
    ),  # Depends on other value
    "grid": tune.grid_search([32, 64, 128]),  # Search over all these values
}

Tune trials

Use Tuner.fit to run and manage hyperparameter tuning and generate your trials. At a minimum, your Tuner call takes a trainable as the first argument and a param_space dictionary that defines the search space.

Tuner.fit() also provides many features, such as logging, checkpointing, and early stopping. The following example shows a basic Tune run for the example objective a * (x ** 2) + b, with a simple search space for a and b:

# Pass in a Trainable class or function, along with a search space "config".
tuner = tune.Tuner(trainable, param_space={"a": 2, "b": 4})
tuner.fit()

Tuner.fit generates a couple of hyperparameter configurations from its arguments and wraps them into Trial objects.

Trials contain a lot of information. For example, you can get the hyperparameter configuration from trial.config, the trial ID from trial.trial_id, the trial’s resource specification from resources_per_trial or trial.placement_group_factory, and many other values.

By default, Tuner.fit runs until all trials stop or error. The following example shows the output of a trial run:

== Status ==
Memory usage on this node: 11.4/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 1/12 CPUs, 0/0 GPUs, 0.0/3.17 GiB heap, 0.0/1.07 GiB objects
Result logdir: /Users/foo/ray_results/myexp
Number of trials: 1 (1 RUNNING)
+----------------------+----------+---------------------+-----------+--------+--------+----------------+-------+
| Trial name           | status   | loc                 |         a |      b |  score | total time (s) |  iter |
|----------------------+----------+---------------------+-----------+--------+--------+----------------+-------|
| Trainable_a826033a | RUNNING  | 10.234.98.164:31115 | 0.303706  | 0.0761 | 0.1289 |        7.54952 |    15 |
+----------------------+----------+---------------------+-----------+--------+--------+----------------+-------+

To run 10 trials, specify the number of samples with num_samples. Tune automatically determines how many trials run in parallel. Instead of the number of samples, you can also specify a time budget in seconds with time_budget_s if you set num_samples=-1.

tuner = tune.Tuner(
    trainable, param_space={"a": 2, "b": 4}, tune_config=tune.TuneConfig(num_samples=10)
)
tuner.fit()

Finally, you can optimize your hyperparameters with more complex search spaces, such as random samples or grid search, through Tune’s search space API. The following example samples a and b uniformly between [0, 1]:

space = {"a": tune.uniform(0, 1), "b": tune.uniform(0, 1)}
tuner = tune.Tuner(
    trainable, param_space=space, tune_config=tune.TuneConfig(num_samples=10)
)
tuner.fit()

To learn more about the ways to configure your Tune runs, see the Tuner API reference.

Tune search algorithms

To optimize the hyperparameters of your training process, use a search algorithm, which suggests hyperparameter configurations. If you don’t specify a search algorithm, Tune uses random search by default, which can give you a good starting point for your hyperparameter optimization.

For example, to use Tune with simple Bayesian optimization through the bayesian-optimization package, first run pip install bayesian-optimization. Then define an algo with BayesOptSearch, pass it as the search_alg argument to tune.TuneConfig, and pass the tune.TuneConfig to the Tuner:

from ray.tune.search.bayesopt import BayesOptSearch

# Define the search space
search_space = {"a": tune.uniform(0, 1), "b": tune.uniform(0, 20)}

algo = BayesOptSearch(random_search_steps=4)

tuner = tune.Tuner(
    trainable,
    tune_config=tune.TuneConfig(
        metric="score",
        mode="min",
        search_alg=algo,
    ),
    run_config=tune.RunConfig(stop={"training_iteration": 20}),
    param_space=search_space,
)
tuner.fit()

Tune has search algorithms that integrate with many popular optimization libraries, such as Hyperopt and Optuna. Tune automatically converts the search space you provide into the search spaces that the search algorithms and underlying libraries expect. For more details, see the search algorithm API documentation.

The following table gives an overview of all available search algorithms in Tune:

Search algorithm	Summary	Website	Code example
Random search and grid search	Random search or grid search		tune_basic_example
AxSearch	Bayesian or bandit optimization	Ax	AX Example
HyperOptSearch	Tree-structured Parzen Estimator	Hyperopt	Running Tune experiments with HyperOpt
BayesOptSearch	Bayesian optimization	BayesianOptimization	BayesOpt Example
TuneBOHB	Bayesian optimization and Hyperband	BOHB	BOHB Example
NevergradSearch	Gradient-free optimization	Nevergrad	Nevergrad Example
OptunaSearch	Optuna search algorithms	Optuna	Running Tune experiments with Optuna

Note

Unlike Tune trial schedulers, Tune search algorithms can’t affect or stop training processes. However, you can use the two together to stop the evaluation of bad trials early.

To implement your own search algorithm, see custom search algorithms.

Tune also provides the following utilities to use with search algorithms:

• Repeated Evaluations (tune.search.Repeater): Runs each sampled hyperparameter with multiple random seeds.

• ConcurrencyLimiter (tune.search.ConcurrencyLimiter): Limits the number of concurrent trials during optimization.

• Shim Instantiation (tune.create_searcher): Creates the search algorithm object from a string.

The preceding Bayesian optimization example tells Tune to stop after 20 training iterations. Stopping trials with explicit rules is useful, but in many cases you can do even better with schedulers.

Tune schedulers

To make your training process more efficient, you can use a trial scheduler. For example, the trainable example on this page minimizes a function in a training loop and calls tune.report() to report incremental results for a hyperparameter configuration that a search algorithm selected. Based on these reported results, a scheduler can decide whether to stop the trial early. If you don’t specify a scheduler, Tune uses a first-in, first-out (FIFO) scheduler by default. The FIFO scheduler passes through the trials that your search algorithm selects in the order the search algorithm picks them, and doesn’t perform any early stopping.

Schedulers can stop, pause, or tweak the hyperparameters of running trials, which can make your hyperparameter tuning process much faster. Unlike search algorithms, trial schedulers don’t select which hyperparameter configurations to evaluate.

All schedulers take a metric, which is the value your trainable reports. The metric is then maximized or minimized according to the mode you provide. To use a scheduler, pass a scheduler argument to tune.TuneConfig, and pass the tune.TuneConfig to the Tuner. The following example uses the Hyperband scheduler, HyperBandScheduler:

from ray.tune.schedulers import HyperBandScheduler

# Create HyperBand scheduler and minimize the score
hyperband = HyperBandScheduler(metric="score", mode="max")

config = {"a": tune.uniform(0, 1), "b": tune.uniform(0, 1)}

tuner = tune.Tuner(
    trainable,
    tune_config=tune.TuneConfig(
        num_samples=20,
        scheduler=hyperband,
    ),
    param_space=config,
)
tuner.fit()

Tune includes distributed implementations of early stopping algorithms such as the median stopping rule, Hyperband, and ASHA. Tune also includes a distributed implementation of Population Based Training (PBT) and Population Based Bandits (PB2).

Tip

The easiest scheduler to start with is ASHAScheduler, which aggressively stops low-performing trials.

Schedulers can dynamically change trial resource requirements during tuning. ResourceChangingScheduler implements this behavior and can wrap around any other scheduler.

When you use schedulers, you might run into compatibility issues, as the following compatibility matrix shows. You can’t use certain schedulers with search algorithms, and certain schedulers require that you implement checkpointing.

Scheduler compatibility matrix

Scheduler	Needs checkpointing?	Compatible with search algorithms?	Example
ASHA	No	Yes	ASHA example
Median stopping rule	No	Yes	Median stopping rule reference
Hyperband	Yes	Yes	Hyperband example
BOHB	Yes	Only TuneBOHB	BOHB example
Population Based Training	Yes	No	PBT example
Population Based Bandits	Yes	No	Basic example, PPO example

To learn more about trial schedulers, see the scheduler API documentation.

Tune ResultGrid

Tuner.fit() returns a ResultGrid object with methods that you can use to analyze your training. The following example shows how to access various metrics from a ResultGrid object, such as the best available trial or the best hyperparameter configuration for that trial:

tuner = tune.Tuner(
    trainable,
    tune_config=tune.TuneConfig(
        metric="score",
        mode="min",
        search_alg=BayesOptSearch(random_search_steps=4),
    ),
    run_config=tune.RunConfig(
        stop={"training_iteration": 20},
    ),
    param_space=config,
)
results = tuner.fit()

best_result = results.get_best_result()  # Get best result object
best_config = best_result.config  # Get best trial's hyperparameters
best_logdir = best_result.path  # Get best trial's result directory
best_checkpoint = best_result.checkpoint  # Get best trial's best checkpoint
best_metrics = best_result.metrics  # Get best trial's last results
best_result_df = best_result.metrics_dataframe  # Get best result as pandas dataframe

This object can also retrieve all training runs as DataFrames, so you can do ad-hoc data analysis over your results.

# Get a dataframe with the last results for each trial
df_results = results.get_dataframe()

# Get a dataframe of results for a specific score or mode
df = results.get_dataframe(filter_metric="score", filter_mode="max")

See the result analysis user guide for more usage examples.

What’s next?

Now that you have a working understanding of Tune, see the following resources:

• User guides: Tutorials for using Tune with your preferred machine learning library.

• Ray Tune Examples: End-to-end examples and templates for using Tune with your preferred machine learning library.

• Get started with Ray Tune: A tutorial that walks you through setting up a Tune experiment.

Further questions or issues?

You can post questions or issues or feedback through the following channels:

1. Discussion Board: For questions about Ray usage or feature requests.

2. GitHub Issues: For bug reports.

3. Ray Slack: For getting in touch with Ray maintainers.

4. StackOverflow: Use the [ray] tag for questions about Ray.