# Key Concepts¶

Let’s quickly walk through the key concepts you need to know to use Tune. In this guide, we’ll be covering the following:

## Trainables¶

Tune will optimize your training process using the Trainable API. To start, let’s try to maximize this objective function:

def objective(x, a, b):
return a * (x ** 0.5) + b


Here’s an example of specifying the objective function using the function-based Trainable API:

def trainable(config):
# config (dict): A dict of hyperparameters.

for x in range(20):
score = objective(x, config["a"], config["b"])

tune.report(score=score)  # This sends the score to Tune.


Now, there’s two Trainable APIs - one being the function-based API that we demonstrated above.

The other is a class-based API. Here’s an example of specifying the objective function using the class-based API:

from ray import tune

class Trainable(tune.Trainable):
def setup(self, config):
# config (dict): A dict of hyperparameters
self.x = 0
self.a = config["a"]
self.b = config["b"]

def step(self):  # This is called iteratively.
score = objective(self.x, self.a, self.b)
self.x += 1
return {"score": score}


Tip

Do not use tune.report within a Trainable class.

See the documentation: Training (tune.Trainable, tune.report) and examples.

## tune.run¶

Use tune.run execute hyperparameter tuning using the core Ray APIs. This function manages your experiment and provides many features such as logging, checkpointing, and early stopping.

# Pass in a Trainable class or function to tune.run.
tune.run(trainable)


This function will report status on the command line until all trials stop (each trial is one instance of a Trainable):

== Status ==
Memory usage on this node: 11.4/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 1/12 CPUs, 0/0 GPUs, 0.0/3.17 GiB heap, 0.0/1.07 GiB objects
Result logdir: /Users/foo/ray_results/myexp
Number of trials: 1 (1 RUNNING)
+----------------------+----------+---------------------+-----------+--------+--------+----------------+-------+
| Trial name           | status   | loc                 |         a |      b |  score | total time (s) |  iter |
|----------------------+----------+---------------------+-----------+--------+--------+----------------+-------|
| MyTrainable_a826033a | RUNNING  | 10.234.98.164:31115 | 0.303706  | 0.0761 | 0.1289 |        7.54952 |    15 |
+----------------------+----------+---------------------+-----------+--------+--------+----------------+-------+


You can also easily run 10 trials. Tune automatically determines how many trials will run in parallel.

tune.run(trainable, num_samples=10)


Finally, you can randomly sample or grid search hyperparameters via Tune’s search space API:

space = {"x": tune.uniform(0, 1)}
tune.run(my_trainable, config=space, num_samples=10)


See more documentation: tune.run.

## Search Algorithms¶

To optimize the hyperparameters of your training process, you will want to use a Search Algorithm which will help suggest better hyperparameters.

# Be sure to first run pip install hyperopt

import hyperopt as hp
from ray.tune.suggest.hyperopt import HyperOptSearch

# Create a HyperOpt search space
space = {
"a": hp.uniform("a", 0, 1),
"b": hp.uniform("b", 0, 20)

# Note: Arbitrary HyperOpt search spaces should be supported!
# "foo": hp.lognormal("foo", 0, 1))
}

# Specify the search space and maximize score
hyperopt = HyperOptSearch(space, metric="score", mode="max")

# Execute 20 trials using HyperOpt and stop after 20 iterations
tune.run(
trainable,
search_alg=hyperopt,
num_samples=20,
stop={"training_iteration": 20}
)


Tune has SearchAlgorithms that integrate with many popular optimization libraries, such as Nevergrad and Hyperopt.

See the documentation: Search Algorithms (tune.suggest).

## Trial Schedulers¶

In addition, you can make your training process more efficient by using a Trial Scheduler.

Trial Schedulers can stop/pause/tweak the hyperparameters of running trials, making your hyperparameter tuning process much faster.

from ray.tune.schedulers import HyperBandScheduler

# Create HyperBand scheduler and maximize score
hyperband = HyperBandScheduler(metric="score", mode="max")

# Execute 20 trials using HyperBand using a search space
configs = {"a": tune.uniform(0, 1), "b": tune.uniform(0, 1)}

tune.run(
MyTrainableClass,
config=configs,
num_samples=20,
scheduler=hyperband
)


Population-based Training and HyperBand are examples of popular optimization algorithms implemented as Trial Schedulers.

Unlike Search Algorithms, Trial Scheduler do not select which hyperparameter configurations to evaluate. However, you can use them together.

See the documentation: Summary.

## Analysis¶

tune.run returns an Analysis object which has methods you can use for analyzing your training.

analysis = tune.run(trainable, search_alg=algo, stop={"training_iteration": 20})

best_hyperparameters = analysis.get_best_config()


This object can also retrieve all training runs as dataframes, allowing you to do ad-hoc data analysis over your results.

# Get a dataframe for the max score seen for each trial
df = analysis.dataframe(metric="score", mode="max")


## What’s Next?¶

Now that you have a working understanding of Tune, check out:

• User Guide & Configuring Tune: A comprehensive overview of Tune’s features.

• Tutorials: Tutorials for using Tune with your preferred machine learning library.

• Examples: End-to-end examples and templates for using Tune with your preferred machine learning library.

• A Basic Tune Tutorial: A simple tutorial that walks you through the process of setting up a Tune experiment.

### Further Questions or Issues?¶

Reach out to us if you have any questions or issues or feedback through the following channels:

1. StackOverflow: For questions about how to use Ray.

2. GitHub Issues: For bug reports and feature requests.