Tune: Scalable Hyperparameter Tuning¶

Tip

We’d love to hear your feedback on using Tune - get in touch!

Tune is a Python library for experiment execution and hyperparameter tuning at any scale. Core features:

Launch a multi-node distributed hyperparameter sweep in less than 10 lines of code.
Supports any machine learning framework, including PyTorch, XGBoost, MXNet, and Keras.
Automatically manages checkpoints and logging to TensorBoard.
Choose among state of the art algorithms such as Population Based Training (PBT), BayesOptSearch, HyperBand/ASHA.
Move your models from training to serving on the same infrastructure with Ray Serve.

Want to get started? Head over to the Key Concepts page.

Tip

Join the Ray community slack to discuss Ray Tune (and other Ray libraries)!

Quick Start¶

To run this example, install the following: pip install 'ray[tune]'.

This example runs a parallel grid search to optimize an example objective function.

from ray import tune


def objective(step, alpha, beta):
    return (0.1 + alpha * step / 100)**(-1) + beta * 0.1


def training_function(config):
    # Hyperparameters
    alpha, beta = config["alpha"], config["beta"]
    for step in range(10):
        # Iterative training function - can be any arbitrary training procedure.
        intermediate_score = objective(step, alpha, beta)
        # Feed the score back back to Tune.
        tune.report(mean_loss=intermediate_score)


analysis = tune.run(
    training_function,
    config={
        "alpha": tune.grid_search([0.001, 0.01, 0.1]),
        "beta": tune.choice([1, 2, 3])
    })

print("Best config: ", analysis.get_best_config(
    metric="mean_loss", mode="min"))

# Get a dataframe for analyzing trial results.
df = analysis.results_df

If TensorBoard is installed, automatically visualize all trial results:

tensorboard --logdir ~/ray_results

If using TF2 and TensorBoard, Tune will also automatically generate TensorBoard HParams output:

Why choose Tune?¶

There are many other hyperparameter optimization libraries out there. If you’re new to Tune, you’re probably wondering, “what makes Tune different?”

Cutting-edge optimization algorithms¶

As a user, you’re probably looking into hyperparameter optimization because you want to quickly increase your model performance.

Tune enables you to leverage a variety of these cutting edge optimization algorithms, reducing the cost of tuning by aggressively terminating bad hyperparameter evaluations, intelligently choosing better parameters to evaluate, or even changing the hyperparameters during training to optimize hyperparameter schedules.

First-class Developer Productivity¶

A key problem with machine learning frameworks is the need to restructure all of your code to fit the framework.

With Tune, you can optimize your model just by adding a few code snippets.

Further, Tune actually removes boilerplate from your code training workflow, automatically managing checkpoints and logging results to tools such as MLFlow and TensorBoard.

Multi-GPU & distributed training out of the box¶

Hyperparameter tuning is known to be highly time-consuming, so it is often necessary to parallelize this process. Most other tuning frameworks require you to implement your own multi-process framework or build your own distributed system to speed up hyperparameter tuning.

However, Tune allows you to transparently parallelize across multiple GPUs and multiple nodes. Tune even has seamless fault tolerance and cloud support, allowing you to scale up your hyperparameter search by 100x while reducing costs by up to 10x by using cheap preemptible instances.

What if I’m already doing hyperparameter tuning?¶

You might be already using an existing hyperparameter tuning tool such as HyperOpt or Bayesian Optimization.

In this situation, Tune actually allows you to power up your existing workflow. Tune’s Search Algorithms integrate with a variety of popular hyperparameter tuning libraries (such as Nevergrad or HyperOpt) and allow you to seamlessly scale up your optimization process – without sacrificing performance.

Reference Materials¶

Here are some reference materials for Tune:

Below are some blog posts and talks about Tune:

[blog] Tune: a Python library for fast hyperparameter tuning at any scale
[blog] Cutting edge hyperparameter tuning with Ray Tune
[blog] Simple hyperparameter and architecture search in tensorflow with Ray Tune
[slides] Talk given at RISECamp 2019
[video] Talk given at RISECamp 2018
[video] A Guide to Modern Hyperparameter Optimization (PyData LA 2019) (slides)

Citing Tune¶

If Tune helps you in your academic research, you are encouraged to cite our paper. Here is an example bibtex:

@article{liaw2018tune,
    title={Tune: A Research Platform for Distributed Model Selection and Training},
    author={Liaw, Richard and Liang, Eric and Nishihara, Robert
            and Moritz, Philipp and Gonzalez, Joseph E and Stoica, Ion},
    journal={arXiv preprint arXiv:1807.05118},
    year={2018}
}

Ray v2.0.0.dev0