Hyperparameter Tuning with Ray Tune (Deprecated API)#

Important

This user guide covers the deprecated Train + Tune integration. See Hyperparameter Tuning with Ray Tune for the new API user guide.

Please see here for information about the deprecation and migration.

Hyperparameter tuning with Ray Tune is natively supported with Ray Train.

../../_images/train-tuner.svg

The Tuner will take in a Trainer and execute multiple training runs, each with different hyperparameter configurations.#

Key Concepts#

There are a number of key concepts when doing hyperparameter optimization with a Tuner:

  • A set of hyperparameters you want to tune in a search space.

  • A search algorithm to effectively optimize your parameters and optionally use a scheduler to stop searches early and speed up your experiments.

  • The search space, search algorithm, scheduler, and Trainer are passed to a Tuner, which runs the hyperparameter tuning workload by evaluating multiple hyperparameters in parallel.

  • Each individual hyperparameter evaluation run is called a trial.

  • The Tuner returns its results as a ResultGrid.

Note

Tuners can also be used to launch hyperparameter tuning without using Ray Train. See the Ray Tune documentation for more guides and examples.

Basic usage#

You can take an existing Trainer and simply pass it into a Tuner.

import ray
import ray.tune
import ray.train
from ray.tune import Tuner
from ray.train.xgboost import XGBoostTrainer

dataset = ray.data.read_csv("s3://anonymous@air-example-data/breast_cancer.csv")

trainer = XGBoostTrainer(
    label_column="target",
    params={
        "objective": "binary:logistic",
        "eval_metric": ["logloss", "error"],
        "max_depth": 4,
    },
    datasets={"train": dataset},
    scaling_config=ray.train.ScalingConfig(num_workers=2),
)

# Create Tuner
tuner = Tuner(
    trainer,
    # Add some parameters to tune
    param_space={"params": {"max_depth": ray.tune.choice([4, 5, 6])}},
    # Specify tuning behavior
    tune_config=ray.tune.TuneConfig(metric="train-logloss", mode="min", num_samples=2),
)
# Run tuning job
tuner.fit()

How to configure a Tuner?#

There are two main configuration objects that can be passed into a Tuner: the TuneConfig and the ray.tune.RunConfig.

The TuneConfig contains tuning specific settings, including:

  • the tuning algorithm to use

  • the metric and mode to rank results

  • the amount of parallelism to use

Here are some common configurations for TuneConfig:

from ray.tune import TuneConfig
from ray.tune.search.bayesopt import BayesOptSearch

tune_config = TuneConfig(
    metric="loss",
    mode="min",
    max_concurrent_trials=10,
    num_samples=100,
    search_alg=BayesOptSearch(),
)

See the TuneConfig API reference for more details.

The ray.tune.RunConfig contains configurations that are more generic than tuning specific settings. This includes:

  • failure/retry configurations

  • verbosity levels

  • the name of the experiment

  • the logging directory

  • checkpoint configurations

  • custom callbacks

  • integration with cloud storage

Below we showcase some common configurations of ray.tune.RunConfig.

import ray.tune

run_config = ray.tune.RunConfig(
    name="MyExperiment",
    storage_path="s3://...",
    checkpoint_config=ray.tune.CheckpointConfig(checkpoint_frequency=2),
)

Search Space configuration#

A Tuner takes in a param_space argument where you can define the search space from which hyperparameter configurations will be sampled.

Depending on the model and dataset, you may want to tune:

  • The training batch size

  • The learning rate for deep learning training (e.g., image classification)

  • The maximum depth for tree-based models (e.g., XGBoost)

You can use a Tuner to tune most arguments and configurations for Ray Train, including but not limited to:

Read more about Tune search spaces here.

Train - Tune gotchas#

There are a couple gotchas about parameter specification when using Tuners with Trainers:

  • By default, configuration dictionaries and config objects will be deep-merged.

  • Parameters that are duplicated in the Trainer and Tuner will be overwritten by the Tuner param_space.

  • Exception: all arguments of the ray.tune.RunConfig and ray.tune.TuneConfig are inherently un-tunable.

See Getting Data in and out of Tune for an example.

Advanced Tuning#

Tuners also offer the ability to tune over different data preprocessing steps and different training/validation datasets, as shown in the following snippet.

import ray.data
import ray.tune
from ray.data.preprocessors import StandardScaler


def get_dataset():
    ds1 = ray.data.read_csv("s3://anonymous@air-example-data/breast_cancer.csv")
    prep_v1 = StandardScaler(["worst radius", "worst area"])
    ds1 = prep_v1.fit_transform(ds1)
    return ds1


def get_another_dataset():
    ds2 = ray.data.read_csv(
        "s3://anonymous@air-example-data/breast_cancer_with_categorical.csv"
    )
    prep_v2 = StandardScaler(["worst concavity", "worst smoothness"])
    ds2 = prep_v2.fit_transform(ds2)
    return ds2


dataset_1 = get_dataset()
dataset_2 = get_another_dataset()

tuner = ray.tune.Tuner(
    trainer,
    param_space={
        "datasets": {
            "train": ray.tune.grid_search([dataset_1, dataset_2]),
        }
        # Your other parameters go here
    },
)