{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "58fc50bc",
"metadata": {},
"source": [
"# Running Tune experiments with HyperOpt\n",
"\n",
"\n",
"
\n",
"\n",
"
\n",
"\n",
"In this tutorial we introduce HyperOpt, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with HyperOpt and, as a result, allow you to seamlessly scale up a Hyperopt optimization process - without sacrificing performance.\n",
"\n",
"HyperOpt provides gradient/derivative-free optimization able to handle noise over the objective landscape, including evolutionary, bandit, and Bayesian optimization algorithms. HyperOpt internally supports search spaces which are continuous, discrete or a mixture of thereof. It also provides a library of functions on which to test the optimization algorithms and compare with other benchmarks.\n",
"\n",
"In this example we minimize a simple objective to briefly demonstrate the usage of HyperOpt with Ray Tune via `HyperOptSearch`. It's useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume `hyperopt==0.2.5` library is installed. To learn more, please refer to [HyperOpt website](http://hyperopt.github.io/hyperopt).\n",
"\n",
"We include a important example on conditional search spaces (stringing together relationships among hyperparameters)."
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "e4586d28",
"metadata": {},
"source": [
"Background information:\n",
"- [HyperOpt website](http://hyperopt.github.io/hyperopt)\n",
"\n",
"Necessary requirements:\n",
"- `pip install \"ray[tune]\" hyperopt==0.2.5`"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "6567f2dc",
"metadata": {
"tags": [
"hide-cell"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: hyperopt==0.2.5 in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (0.2.5)\n",
"Requirement already satisfied: numpy in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (from hyperopt==0.2.5) (2.2.3)\n",
"Requirement already satisfied: scipy in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (from hyperopt==0.2.5) (1.15.2)\n",
"Requirement already satisfied: six in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (from hyperopt==0.2.5) (1.17.0)\n",
"Requirement already satisfied: networkx>=2.2 in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (from hyperopt==0.2.5) (3.4.2)\n",
"Requirement already satisfied: future in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (from hyperopt==0.2.5) (1.0.0)\n",
"Requirement already satisfied: tqdm in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (from hyperopt==0.2.5) (4.67.1)\n",
"Requirement already satisfied: cloudpickle in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (from hyperopt==0.2.5) (3.1.1)\n"
]
}
],
"source": [
"# install in a hidden cell\n",
"# !pip install \"ray[tune]\"\n",
"!pip install hyperopt==0.2.5"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "b8e9e0cd",
"metadata": {},
"source": [
"Click below to see all the imports we need for this example."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "6592315e",
"metadata": {
"tags": [
"hide-input",
"hide-output"
]
},
"outputs": [],
"source": [
"import time\n",
"\n",
"import ray\n",
"from ray import tune\n",
"from ray.tune.search import ConcurrencyLimiter\n",
"from ray.tune.search.hyperopt import HyperOptSearch\n",
"from hyperopt import hp"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "d4b6d1d5",
"metadata": {},
"source": [
"Let's start by defining a simple evaluation function.\n",
"We artificially sleep for a bit (`0.1` seconds) to simulate a long-running ML experiment.\n",
"This setup assumes that we're running multiple `step`s of an experiment and try to tune two hyperparameters,\n",
"namely `width` and `height`."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "12d4efc8",
"metadata": {},
"outputs": [],
"source": [
"def evaluate(step, width, height):\n",
" time.sleep(0.1)\n",
" return (0.1 + width * step / 100) ** (-1) + height * 0.1"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "4f4f5aa2",
"metadata": {},
"source": [
"Next, our ``objective`` function takes a Tune ``config``, evaluates the `score` of your experiment in a training loop,\n",
"and uses `tune.report` to report the `score` back to Tune."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "c9818009",
"metadata": {},
"outputs": [],
"source": [
"def objective(config):\n",
" for step in range(config[\"steps\"]):\n",
" score = evaluate(step, config[\"width\"], config[\"height\"])\n",
" tune.report({\"iterations\": step, \"mean_loss\": score})"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "33eddcb9",
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "59f8090dd37c473cb24b97b6b28d109b",
"version_major": 2,
"version_minor": 0
},
"text/html": [
"
\n"
],
"text/plain": [
"RayContext(dashboard_url='', python_version='3.11.11', ray_version='2.42.1', ray_commit='c2e38f7b75be223c0c033986472daada8622d64f')"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ray.init(configure_logging=False)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "5be35d5e",
"metadata": {},
"source": [
"While defining the search algorithm, we may choose to provide an initial set of hyperparameters that we believe are especially promising or informative, and\n",
"pass this information as a helpful starting point for the `HyperOptSearch` object.\n",
"\n",
"We also set the maximum concurrent trials to `4` with a `ConcurrencyLimiter`."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "d4615bed",
"metadata": {
"lines_to_next_cell": 0
},
"outputs": [],
"source": [
"initial_params = [\n",
" {\"width\": 1, \"height\": 2, \"activation\": \"relu\"},\n",
" {\"width\": 4, \"height\": 2, \"activation\": \"tanh\"},\n",
"]\n",
"algo = HyperOptSearch(points_to_evaluate=initial_params)\n",
"algo = ConcurrencyLimiter(algo, max_concurrent=4)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "2a51e7c1",
"metadata": {},
"source": [
"The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to `1000` samples.\n",
"(you can decrease this if it takes too long on your machine)."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "2dbb2be0",
"metadata": {},
"outputs": [],
"source": [
"num_samples = 1000"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "950558ed",
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [],
"source": [
"# If 1000 samples take too long, you can reduce this number.\n",
"# We override this number here for our smoke tests.\n",
"num_samples = 10"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "6e3629cb",
"metadata": {},
"source": [
"Next we define a search space. The critical assumption is that the optimal hyperparameters live within this space. Yet, if the space is very large, then those hyperparameters may be difficult to find in a short amount of time."
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "65189946",
"metadata": {},
"outputs": [],
"source": [
"search_config = {\n",
" \"steps\": 100,\n",
" \"width\": tune.uniform(0, 20),\n",
" \"height\": tune.uniform(-100, 100),\n",
" \"activation\": tune.choice([\"relu\", \"tanh\"])\n",
"}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "1b94c93b",
"metadata": {},
"source": [
"Finally, we run the experiment to `\"min\"`imize the \"mean_loss\" of the `objective` by searching `search_config` via `algo`, `num_samples` times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute `tuner.fit()`."
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "9a99a3a7",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
\n",
"
\n",
"
Tune Status
\n",
"
\n",
"\n",
"Current time: | 2025-02-18 13:14:59 |
\n",
"Running for: | 00:00:36.03 |
\n",
"Memory: | 22.1/36.0 GiB |
\n",
"\n",
"
\n",
"
\n",
"
\n",
"
\n",
"
System Info
\n",
" Using FIFO scheduling algorithm.
Logical resource usage: 1.0/12 CPUs, 0/0 GPUs\n",
" \n",
" \n",
"
\n",
"
\n",
"
\n",
"
Trial Status
\n",
"
\n",
"\n",
"Trial name | status | loc | activation | height | steps | width | loss | iter | total time (s) | iterations |
\n",
"\n",
"\n",
"objective_5b05c00a | TERMINATED | 127.0.0.1:50205 | relu | 2 | 100 | 1 | 1.11743 | 100 | 10.335 | 99 |
\n",
"objective_b813f49d | TERMINATED | 127.0.0.1:50207 | tanh | 2 | 100 | 4 | 0.446305 | 100 | 10.3299 | 99 |
\n",
"objective_6dadd2bd | TERMINATED | 127.0.0.1:50212 | tanh | -40.9318 | 100 | 6.20615 | -3.93303 | 100 | 10.3213 | 99 |
\n",
"objective_9faffc0f | TERMINATED | 127.0.0.1:50217 | tanh | 91.9688 | 100 | 9.25147 | 9.30488 | 100 | 10.353 | 99 |
\n",
"objective_7834e74c | TERMINATED | 127.0.0.1:50266 | tanh | -17.9521 | 100 | 11.436 | -1.70766 | 100 | 10.3753 | 99 |
\n",
"objective_741253c7 | TERMINATED | 127.0.0.1:50271 | tanh | 58.1279 | 100 | 0.737879 | 7.01689 | 100 | 10.3565 | 99 |
\n",
"objective_39682bcf | TERMINATED | 127.0.0.1:50272 | tanh | -31.2589 | 100 | 4.89265 | -2.92361 | 100 | 10.3225 | 99 |
\n",
"objective_bfc7e150 | TERMINATED | 127.0.0.1:50274 | tanh | -14.7877 | 100 | 7.36477 | -1.34347 | 100 | 10.3744 | 99 |
\n",
"objective_e1f1a193 | TERMINATED | 127.0.0.1:50314 | tanh | 50.9579 | 100 | 17.4499 | 5.15334 | 100 | 10.3675 | 99 |
\n",
"objective_1192dd4e | TERMINATED | 127.0.0.1:50316 | tanh | 66.5306 | 100 | 16.9549 | 6.71228 | 100 | 10.3478 | 99 |
\n",
"\n",
"
\n",
"
\n",
"
\n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"tuner = tune.Tuner(\n",
" objective,\n",
" tune_config=tune.TuneConfig(\n",
" metric=\"mean_loss\",\n",
" mode=\"min\",\n",
" search_alg=algo,\n",
" num_samples=num_samples,\n",
" ),\n",
" param_space=search_config,\n",
")\n",
"results = tuner.fit()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "49be6f01",
"metadata": {},
"source": [
"Here are the hyperparameters found to minimize the mean loss of the defined objective."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "7036798c",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best hyperparameters found were: {'steps': 100, 'width': 6.206149011253133, 'height': -40.93182668460948, 'activation': 'tanh'}\n"
]
}
],
"source": [
"print(\"Best hyperparameters found were: \", results.get_best_result().config)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "504e9d2a",
"metadata": {},
"source": [
"## Conditional search spaces\n",
"\n",
"Sometimes we may want to build a more complicated search space that has conditional dependencies on other hyperparameters. In this case, we pass a nested dictionary to `objective_two`, which has been slightly adjusted from `objective` to deal with the conditional search space."
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "2f7b5449",
"metadata": {},
"outputs": [],
"source": [
"def evaluation_fn(step, width, height, mult=1):\n",
" return (0.1 + width * step / 100) ** (-1) + height * 0.1 * mult"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "4b83b81c",
"metadata": {},
"outputs": [],
"source": [
"def objective_two(config):\n",
" width, height = config[\"width\"], config[\"height\"]\n",
" sub_dict = config[\"activation\"]\n",
" mult = sub_dict.get(\"mult\", 1)\n",
" \n",
" for step in range(config[\"steps\"]):\n",
" intermediate_score = evaluation_fn(step, width, height, mult)\n",
" tune.report({\"iterations\": step, \"mean_loss\": intermediate_score})\n",
" time.sleep(0.1)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "75cea99e",
"metadata": {},
"outputs": [],
"source": [
"conditional_space = {\n",
" \"activation\": hp.choice(\n",
" \"activation\",\n",
" [\n",
" {\"activation\": \"relu\", \"mult\": hp.uniform(\"mult\", 1, 2)},\n",
" {\"activation\": \"tanh\"},\n",
" ],\n",
" ),\n",
" \"width\": hp.uniform(\"width\", 0, 20),\n",
" \"height\": hp.uniform(\"height\", -100, 100),\n",
" \"steps\": 100,\n",
"}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "7df282c1",
"metadata": {},
"source": [
"Now we the define the search algorithm built from `HyperOptSearch` constrained by `ConcurrencyLimiter`. When the hyperparameter search space is conditional, we pass it (`conditional_space`) into `HyperOptSearch`."
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "ea2c71a6",
"metadata": {},
"outputs": [],
"source": [
"algo = HyperOptSearch(space=conditional_space, metric=\"mean_loss\", mode=\"min\")\n",
"algo = ConcurrencyLimiter(algo, max_concurrent=4)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "630f84ab",
"metadata": {},
"source": [
"Now we run the experiment, this time with an empty `config` because we instead provided `space` to the `HyperOptSearch` `search_alg`."
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "14111e9e",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
\n",
"
\n",
"
Tune Status
\n",
"
\n",
"\n",
"Current time: | 2025-02-18 13:15:34 |
\n",
"Running for: | 00:00:34.71 |
\n",
"Memory: | 22.9/36.0 GiB |
\n",
"\n",
"
\n",
"
\n",
"
\n",
"
\n",
"
System Info
\n",
" Using FIFO scheduling algorithm.
Logical resource usage: 1.0/12 CPUs, 0/0 GPUs\n",
" \n",
" \n",
"
\n",
"
\n",
"
\n",
"
Trial Status
\n",
"
\n",
"\n",
"Trial name | status | loc | activation/activatio\n",
"n | activation/mult | height | steps | width | loss | iter | total time (s) | iterations |
\n",
"\n",
"\n",
"objective_two_13de5867 | TERMINATED | 127.0.0.1:50350 | tanh | | -35.0329 | 100 | 13.2254 | -3.42749 | 100 | 10.2102 | 99 |
\n",
"objective_two_3100f5ee | TERMINATED | 127.0.0.1:50355 | relu | 1.44584 | 76.2581 | 100 | 0.123165 | 15.5316 | 100 | 10.2683 | 99 |
\n",
"objective_two_6b4044aa | TERMINATED | 127.0.0.1:50356 | relu | 1.67475 | 57.9612 | 100 | 19.4794 | 9.75866 | 100 | 10.2724 | 99 |
\n",
"objective_two_4aa1269b | TERMINATED | 127.0.0.1:50357 | tanh | | -9.95686 | 100 | 12.9749 | -0.918437 | 100 | 10.2373 | 99 |
\n",
"objective_two_d8c42c9f | TERMINATED | 127.0.0.1:50402 | tanh | | -96.6184 | 100 | 8.03869 | -9.53774 | 100 | 10.2407 | 99 |
\n",
"objective_two_de956d10 | TERMINATED | 127.0.0.1:50404 | relu | 1.06986 | 9.64996 | 100 | 0.672962 | 2.3375 | 100 | 10.2427 | 99 |
\n",
"objective_two_0e2b2751 | TERMINATED | 127.0.0.1:50413 | tanh | | -82.9292 | 100 | 17.4889 | -8.2355 | 100 | 10.293 | 99 |
\n",
"objective_two_dad93a03 | TERMINATED | 127.0.0.1:50415 | relu | 1.85364 | -63.6309 | 100 | 16.6414 | -11.7345 | 100 | 10.285 | 99 |
\n",
"objective_two_e472727a | TERMINATED | 127.0.0.1:50442 | relu | 1.2359 | -75.2253 | 100 | 7.49782 | -9.16415 | 100 | 10.2506 | 99 |
\n",
"objective_two_5d1eacff | TERMINATED | 127.0.0.1:50449 | tanh | | -20.158 | 100 | 6.18643 | -1.85514 | 100 | 10.2566 | 99 |
\n",
"\n",
"
\n",
"
\n",
"
\n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"tuner = tune.Tuner(\n",
" objective_two,\n",
" tune_config=tune.TuneConfig(\n",
" metric=\"mean_loss\",\n",
" mode=\"min\",\n",
" search_alg=algo,\n",
" num_samples=num_samples,\n",
" ),\n",
")\n",
"results = tuner.fit()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "e6172afa",
"metadata": {},
"source": [
"Finally, we again show the hyperparameters that minimize the mean loss defined by the score of the objective function above. "
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "03c3fc49",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best hyperparameters found were: {'activation': {'activation': 'relu', 'mult': 1.8536380640438768}, 'height': -63.630920754630125, 'steps': 100, 'width': 16.641403933591928}\n"
]
}
],
"source": [
"print(\"Best hyperparameters found were: \", results.get_best_result().config)"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "2f7b72d3",
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [],
"source": [
"ray.shutdown()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "hyperopt_example",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.11"
},
"orphan": true
},
"nbformat": 4,
"nbformat_minor": 5
}