{ "cells": [ { "attachments": {}, "cell_type": "markdown", "id": "58fc50bc", "metadata": {}, "source": [ "# Running Tune experiments with HyperOpt\n", "\n", "\n", " \"try-anyscale-quickstart\"\n", "\n", "

\n", "\n", "In this tutorial we introduce HyperOpt, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with HyperOpt and, as a result, allow you to seamlessly scale up a Hyperopt optimization process - without sacrificing performance.\n", "\n", "HyperOpt provides gradient/derivative-free optimization able to handle noise over the objective landscape, including evolutionary, bandit, and Bayesian optimization algorithms. HyperOpt internally supports search spaces which are continuous, discrete or a mixture of thereof. It also provides a library of functions on which to test the optimization algorithms and compare with other benchmarks.\n", "\n", "In this example we minimize a simple objective to briefly demonstrate the usage of HyperOpt with Ray Tune via `HyperOptSearch`. It's useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume `hyperopt==0.2.5` library is installed. To learn more, please refer to [HyperOpt website](http://hyperopt.github.io/hyperopt).\n", "\n", "We include a important example on conditional search spaces (stringing together relationships among hyperparameters)." ] }, { "attachments": {}, "cell_type": "markdown", "id": "e4586d28", "metadata": {}, "source": [ "Background information:\n", "- [HyperOpt website](http://hyperopt.github.io/hyperopt)\n", "\n", "Necessary requirements:\n", "- `pip install \"ray[tune]\" hyperopt==0.2.5`" ] }, { "cell_type": "code", "execution_count": 1, "id": "6567f2dc", "metadata": { "tags": [ "hide-cell" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: hyperopt==0.2.5 in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (0.2.5)\n", "Requirement already satisfied: numpy in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (from hyperopt==0.2.5) (2.2.3)\n", "Requirement already satisfied: scipy in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (from hyperopt==0.2.5) (1.15.2)\n", "Requirement already satisfied: six in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (from hyperopt==0.2.5) (1.17.0)\n", "Requirement already satisfied: networkx>=2.2 in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (from hyperopt==0.2.5) (3.4.2)\n", "Requirement already satisfied: future in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (from hyperopt==0.2.5) (1.0.0)\n", "Requirement already satisfied: tqdm in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (from hyperopt==0.2.5) (4.67.1)\n", "Requirement already satisfied: cloudpickle in /opt/miniconda3/envs/hyperopt_example/lib/python3.11/site-packages (from hyperopt==0.2.5) (3.1.1)\n" ] } ], "source": [ "# install in a hidden cell\n", "# !pip install \"ray[tune]\"\n", "!pip install hyperopt==0.2.5" ] }, { "attachments": {}, "cell_type": "markdown", "id": "b8e9e0cd", "metadata": {}, "source": [ "Click below to see all the imports we need for this example." ] }, { "cell_type": "code", "execution_count": 2, "id": "6592315e", "metadata": { "tags": [ "hide-input", "hide-output" ] }, "outputs": [], "source": [ "import time\n", "\n", "import ray\n", "from ray import tune\n", "from ray.tune.search import ConcurrencyLimiter\n", "from ray.tune.search.hyperopt import HyperOptSearch\n", "from hyperopt import hp" ] }, { "attachments": {}, "cell_type": "markdown", "id": "d4b6d1d5", "metadata": {}, "source": [ "Let's start by defining a simple evaluation function.\n", "We artificially sleep for a bit (`0.1` seconds) to simulate a long-running ML experiment.\n", "This setup assumes that we're running multiple `step`s of an experiment and try to tune two hyperparameters,\n", "namely `width` and `height`." ] }, { "cell_type": "code", "execution_count": 3, "id": "12d4efc8", "metadata": {}, "outputs": [], "source": [ "def evaluate(step, width, height):\n", " time.sleep(0.1)\n", " return (0.1 + width * step / 100) ** (-1) + height * 0.1" ] }, { "attachments": {}, "cell_type": "markdown", "id": "4f4f5aa2", "metadata": {}, "source": [ "Next, our ``objective`` function takes a Tune ``config``, evaluates the `score` of your experiment in a training loop,\n", "and uses `tune.report` to report the `score` back to Tune." ] }, { "cell_type": "code", "execution_count": 4, "id": "c9818009", "metadata": {}, "outputs": [], "source": [ "def objective(config):\n", " for step in range(config[\"steps\"]):\n", " score = evaluate(step, config[\"width\"], config[\"height\"])\n", " tune.report({\"iterations\": step, \"mean_loss\": score})" ] }, { "cell_type": "code", "execution_count": 5, "id": "33eddcb9", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "59f8090dd37c473cb24b97b6b28d109b", "version_major": 2, "version_minor": 0 }, "text/html": [ "
\n", "
\n", "
\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Python version:3.11.11
Ray version:2.42.1
\n", "\n", "
\n", "
\n" ], "text/plain": [ "RayContext(dashboard_url='', python_version='3.11.11', ray_version='2.42.1', ray_commit='c2e38f7b75be223c0c033986472daada8622d64f')" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "ray.init(configure_logging=False)" ] }, { "attachments": {}, "cell_type": "markdown", "id": "5be35d5e", "metadata": {}, "source": [ "While defining the search algorithm, we may choose to provide an initial set of hyperparameters that we believe are especially promising or informative, and\n", "pass this information as a helpful starting point for the `HyperOptSearch` object.\n", "\n", "We also set the maximum concurrent trials to `4` with a `ConcurrencyLimiter`." ] }, { "cell_type": "code", "execution_count": 6, "id": "d4615bed", "metadata": { "lines_to_next_cell": 0 }, "outputs": [], "source": [ "initial_params = [\n", " {\"width\": 1, \"height\": 2, \"activation\": \"relu\"},\n", " {\"width\": 4, \"height\": 2, \"activation\": \"tanh\"},\n", "]\n", "algo = HyperOptSearch(points_to_evaluate=initial_params)\n", "algo = ConcurrencyLimiter(algo, max_concurrent=4)" ] }, { "attachments": {}, "cell_type": "markdown", "id": "2a51e7c1", "metadata": {}, "source": [ "The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to `1000` samples.\n", "(you can decrease this if it takes too long on your machine)." ] }, { "cell_type": "code", "execution_count": 7, "id": "2dbb2be0", "metadata": {}, "outputs": [], "source": [ "num_samples = 1000" ] }, { "cell_type": "code", "execution_count": 8, "id": "950558ed", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "# If 1000 samples take too long, you can reduce this number.\n", "# We override this number here for our smoke tests.\n", "num_samples = 10" ] }, { "attachments": {}, "cell_type": "markdown", "id": "6e3629cb", "metadata": {}, "source": [ "Next we define a search space. The critical assumption is that the optimal hyperparameters live within this space. Yet, if the space is very large, then those hyperparameters may be difficult to find in a short amount of time." ] }, { "cell_type": "code", "execution_count": 9, "id": "65189946", "metadata": {}, "outputs": [], "source": [ "search_config = {\n", " \"steps\": 100,\n", " \"width\": tune.uniform(0, 20),\n", " \"height\": tune.uniform(-100, 100),\n", " \"activation\": tune.choice([\"relu\", \"tanh\"])\n", "}" ] }, { "attachments": {}, "cell_type": "markdown", "id": "1b94c93b", "metadata": {}, "source": [ "Finally, we run the experiment to `\"min\"`imize the \"mean_loss\" of the `objective` by searching `search_config` via `algo`, `num_samples` times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute `tuner.fit()`." ] }, { "cell_type": "code", "execution_count": 10, "id": "9a99a3a7", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "
\n", "
\n", "

Tune Status

\n", " \n", "\n", "\n", "\n", "\n", "\n", "
Current time:2025-02-18 13:14:59
Running for: 00:00:36.03
Memory: 22.1/36.0 GiB
\n", "
\n", "
\n", "
\n", "

System Info

\n", " Using FIFO scheduling algorithm.
Logical resource usage: 1.0/12 CPUs, 0/0 GPUs\n", "
\n", " \n", "
\n", "
\n", "
\n", "

Trial Status

\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Trial name status loc activation height steps width loss iter total time (s) iterations
objective_5b05c00aTERMINATED127.0.0.1:50205relu 2 100 1 1.11743 100 10.335 99
objective_b813f49dTERMINATED127.0.0.1:50207tanh 2 100 4 0.446305 100 10.3299 99
objective_6dadd2bdTERMINATED127.0.0.1:50212tanh -40.9318 100 6.20615 -3.93303 100 10.3213 99
objective_9faffc0fTERMINATED127.0.0.1:50217tanh 91.9688 100 9.25147 9.30488 100 10.353 99
objective_7834e74cTERMINATED127.0.0.1:50266tanh -17.9521 10011.436 -1.70766 100 10.3753 99
objective_741253c7TERMINATED127.0.0.1:50271tanh 58.1279 100 0.737879 7.01689 100 10.3565 99
objective_39682bcfTERMINATED127.0.0.1:50272tanh -31.2589 100 4.89265 -2.92361 100 10.3225 99
objective_bfc7e150TERMINATED127.0.0.1:50274tanh -14.7877 100 7.36477 -1.34347 100 10.3744 99
objective_e1f1a193TERMINATED127.0.0.1:50314tanh 50.9579 10017.4499 5.15334 100 10.3675 99
objective_1192dd4eTERMINATED127.0.0.1:50316tanh 66.5306 10016.9549 6.71228 100 10.3478 99
\n", "
\n", "
\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "tuner = tune.Tuner(\n", " objective,\n", " tune_config=tune.TuneConfig(\n", " metric=\"mean_loss\",\n", " mode=\"min\",\n", " search_alg=algo,\n", " num_samples=num_samples,\n", " ),\n", " param_space=search_config,\n", ")\n", "results = tuner.fit()" ] }, { "attachments": {}, "cell_type": "markdown", "id": "49be6f01", "metadata": {}, "source": [ "Here are the hyperparameters found to minimize the mean loss of the defined objective." ] }, { "cell_type": "code", "execution_count": 11, "id": "7036798c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best hyperparameters found were: {'steps': 100, 'width': 6.206149011253133, 'height': -40.93182668460948, 'activation': 'tanh'}\n" ] } ], "source": [ "print(\"Best hyperparameters found were: \", results.get_best_result().config)" ] }, { "attachments": {}, "cell_type": "markdown", "id": "504e9d2a", "metadata": {}, "source": [ "## Conditional search spaces\n", "\n", "Sometimes we may want to build a more complicated search space that has conditional dependencies on other hyperparameters. In this case, we pass a nested dictionary to `objective_two`, which has been slightly adjusted from `objective` to deal with the conditional search space." ] }, { "cell_type": "code", "execution_count": 12, "id": "2f7b5449", "metadata": {}, "outputs": [], "source": [ "def evaluation_fn(step, width, height, mult=1):\n", " return (0.1 + width * step / 100) ** (-1) + height * 0.1 * mult" ] }, { "cell_type": "code", "execution_count": 13, "id": "4b83b81c", "metadata": {}, "outputs": [], "source": [ "def objective_two(config):\n", " width, height = config[\"width\"], config[\"height\"]\n", " sub_dict = config[\"activation\"]\n", " mult = sub_dict.get(\"mult\", 1)\n", " \n", " for step in range(config[\"steps\"]):\n", " intermediate_score = evaluation_fn(step, width, height, mult)\n", " tune.report({\"iterations\": step, \"mean_loss\": intermediate_score})\n", " time.sleep(0.1)" ] }, { "cell_type": "code", "execution_count": 14, "id": "75cea99e", "metadata": {}, "outputs": [], "source": [ "conditional_space = {\n", " \"activation\": hp.choice(\n", " \"activation\",\n", " [\n", " {\"activation\": \"relu\", \"mult\": hp.uniform(\"mult\", 1, 2)},\n", " {\"activation\": \"tanh\"},\n", " ],\n", " ),\n", " \"width\": hp.uniform(\"width\", 0, 20),\n", " \"height\": hp.uniform(\"height\", -100, 100),\n", " \"steps\": 100,\n", "}" ] }, { "attachments": {}, "cell_type": "markdown", "id": "7df282c1", "metadata": {}, "source": [ "Now we the define the search algorithm built from `HyperOptSearch` constrained by `ConcurrencyLimiter`. When the hyperparameter search space is conditional, we pass it (`conditional_space`) into `HyperOptSearch`." ] }, { "cell_type": "code", "execution_count": 15, "id": "ea2c71a6", "metadata": {}, "outputs": [], "source": [ "algo = HyperOptSearch(space=conditional_space, metric=\"mean_loss\", mode=\"min\")\n", "algo = ConcurrencyLimiter(algo, max_concurrent=4)" ] }, { "attachments": {}, "cell_type": "markdown", "id": "630f84ab", "metadata": {}, "source": [ "Now we run the experiment, this time with an empty `config` because we instead provided `space` to the `HyperOptSearch` `search_alg`." ] }, { "cell_type": "code", "execution_count": 16, "id": "14111e9e", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "
\n", "
\n", "

Tune Status

\n", " \n", "\n", "\n", "\n", "\n", "\n", "
Current time:2025-02-18 13:15:34
Running for: 00:00:34.71
Memory: 22.9/36.0 GiB
\n", "
\n", "
\n", "
\n", "

System Info

\n", " Using FIFO scheduling algorithm.
Logical resource usage: 1.0/12 CPUs, 0/0 GPUs\n", "
\n", " \n", "
\n", "
\n", "
\n", "

Trial Status

\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Trial name status loc activation/activatio\n", "n activation/mult height steps width loss iter total time (s) iterations
objective_two_13de5867TERMINATED127.0.0.1:50350tanh -35.0329 10013.2254 -3.42749 100 10.2102 99
objective_two_3100f5eeTERMINATED127.0.0.1:50355relu 1.44584 76.2581 100 0.123165 15.5316 100 10.2683 99
objective_two_6b4044aaTERMINATED127.0.0.1:50356relu 1.67475 57.9612 10019.4794 9.75866 100 10.2724 99
objective_two_4aa1269bTERMINATED127.0.0.1:50357tanh -9.95686 10012.9749 -0.918437 100 10.2373 99
objective_two_d8c42c9fTERMINATED127.0.0.1:50402tanh -96.6184 100 8.03869 -9.53774 100 10.2407 99
objective_two_de956d10TERMINATED127.0.0.1:50404relu 1.06986 9.64996 100 0.672962 2.3375 100 10.2427 99
objective_two_0e2b2751TERMINATED127.0.0.1:50413tanh -82.9292 10017.4889 -8.2355 100 10.293 99
objective_two_dad93a03TERMINATED127.0.0.1:50415relu 1.85364-63.6309 10016.6414 -11.7345 100 10.285 99
objective_two_e472727aTERMINATED127.0.0.1:50442relu 1.2359 -75.2253 100 7.49782 -9.16415 100 10.2506 99
objective_two_5d1eacffTERMINATED127.0.0.1:50449tanh -20.158 100 6.18643 -1.85514 100 10.2566 99
\n", "
\n", "
\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "tuner = tune.Tuner(\n", " objective_two,\n", " tune_config=tune.TuneConfig(\n", " metric=\"mean_loss\",\n", " mode=\"min\",\n", " search_alg=algo,\n", " num_samples=num_samples,\n", " ),\n", ")\n", "results = tuner.fit()" ] }, { "attachments": {}, "cell_type": "markdown", "id": "e6172afa", "metadata": {}, "source": [ "Finally, we again show the hyperparameters that minimize the mean loss defined by the score of the objective function above. " ] }, { "cell_type": "code", "execution_count": 17, "id": "03c3fc49", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best hyperparameters found were: {'activation': {'activation': 'relu', 'mult': 1.8536380640438768}, 'height': -63.630920754630125, 'steps': 100, 'width': 16.641403933591928}\n" ] } ], "source": [ "print(\"Best hyperparameters found were: \", results.get_best_result().config)" ] }, { "cell_type": "code", "execution_count": 18, "id": "2f7b72d3", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "ray.shutdown()" ] } ], "metadata": { "kernelspec": { "display_name": "hyperopt_example", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.11" }, "orphan": true }, "nbformat": 4, "nbformat_minor": 5 }