{ "cells": [ { "attachments": {}, "cell_type": "markdown", "id": "db54cdf9", "metadata": {}, "source": [ "# Running Tune experiments with BayesOpt\n", "\n", "In this tutorial we introduce BayesOpt, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with BayesOpt and, as a result, allow you to seamlessly scale up a BayesOpt optimization process - without sacrificing performance.\n", "\n", "BayesOpt is a constrained global optimization package utilizing Bayesian inference on gaussian processes, where the emphasis is on finding the maximum value of an unknown function in as few iterations as possible. BayesOpt's techniques are particularly suited for optimization of high cost functions, situations where the balance between exploration and exploitation is important. Therefore BayesOpt falls in the domain of \"derivative-free\" and \"black-box\" optimization. In this example we minimize a simple objective to briefly demonstrate the usage of BayesOpt with Ray Tune via `BayesOptSearch`, including conditional search spaces. It's useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume `bayesian-optimization==1.2.0` library is installed. To learn more, please refer to [BayesOpt website](https://github.com/fmfn/BayesianOptimization)." ] }, { "cell_type": "code", "execution_count": 1, "id": "7ed16354", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: bayesian-optimization==1.2.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (1.2.0)\n", "Requirement already satisfied: scikit-learn>=0.18.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from bayesian-optimization==1.2.0) (0.24.2)\n", "Requirement already satisfied: numpy>=1.9.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from bayesian-optimization==1.2.0) (1.21.6)\n", "Requirement already satisfied: scipy>=0.14.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from bayesian-optimization==1.2.0) (1.4.1)\n", "Requirement already satisfied: joblib>=0.11 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-learn>=0.18.0->bayesian-optimization==1.2.0) (1.1.0)\n", "Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-learn>=0.18.0->bayesian-optimization==1.2.0) (3.0.0)\n", "\u001b[33mWARNING: There was an error checking the latest version of pip.\u001b[0m\u001b[33m\n", "\u001b[0m" ] } ], "source": [ "# !pip install ray[tune]\n", "!pip install bayesian-optimization==1.2.0" ] }, { "attachments": {}, "cell_type": "markdown", "id": "2236f834", "metadata": {}, "source": [ "Click below to see all the imports we need for this example.\n", "You can also launch directly into a Binder instance to run this notebook yourself.\n", "Just click on the rocket symbol at the top of the navigation." ] }, { "cell_type": "code", "execution_count": 2, "id": "6d36c78b", "metadata": { "tags": [ "hide-input" ] }, "outputs": [], "source": [ "import time\n", "\n", "import ray\n", "from ray import train, tune\n", "from ray.tune.search import ConcurrencyLimiter\n", "from ray.tune.search.bayesopt import BayesOptSearch" ] }, { "attachments": {}, "cell_type": "markdown", "id": "6257a3a8", "metadata": {}, "source": [ "Let's start by defining a simple evaluation function.\n", "We artificially sleep for a bit (`0.1` seconds) to simulate a long-running ML experiment.\n", "This setup assumes that we're running multiple `step`s of an experiment and try to tune two hyperparameters,\n", "namely `width` and `height`." ] }, { "cell_type": "code", "execution_count": 3, "id": "646c75a9", "metadata": {}, "outputs": [], "source": [ "def evaluate(step, width, height):\n", " time.sleep(0.1)\n", " return (0.1 + width * step / 100) ** (-1) + height * 0.1" ] }, { "attachments": {}, "cell_type": "markdown", "id": "d89b7fdc", "metadata": {}, "source": [ "Next, our ``objective`` function takes a Tune ``config``, evaluates the `score` of your experiment in a training loop,\n", "and uses `train.report` to report the `score` back to Tune." ] }, { "cell_type": "code", "execution_count": 4, "id": "e9adf637", "metadata": {}, "outputs": [], "source": [ "def objective(config):\n", " for step in range(config[\"steps\"]):\n", " score = evaluate(step, config[\"width\"], config[\"height\"])\n", " train.report({\"iterations\": step, \"mean_loss\": score})" ] }, { "cell_type": "code", "execution_count": null, "id": "bc634b1d", "metadata": { "lines_to_next_cell": 0, "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "ray.init(configure_logging=False)" ] }, { "attachments": {}, "cell_type": "markdown", "id": "0b9a2c4d", "metadata": {}, "source": [ "Now we define the search algorithm built from `BayesOptSearch`, constrained to a maximum of `4` concurrent trials with a `ConcurrencyLimiter`." ] }, { "cell_type": "code", "execution_count": 6, "id": "6f1d2fe7", "metadata": {}, "outputs": [], "source": [ "algo = BayesOptSearch(utility_kwargs={\"kind\": \"ucb\", \"kappa\": 2.5, \"xi\": 0.0})\n", "algo = ConcurrencyLimiter(algo, max_concurrent=4)" ] }, { "attachments": {}, "cell_type": "markdown", "id": "27963e39", "metadata": {}, "source": [ "The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to `1000` samples.\n", "(you can decrease this if it takes too long on your machine)." ] }, { "cell_type": "code", "execution_count": 7, "id": "d777201c", "metadata": {}, "outputs": [], "source": [ "num_samples = 1000" ] }, { "cell_type": "code", "execution_count": 8, "id": "bb5f39a6", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "# If 1000 samples take too long, you can reduce this number.\n", "# We override this number here for our smoke tests.\n", "num_samples = 10" ] }, { "attachments": {}, "cell_type": "markdown", "id": "752523c8", "metadata": {}, "source": [ "Next we define a search space. The critical assumption is that the optimal hyperparameters live within this space. Yet, if the space is very large, then those hyperparameters may be difficult to find in a short amount of time." ] }, { "cell_type": "code", "execution_count": 9, "id": "116f8757", "metadata": {}, "outputs": [], "source": [ "search_space = {\n", " \"steps\": 100,\n", " \"width\": tune.uniform(0, 20),\n", " \"height\": tune.uniform(-100, 100),\n", "}" ] }, { "attachments": {}, "cell_type": "markdown", "id": "1754bf85", "metadata": {}, "source": [ "Finally, we run the experiment to `\"min\"`imize the \"mean_loss\" of the `objective` by searching `search_config` via `algo`, `num_samples` times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute `tuner.fit()`." ] }, { "cell_type": "code", "execution_count": 10, "id": "5c44a0c5", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\n" ] }, { "data": { "text/html": [ "== Status ==
Current time: 2022-07-22 15:30:53 (running for 00:00:43.91)
Memory usage on this node: 10.4/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/4.47 GiB heap, 0.0/2.0 GiB objects
Current best trial: d42ac71c with mean_loss=-9.536507956046009 and parameters={'steps': 100, 'width': 19.398197043239886, 'height': -95.88310114083951}
Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-30-08
Number of trials: 10/10 (10 TERMINATED)
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Trial name status loc height width loss iter total time (s) iterations neg_mean_loss
objective_c9daa5d4TERMINATED127.0.0.1:46960-25.092 19.0143 -2.45636 100 10.9865 99 2.45636
objective_cb9bc830TERMINATED127.0.0.1:46968 46.398811.9732 4.72354 100 11.5661 99 -4.72354
objective_cb9d338cTERMINATED127.0.0.1:46969-68.7963 3.11989-6.56602 100 11.648 99 6.56602
objective_cb9e97e0TERMINATED127.0.0.1:46970-88.383317.3235 -8.78036 100 11.6948 99 8.78036
objective_d229961eTERMINATED127.0.0.1:47009 20.223 14.1615 2.09312 100 10.8549 99 -2.09312
objective_d42ac71cTERMINATED127.0.0.1:47036-95.883119.3982 -9.53651 100 10.7931 99 9.53651
objective_d43ca61cTERMINATED127.0.0.1:47039 66.4885 4.24678 6.88118 100 10.7606 99 -6.88118
objective_d43fb190TERMINATED127.0.0.1:47040-63.635 3.66809-6.09551 100 10.7997 99 6.09551
objective_da1ff46cTERMINATED127.0.0.1:47057-39.151610.4951 -3.81983 100 10.7762 99 3.81983
objective_dc25c796TERMINATED127.0.0.1:47062-13.611 5.82458-1.19064 100 10.7213 99 1.19064


" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_c9daa5d4:\n", " date: 2022-07-22_15-30-12\n", " done: false\n", " experiment_id: 422a6d2a512a470480e33913d7825a7a\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 7.490802376947249\n", " neg_mean_loss: -7.490802376947249\n", " node_ip: 127.0.0.1\n", " pid: 46960\n", " time_since_restore: 0.1042318344116211\n", " time_this_iter_s: 0.1042318344116211\n", " time_total_s: 0.1042318344116211\n", " timestamp: 1658500212\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: c9daa5d4\n", " warmup_time: 0.0032601356506347656\n", " \n", "Result for objective_cb9bc830:\n", " date: 2022-07-22_15-30-15\n", " done: false\n", " experiment_id: 3a9a6bef89ec4b57bd0fa24dd3b407e6\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 14.639878836228101\n", " neg_mean_loss: -14.639878836228101\n", " node_ip: 127.0.0.1\n", " pid: 46968\n", " time_since_restore: 0.10442280769348145\n", " time_this_iter_s: 0.10442280769348145\n", " time_total_s: 0.10442280769348145\n", " timestamp: 1658500215\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: cb9bc830\n", " warmup_time: 0.0038840770721435547\n", " \n", "Result for objective_cb9e97e0:\n", " date: 2022-07-22_15-30-15\n", " done: false\n", " experiment_id: b0266e323ced4991b155344b34c25c59\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 1.1616722433639897\n", " neg_mean_loss: -1.1616722433639897\n", " node_ip: 127.0.0.1\n", " pid: 46970\n", " time_since_restore: 0.10328483581542969\n", " time_this_iter_s: 0.10328483581542969\n", " time_total_s: 0.10328483581542969\n", " timestamp: 1658500215\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: cb9e97e0\n", " warmup_time: 0.004090070724487305\n", " \n", "Result for objective_cb9d338c:\n", " date: 2022-07-22_15-30-15\n", " done: false\n", " experiment_id: 2731a83e40eb468fb79e19f872b8f597\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 3.120372808848731\n", " neg_mean_loss: -3.120372808848731\n", " node_ip: 127.0.0.1\n", " pid: 46969\n", " time_since_restore: 0.1042470932006836\n", " time_this_iter_s: 0.1042470932006836\n", " time_total_s: 0.1042470932006836\n", " timestamp: 1658500215\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: cb9d338c\n", " warmup_time: 0.003387928009033203\n", " \n", "Result for objective_c9daa5d4:\n", " date: 2022-07-22_15-30-17\n", " done: false\n", " experiment_id: 422a6d2a512a470480e33913d7825a7a\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 45\n", " iterations_since_restore: 46\n", " mean_loss: -2.393676542940848\n", " neg_mean_loss: 2.393676542940848\n", " node_ip: 127.0.0.1\n", " pid: 46960\n", " time_since_restore: 5.1730430126190186\n", " time_this_iter_s: 0.10674905776977539\n", " time_total_s: 5.1730430126190186\n", " timestamp: 1658500217\n", " timesteps_since_restore: 0\n", " training_iteration: 46\n", " trial_id: c9daa5d4\n", " warmup_time: 0.0032601356506347656\n", " \n", "Result for objective_cb9bc830:\n", " date: 2022-07-22_15-30-20\n", " done: false\n", " experiment_id: 3a9a6bef89ec4b57bd0fa24dd3b407e6\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 4.8144784432736065\n", " neg_mean_loss: -4.8144784432736065\n", " node_ip: 127.0.0.1\n", " pid: 46968\n", " time_since_restore: 5.1083409786224365\n", " time_this_iter_s: 0.10834097862243652\n", " time_total_s: 5.1083409786224365\n", " timestamp: 1658500220\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: cb9bc830\n", " warmup_time: 0.0038840770721435547\n", " \n", "Result for objective_cb9e97e0:\n", " date: 2022-07-22_15-30-20\n", " done: false\n", " experiment_id: b0266e323ced4991b155344b34c25c59\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -8.716998803293404\n", " neg_mean_loss: 8.716998803293404\n", " node_ip: 127.0.0.1\n", " pid: 46970\n", " time_since_restore: 5.117117881774902\n", " time_this_iter_s: 0.10473918914794922\n", " time_total_s: 5.117117881774902\n", " timestamp: 1658500220\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: cb9e97e0\n", " warmup_time: 0.004090070724487305\n", " \n", "Result for objective_cb9d338c:\n", " date: 2022-07-22_15-30-20\n", " done: false\n", " experiment_id: 2731a83e40eb468fb79e19f872b8f597\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -6.241199660085543\n", " neg_mean_loss: 6.241199660085543\n", " node_ip: 127.0.0.1\n", " pid: 46969\n", " time_since_restore: 5.1075780391693115\n", " time_this_iter_s: 0.1051321029663086\n", " time_total_s: 5.1075780391693115\n", " timestamp: 1658500220\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: cb9d338c\n", " warmup_time: 0.003387928009033203\n", " \n", "Result for objective_c9daa5d4:\n", " date: 2022-07-22_15-30-22\n", " done: false\n", " experiment_id: 422a6d2a512a470480e33913d7825a7a\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 92\n", " iterations_since_restore: 93\n", " mean_loss: -2.452357296882761\n", " neg_mean_loss: 2.452357296882761\n", " node_ip: 127.0.0.1\n", " pid: 46960\n", " time_since_restore: 10.23116397857666\n", " time_this_iter_s: 0.10653018951416016\n", " time_total_s: 10.23116397857666\n", " timestamp: 1658500222\n", " timesteps_since_restore: 0\n", " training_iteration: 93\n", " trial_id: c9daa5d4\n", " warmup_time: 0.0032601356506347656\n", " \n", "Result for objective_c9daa5d4:\n", " date: 2022-07-22_15-30-23\n", " done: true\n", " experiment_id: 422a6d2a512a470480e33913d7825a7a\n", " experiment_tag: 1_height=-25.0920,steps=100,width=19.0143\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -2.456355072354658\n", " neg_mean_loss: 2.456355072354658\n", " node_ip: 127.0.0.1\n", " pid: 46960\n", " time_since_restore: 10.986503839492798\n", " time_this_iter_s: 0.10757803916931152\n", " time_total_s: 10.986503839492798\n", " timestamp: 1658500223\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: c9daa5d4\n", " warmup_time: 0.0032601356506347656\n", " \n", "Result for objective_cb9bc830:\n", " date: 2022-07-22_15-30-24\n", " done: false\n", " experiment_id: 3a9a6bef89ec4b57bd0fa24dd3b407e6\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 91\n", " iterations_since_restore: 92\n", " mean_loss: 4.73082443425139\n", " neg_mean_loss: -4.73082443425139\n", " node_ip: 127.0.0.1\n", " pid: 46968\n", " time_since_restore: 9.829612970352173\n", " time_this_iter_s: 0.10725593566894531\n", " time_total_s: 9.829612970352173\n", " timestamp: 1658500224\n", " timesteps_since_restore: 0\n", " training_iteration: 92\n", " trial_id: cb9bc830\n", " warmup_time: 0.0038840770721435547\n", " \n", "Result for objective_cb9e97e0:\n", " date: 2022-07-22_15-30-24\n", " done: false\n", " experiment_id: b0266e323ced4991b155344b34c25c59\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 90\n", " iterations_since_restore: 91\n", " mean_loss: -8.774597648541096\n", " neg_mean_loss: 8.774597648541096\n", " node_ip: 127.0.0.1\n", " pid: 46970\n", " time_since_restore: 9.72621202468872\n", " time_this_iter_s: 0.10692906379699707\n", " time_total_s: 9.72621202468872\n", " timestamp: 1658500224\n", " timesteps_since_restore: 0\n", " training_iteration: 91\n", " trial_id: cb9e97e0\n", " warmup_time: 0.004090070724487305\n", " \n", "Result for objective_cb9d338c:\n", " date: 2022-07-22_15-30-24\n", " done: false\n", " experiment_id: 2731a83e40eb468fb79e19f872b8f597\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 90\n", " iterations_since_restore: 91\n", " mean_loss: -6.535736572413468\n", " neg_mean_loss: 6.535736572413468\n", " node_ip: 127.0.0.1\n", " pid: 46969\n", " time_since_restore: 9.71235203742981\n", " time_this_iter_s: 0.10665416717529297\n", " time_total_s: 9.71235203742981\n", " timestamp: 1658500224\n", " timesteps_since_restore: 0\n", " training_iteration: 91\n", " trial_id: cb9d338c\n", " warmup_time: 0.003387928009033203\n", " \n", "Result for objective_d229961e:\n", " date: 2022-07-22_15-30-25\n", " done: false\n", " experiment_id: d8bb04569c644d6fabad5064c1828ba3\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 12.022300234864176\n", " neg_mean_loss: -12.022300234864176\n", " node_ip: 127.0.0.1\n", " pid: 47009\n", " time_since_restore: 0.1041719913482666\n", " time_this_iter_s: 0.1041719913482666\n", " time_total_s: 0.1041719913482666\n", " timestamp: 1658500225\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: d229961e\n", " warmup_time: 0.003198862075805664\n", " \n", "Result for objective_cb9bc830:\n", " date: 2022-07-22_15-30-26\n", " done: true\n", " experiment_id: 3a9a6bef89ec4b57bd0fa24dd3b407e6\n", " experiment_tag: 2_height=46.3988,steps=100,width=11.9732\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 4.723536776402224\n", " neg_mean_loss: -4.723536776402224\n", " node_ip: 127.0.0.1\n", " pid: 46968\n", " time_since_restore: 11.566141843795776\n", " time_this_iter_s: 0.10738396644592285\n", " time_total_s: 11.566141843795776\n", " timestamp: 1658500226\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: cb9bc830\n", " warmup_time: 0.0038840770721435547\n", " \n", "Result for objective_cb9d338c:\n", " date: 2022-07-22_15-30-26\n", " done: true\n", " experiment_id: 2731a83e40eb468fb79e19f872b8f597\n", " experiment_tag: 3_height=-68.7963,steps=100,width=3.1199\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -6.566018929214734\n", " neg_mean_loss: 6.566018929214734\n", " node_ip: 127.0.0.1\n", " pid: 46969\n", " time_since_restore: 11.647998809814453\n", " time_this_iter_s: 0.1123647689819336\n", " time_total_s: 11.647998809814453\n", " timestamp: 1658500226\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: cb9d338c\n", " warmup_time: 0.003387928009033203\n", " \n", "Result for objective_cb9e97e0:\n", " date: 2022-07-22_15-30-26\n", " done: true\n", " experiment_id: b0266e323ced4991b155344b34c25c59\n", " experiment_tag: 4_height=-88.3833,steps=100,width=17.3235\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -8.780357708936942\n", " neg_mean_loss: 8.780357708936942\n", " node_ip: 127.0.0.1\n", " pid: 46970\n", " time_since_restore: 11.694752931594849\n", " time_this_iter_s: 0.12678027153015137\n", " time_total_s: 11.694752931594849\n", " timestamp: 1658500226\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: cb9e97e0\n", " warmup_time: 0.004090070724487305\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_d42ac71c:\n", " date: 2022-07-22_15-30-29\n", " done: false\n", " experiment_id: 3fdfaecb7adc4c5cb54c0aa76849d532\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 0.41168988591604894\n", " neg_mean_loss: -0.41168988591604894\n", " node_ip: 127.0.0.1\n", " pid: 47036\n", " time_since_restore: 0.10324597358703613\n", " time_this_iter_s: 0.10324597358703613\n", " time_total_s: 0.10324597358703613\n", " timestamp: 1658500229\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: d42ac71c\n", " warmup_time: 0.0028409957885742188\n", " \n", "Result for objective_d43ca61c:\n", " date: 2022-07-22_15-30-29\n", " done: false\n", " experiment_id: 8f92f519ea5443be9efd6f4a8937b8ee\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 16.648852816008436\n", " neg_mean_loss: -16.648852816008436\n", " node_ip: 127.0.0.1\n", " pid: 47039\n", " time_since_restore: 0.10412001609802246\n", " time_this_iter_s: 0.10412001609802246\n", " time_total_s: 0.10412001609802246\n", " timestamp: 1658500229\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: d43ca61c\n", " warmup_time: 0.002924203872680664\n", " \n", "Result for objective_d43fb190:\n", " date: 2022-07-22_15-30-29\n", " done: false\n", " experiment_id: 18283da742c74042ad3db1846fa7b460\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 3.6364993441420124\n", " neg_mean_loss: -3.6364993441420124\n", " node_ip: 127.0.0.1\n", " pid: 47040\n", " time_since_restore: 0.10391902923583984\n", " time_this_iter_s: 0.10391902923583984\n", " time_total_s: 0.10391902923583984\n", " timestamp: 1658500229\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: d43fb190\n", " warmup_time: 0.0027680397033691406\n", " \n", "Result for objective_d229961e:\n", " date: 2022-07-22_15-30-30\n", " done: false\n", " experiment_id: d8bb04569c644d6fabad5064c1828ba3\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 46\n", " iterations_since_restore: 47\n", " mean_loss: 2.1734885512401174\n", " neg_mean_loss: -2.1734885512401174\n", " node_ip: 127.0.0.1\n", " pid: 47009\n", " time_since_restore: 5.153247117996216\n", " time_this_iter_s: 0.10638809204101562\n", " time_total_s: 5.153247117996216\n", " timestamp: 1658500230\n", " timesteps_since_restore: 0\n", " training_iteration: 47\n", " trial_id: d229961e\n", " warmup_time: 0.003198862075805664\n", " \n", "Result for objective_d42ac71c:\n", " date: 2022-07-22_15-30-34\n", " done: false\n", " experiment_id: 3fdfaecb7adc4c5cb54c0aa76849d532\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 46\n", " iterations_since_restore: 47\n", " mean_loss: -9.477484325687673\n", " neg_mean_loss: 9.477484325687673\n", " node_ip: 127.0.0.1\n", " pid: 47036\n", " time_since_restore: 5.123893976211548\n", " time_this_iter_s: 0.10898423194885254\n", " time_total_s: 5.123893976211548\n", " timestamp: 1658500234\n", " timesteps_since_restore: 0\n", " training_iteration: 47\n", " trial_id: d42ac71c\n", " warmup_time: 0.0028409957885742188\n", " \n", "Result for objective_d43ca61c:\n", " date: 2022-07-22_15-30-34\n", " done: false\n", " experiment_id: 8f92f519ea5443be9efd6f4a8937b8ee\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 7.12595486600941\n", " neg_mean_loss: -7.12595486600941\n", " node_ip: 127.0.0.1\n", " pid: 47039\n", " time_since_restore: 5.194939136505127\n", " time_this_iter_s: 0.10889291763305664\n", " time_total_s: 5.194939136505127\n", " timestamp: 1658500234\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: d43ca61c\n", " warmup_time: 0.002924203872680664\n", " \n", "Result for objective_d43fb190:\n", " date: 2022-07-22_15-30-34\n", " done: false\n", " experiment_id: 18283da742c74042ad3db1846fa7b460\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -5.815255760980219\n", " neg_mean_loss: 5.815255760980219\n", " node_ip: 127.0.0.1\n", " pid: 47040\n", " time_since_restore: 5.2366979122161865\n", " time_this_iter_s: 0.10901784896850586\n", " time_total_s: 5.2366979122161865\n", " timestamp: 1658500234\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: d43fb190\n", " warmup_time: 0.0027680397033691406\n", " \n", "Result for objective_d229961e:\n", " date: 2022-07-22_15-30-35\n", " done: false\n", " experiment_id: d8bb04569c644d6fabad5064c1828ba3\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " mean_loss: 2.097657333615391\n", " neg_mean_loss: -2.097657333615391\n", " node_ip: 127.0.0.1\n", " pid: 47009\n", " time_since_restore: 10.209784984588623\n", " time_this_iter_s: 0.10757803916931152\n", " time_total_s: 10.209784984588623\n", " timestamp: 1658500235\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: d229961e\n", " warmup_time: 0.003198862075805664\n", " \n", "Result for objective_d229961e:\n", " date: 2022-07-22_15-30-36\n", " done: true\n", " experiment_id: d8bb04569c644d6fabad5064c1828ba3\n", " experiment_tag: 5_height=20.2230,steps=100,width=14.1615\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 2.093122581973529\n", " neg_mean_loss: -2.093122581973529\n", " node_ip: 127.0.0.1\n", " pid: 47009\n", " time_since_restore: 10.854872226715088\n", " time_this_iter_s: 0.10703516006469727\n", " time_total_s: 10.854872226715088\n", " timestamp: 1658500236\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: d229961e\n", " warmup_time: 0.003198862075805664\n", " \n", "Result for objective_da1ff46c:\n", " date: 2022-07-22_15-30-39\n", " done: false\n", " experiment_id: 9163132451a14ace8ddf394aeaae9018\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 6.0848448591907545\n", " neg_mean_loss: -6.0848448591907545\n", " node_ip: 127.0.0.1\n", " pid: 47057\n", " time_since_restore: 0.10405993461608887\n", " time_this_iter_s: 0.10405993461608887\n", " time_total_s: 0.10405993461608887\n", " timestamp: 1658500239\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: da1ff46c\n", " warmup_time: 0.0030031204223632812\n", " \n", "Result for objective_d42ac71c:\n", " date: 2022-07-22_15-30-39\n", " done: false\n", " experiment_id: 3fdfaecb7adc4c5cb54c0aa76849d532\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " mean_loss: -9.533184304791206\n", " neg_mean_loss: 9.533184304791206\n", " node_ip: 127.0.0.1\n", " pid: 47036\n", " time_since_restore: 10.145818948745728\n", " time_this_iter_s: 0.10763311386108398\n", " time_total_s: 10.145818948745728\n", " timestamp: 1658500239\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: d42ac71c\n", " warmup_time: 0.0028409957885742188\n", " \n", "Result for objective_d43ca61c:\n", " date: 2022-07-22_15-30-39\n", " done: false\n", " experiment_id: 8f92f519ea5443be9efd6f4a8937b8ee\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 6.893233568918634\n", " neg_mean_loss: -6.893233568918634\n", " node_ip: 127.0.0.1\n", " pid: 47039\n", " time_since_restore: 10.217039108276367\n", " time_this_iter_s: 0.10719418525695801\n", " time_total_s: 10.217039108276367\n", " timestamp: 1658500239\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: d43ca61c\n", " warmup_time: 0.002924203872680664\n", " \n", "Result for objective_d43fb190:\n", " date: 2022-07-22_15-30-39\n", " done: false\n", " experiment_id: 18283da742c74042ad3db1846fa7b460\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -6.08165210701758\n", " neg_mean_loss: 6.08165210701758\n", " node_ip: 127.0.0.1\n", " pid: 47040\n", " time_since_restore: 10.262099027633667\n", " time_this_iter_s: 0.10874485969543457\n", " time_total_s: 10.262099027633667\n", " timestamp: 1658500239\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: d43fb190\n", " warmup_time: 0.0027680397033691406\n", " \n", "Result for objective_d42ac71c:\n", " date: 2022-07-22_15-30-39\n", " done: true\n", " experiment_id: 3fdfaecb7adc4c5cb54c0aa76849d532\n", " experiment_tag: 6_height=-95.8831,steps=100,width=19.3982\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -9.536507956046009\n", " neg_mean_loss: 9.536507956046009\n", " node_ip: 127.0.0.1\n", " pid: 47036\n", " time_since_restore: 10.793061017990112\n", " time_this_iter_s: 0.10741710662841797\n", " time_total_s: 10.793061017990112\n", " timestamp: 1658500239\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: d42ac71c\n", " warmup_time: 0.0028409957885742188\n", " \n", "Result for objective_d43ca61c:\n", " date: 2022-07-22_15-30-40\n", " done: true\n", " experiment_id: 8f92f519ea5443be9efd6f4a8937b8ee\n", " experiment_tag: 7_height=66.4885,steps=100,width=4.2468\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 6.881177852950684\n", " neg_mean_loss: -6.881177852950684\n", " node_ip: 127.0.0.1\n", " pid: 47039\n", " time_since_restore: 10.760617017745972\n", " time_this_iter_s: 0.10911297798156738\n", " time_total_s: 10.760617017745972\n", " timestamp: 1658500240\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: d43ca61c\n", " warmup_time: 0.002924203872680664\n", " \n", "Result for objective_d43fb190:\n", " date: 2022-07-22_15-30-40\n", " done: true\n", " experiment_id: 18283da742c74042ad3db1846fa7b460\n", " experiment_tag: 8_height=-63.6350,steps=100,width=3.6681\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -6.09550539698523\n", " neg_mean_loss: 6.09550539698523\n", " node_ip: 127.0.0.1\n", " pid: 47040\n", " time_since_restore: 10.799743175506592\n", " time_this_iter_s: 0.1067342758178711\n", " time_total_s: 10.799743175506592\n", " timestamp: 1658500240\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: d43fb190\n", " warmup_time: 0.0027680397033691406\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_dc25c796:\n", " date: 2022-07-22_15-30-42\n", " done: false\n", " experiment_id: c0f302c32b284f8e99dbdfa90657ee7d\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 8.638900372842315\n", " neg_mean_loss: -8.638900372842315\n", " node_ip: 127.0.0.1\n", " pid: 47062\n", " time_since_restore: 0.10459494590759277\n", " time_this_iter_s: 0.10459494590759277\n", " time_total_s: 0.10459494590759277\n", " timestamp: 1658500242\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: dc25c796\n", " warmup_time: 0.002794981002807617\n", " \n", "Result for objective_da1ff46c:\n", " date: 2022-07-22_15-30-44\n", " done: false\n", " experiment_id: 9163132451a14ace8ddf394aeaae9018\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -3.7164550549457847\n", " neg_mean_loss: 3.7164550549457847\n", " node_ip: 127.0.0.1\n", " pid: 47057\n", " time_since_restore: 5.180424928665161\n", " time_this_iter_s: 0.10843396186828613\n", " time_total_s: 5.180424928665161\n", " timestamp: 1658500244\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: da1ff46c\n", " warmup_time: 0.0030031204223632812\n", " \n", "Result for objective_dc25c796:\n", " date: 2022-07-22_15-30-47\n", " done: false\n", " experiment_id: c0f302c32b284f8e99dbdfa90657ee7d\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -1.0086834162426133\n", " neg_mean_loss: 1.0086834162426133\n", " node_ip: 127.0.0.1\n", " pid: 47062\n", " time_since_restore: 5.151978015899658\n", " time_this_iter_s: 0.10736894607543945\n", " time_total_s: 5.151978015899658\n", " timestamp: 1658500247\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: dc25c796\n", " warmup_time: 0.002794981002807617\n", " \n", "Result for objective_da1ff46c:\n", " date: 2022-07-22_15-30-49\n", " done: false\n", " experiment_id: 9163132451a14ace8ddf394aeaae9018\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -3.814808150093952\n", " neg_mean_loss: 3.814808150093952\n", " node_ip: 127.0.0.1\n", " pid: 47057\n", " time_since_restore: 10.23661208152771\n", " time_this_iter_s: 0.1076211929321289\n", " time_total_s: 10.23661208152771\n", " timestamp: 1658500249\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: da1ff46c\n", " warmup_time: 0.0030031204223632812\n", " \n", "Result for objective_da1ff46c:\n", " date: 2022-07-22_15-30-49\n", " done: true\n", " experiment_id: 9163132451a14ace8ddf394aeaae9018\n", " experiment_tag: 9_height=-39.1516,steps=100,width=10.4951\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -3.819827867781687\n", " neg_mean_loss: 3.819827867781687\n", " node_ip: 127.0.0.1\n", " pid: 47057\n", " time_since_restore: 10.77621078491211\n", " time_this_iter_s: 0.10817480087280273\n", " time_total_s: 10.77621078491211\n", " timestamp: 1658500249\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: da1ff46c\n", " warmup_time: 0.0030031204223632812\n", " \n", "Result for objective_dc25c796:\n", " date: 2022-07-22_15-30-52\n", " done: false\n", " experiment_id: c0f302c32b284f8e99dbdfa90657ee7d\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -1.1817308993292515\n", " neg_mean_loss: 1.1817308993292515\n", " node_ip: 127.0.0.1\n", " pid: 47062\n", " time_since_restore: 10.179337978363037\n", " time_this_iter_s: 0.1043100357055664\n", " time_total_s: 10.179337978363037\n", " timestamp: 1658500252\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: dc25c796\n", " warmup_time: 0.002794981002807617\n", " \n", "Result for objective_dc25c796:\n", " date: 2022-07-22_15-30-53\n", " done: true\n", " experiment_id: c0f302c32b284f8e99dbdfa90657ee7d\n", " experiment_tag: 10_height=-13.6110,steps=100,width=5.8246\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -1.190635502081924\n", " neg_mean_loss: 1.190635502081924\n", " node_ip: 127.0.0.1\n", " pid: 47062\n", " time_since_restore: 10.721266031265259\n", " time_this_iter_s: 0.10741806030273438\n", " time_total_s: 10.721266031265259\n", " timestamp: 1658500253\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: dc25c796\n", " warmup_time: 0.002794981002807617\n", " \n" ] } ], "source": [ "tuner = tune.Tuner(\n", " objective,\n", " tune_config=tune.TuneConfig(\n", " metric=\"mean_loss\",\n", " mode=\"min\",\n", " search_alg=algo,\n", " num_samples=num_samples,\n", " ),\n", " param_space=search_space,\n", ")\n", "results = tuner.fit()" ] }, { "attachments": {}, "cell_type": "markdown", "id": "477f099b", "metadata": {}, "source": [ "Here are the hyperparamters found to minimize the mean loss of the defined objective." ] }, { "cell_type": "code", "execution_count": 11, "id": "3488aefa", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best hyperparameters found were: {'steps': 100, 'width': 19.398197043239886, 'height': -95.88310114083951}\n" ] } ], "source": [ "print(\"Best hyperparameters found were: \", results.get_best_result().config)" ] }, { "cell_type": "code", "execution_count": 12, "id": "2936353a", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "ray.shutdown()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.7" }, "orphan": true }, "nbformat": 4, "nbformat_minor": 5 }