{ "cells": [ { "attachments": {}, "cell_type": "markdown", "id": "58fc50bc", "metadata": {}, "source": [ "# Running Tune experiments with HyperOpt\n", "\n", "In this tutorial we introduce HyperOpt, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with HyperOpt and, as a result, allow you to seamlessly scale up a Hyperopt optimization process - without sacrificing performance.\n", "\n", "HyperOpt provides gradient/derivative-free optimization able to handle noise over the objective landscape, including evolutionary, bandit, and Bayesian optimization algorithms. Nevergrad internally supports search spaces which are continuous, discrete or a mixture of thereof. It also provides a library of functions on which to test the optimization algorithms and compare with other benchmarks.\n", "\n", "In this example we minimize a simple objective to briefly demonstrate the usage of HyperOpt with Ray Tune via `HyperOptSearch`. It's useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume `hyperopt==0.2.5` library is installed. To learn more, please refer to [HyperOpt website](http://hyperopt.github.io/hyperopt).\n", "\n", "We include a important example on conditional search spaces (stringing together relationships among hyperparameters)." ] }, { "attachments": {}, "cell_type": "markdown", "id": "e4586d28", "metadata": {}, "source": [ "Background information:\n", "- [HyperOpt website](http://hyperopt.github.io/hyperopt)\n", "\n", "Necessary requirements:\n", "- `pip install ray[tune]`\n", "- `pip install hyperopt==0.2.5`" ] }, { "cell_type": "code", "execution_count": 1, "id": "6567f2dc", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Collecting hyperopt==0.2.5\n", " Using cached hyperopt-0.2.5-py2.py3-none-any.whl (965 kB)\n", "Requirement already satisfied: numpy in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from hyperopt==0.2.5) (1.21.6)\n", "Requirement already satisfied: cloudpickle in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from hyperopt==0.2.5) (2.0.0)\n", "Requirement already satisfied: six in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from hyperopt==0.2.5) (1.16.0)\n", "Requirement already satisfied: scipy in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from hyperopt==0.2.5) (1.4.1)\n", "Requirement already satisfied: future in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from hyperopt==0.2.5) (0.18.2)\n", "Requirement already satisfied: networkx>=2.2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from hyperopt==0.2.5) (2.6.3)\n", "Requirement already satisfied: tqdm in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from hyperopt==0.2.5) (4.64.0)\n", "Installing collected packages: hyperopt\n", " Attempting uninstall: hyperopt\n", " Found existing installation: hyperopt 0.2.7\n", " Uninstalling hyperopt-0.2.7:\n", " Successfully uninstalled hyperopt-0.2.7\n", "Successfully installed hyperopt-0.2.5\n", "\u001b[33mWARNING: There was an error checking the latest version of pip.\u001b[0m\u001b[33m\n", "\u001b[0m" ] } ], "source": [ "# !pip install ray[tune]\n", "!pip install hyperopt==0.2.5" ] }, { "attachments": {}, "cell_type": "markdown", "id": "b8e9e0cd", "metadata": {}, "source": [ "Click below to see all the imports we need for this example.\n", "You can also launch directly into a Binder instance to run this notebook yourself.\n", "Just click on the rocket symbol at the top of the navigation." ] }, { "cell_type": "code", "execution_count": 2, "id": "6592315e", "metadata": { "tags": [ "hide-input" ] }, "outputs": [], "source": [ "import time\n", "\n", "import ray\n", "from ray import train, tune\n", "from ray.tune.search import ConcurrencyLimiter\n", "from ray.tune.search.hyperopt import HyperOptSearch\n", "from hyperopt import hp" ] }, { "attachments": {}, "cell_type": "markdown", "id": "d4b6d1d5", "metadata": {}, "source": [ "Let's start by defining a simple evaluation function.\n", "We artificially sleep for a bit (`0.1` seconds) to simulate a long-running ML experiment.\n", "This setup assumes that we're running multiple `step`s of an experiment and try to tune two hyperparameters,\n", "namely `width` and `height`." ] }, { "cell_type": "code", "execution_count": 3, "id": "12d4efc8", "metadata": {}, "outputs": [], "source": [ "def evaluate(step, width, height):\n", " time.sleep(0.1)\n", " return (0.1 + width * step / 100) ** (-1) + height * 0.1" ] }, { "attachments": {}, "cell_type": "markdown", "id": "4f4f5aa2", "metadata": {}, "source": [ "Next, our ``objective`` function takes a Tune ``config``, evaluates the `score` of your experiment in a training loop,\n", "and uses `train.report` to report the `score` back to Tune." ] }, { "cell_type": "code", "execution_count": 4, "id": "c9818009", "metadata": {}, "outputs": [], "source": [ "def objective(config):\n", " for step in range(config[\"steps\"]):\n", " score = evaluate(step, config[\"width\"], config[\"height\"])\n", " train.report({\"iterations\": step, \"mean_loss\": score})" ] }, { "cell_type": "code", "execution_count": null, "id": "33eddcb9", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "ray.init(configure_logging=False)" ] }, { "attachments": {}, "cell_type": "markdown", "id": "5be35d5e", "metadata": {}, "source": [ "While defining the search algorithm, we may choose to provide an initial set of hyperparameters that we believe are especially promising or informative, and\n", "pass this information as a helpful starting point for the `HyperOptSearch` object.\n", "\n", "We also set the maximum concurrent trials to `4` with a `ConcurrencyLimiter`." ] }, { "cell_type": "code", "execution_count": 6, "id": "d4615bed", "metadata": { "lines_to_next_cell": 0 }, "outputs": [], "source": [ "initial_params = [\n", " {\"width\": 1, \"height\": 2, \"activation\": \"relu\"},\n", " {\"width\": 4, \"height\": 2, \"activation\": \"tanh\"},\n", "]\n", "algo = HyperOptSearch(points_to_evaluate=initial_params)\n", "algo = ConcurrencyLimiter(algo, max_concurrent=4)" ] }, { "attachments": {}, "cell_type": "markdown", "id": "2a51e7c1", "metadata": {}, "source": [ "The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to `1000` samples.\n", "(you can decrease this if it takes too long on your machine)." ] }, { "cell_type": "code", "execution_count": 7, "id": "2dbb2be0", "metadata": {}, "outputs": [], "source": [ "num_samples = 1000" ] }, { "cell_type": "code", "execution_count": 8, "id": "950558ed", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "# If 1000 samples take too long, you can reduce this number.\n", "# We override this number here for our smoke tests.\n", "num_samples = 10" ] }, { "attachments": {}, "cell_type": "markdown", "id": "6e3629cb", "metadata": {}, "source": [ "Next we define a search space. The critical assumption is that the optimal hyperparamters live within this space. Yet, if the space is very large, then those hyperparameters may be difficult to find in a short amount of time." ] }, { "cell_type": "code", "execution_count": 9, "id": "65189946", "metadata": {}, "outputs": [], "source": [ "search_config = {\n", " \"steps\": 100,\n", " \"width\": tune.uniform(0, 20),\n", " \"height\": tune.uniform(-100, 100),\n", " \"activation\": tune.choice([\"relu\", \"tanh\"])\n", "}" ] }, { "attachments": {}, "cell_type": "markdown", "id": "1b94c93b", "metadata": {}, "source": [ "Finally, we run the experiment to `\"min\"`imize the \"mean_loss\" of the `objective` by searching `search_config` via `algo`, `num_samples` times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute `tuner.fit()`." ] }, { "cell_type": "code", "execution_count": 10, "id": "9a99a3a7", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "\n" ] }, { "data": { "text/html": [ "== Status ==
Current time: 2022-07-22 15:31:49 (running for 00:00:43.71)
Memory usage on this node: 10.4/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.29 GiB heap, 0.0/2.0 GiB objects
Current best trial: f59fe9d6 with mean_loss=-2.5719085451008423 and parameters={'steps': 100, 'width': 5.584304853357766, 'height': -27.49576980043919, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-31-04
Number of trials: 10/10 (10 TERMINATED)
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Trial name status loc activation height steps width loss iter total time (s) iterations neg_mean_loss
objective_eb74f122TERMINATED127.0.0.1:47156relu 2 100 1 1.11743 100 10.9008 99 -1.11743
objective_ed2010ecTERMINATED127.0.0.1:47161tanh 2 100 4 0.446305 100 11.5098 99 -0.446305
objective_ed217cf2TERMINATED127.0.0.1:47162tanh -20.6075 10013.061 -1.98401 100 11.5623 99 1.98401
objective_ed2322beTERMINATED127.0.0.1:47163tanh 19.9564 10010.6836 2.0893 100 11.6053 99 -2.0893
objective_f3a0bef8TERMINATED127.0.0.1:47180tanh -7.43915 100 2.23969-0.312378 100 10.7378 99 0.312378
objective_f59fe9d6TERMINATED127.0.0.1:47185tanh -27.4958 100 5.5843 -2.57191 100 10.7145 99 2.57191
objective_f5aedf9aTERMINATED127.0.0.1:47188tanh 48.706 10019.7352 4.92153 100 10.7341 99 -4.92153
objective_f5b1ec08TERMINATED127.0.0.1:47189tanh 4.14098 10016.2739 0.475784 100 10.7034 99 -0.475784
objective_fb926d32TERMINATED127.0.0.1:47205tanh 44.778 10010.0724 4.57708 100 13.1109 99 -4.57708
objective_fd91e28eTERMINATED127.0.0.1:47214relu -2.9623 10011.8215 -0.211508 100 10.7934 99 0.211508


" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_eb74f122:\n", " date: 2022-07-22_15-31-08\n", " done: false\n", " experiment_id: 0ed1b6ba6d99477dba0632102d1bc531\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 10.2\n", " neg_mean_loss: -10.2\n", " node_ip: 127.0.0.1\n", " pid: 47156\n", " time_since_restore: 0.10458922386169434\n", " time_this_iter_s: 0.10458922386169434\n", " time_total_s: 0.10458922386169434\n", " timestamp: 1658500268\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: eb74f122\n", " warmup_time: 0.002730131149291992\n", " \n", "Result for objective_ed2010ec:\n", " date: 2022-07-22_15-31-11\n", " done: false\n", " experiment_id: 3acf6a8ccf8442a7adcf98cc92362216\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 10.2\n", " neg_mean_loss: -10.2\n", " node_ip: 127.0.0.1\n", " pid: 47161\n", " time_since_restore: 0.1025240421295166\n", " time_this_iter_s: 0.1025240421295166\n", " time_total_s: 0.1025240421295166\n", " timestamp: 1658500271\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: ed2010ec\n", " warmup_time: 0.0034351348876953125\n", " \n", "Result for objective_ed2322be:\n", " date: 2022-07-22_15-31-11\n", " done: false\n", " experiment_id: e95895ab00b54841933d324c3de8f58e\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 11.99564236159307\n", " neg_mean_loss: -11.99564236159307\n", " node_ip: 127.0.0.1\n", " pid: 47163\n", " time_since_restore: 0.10490107536315918\n", " time_this_iter_s: 0.10490107536315918\n", " time_total_s: 0.10490107536315918\n", " timestamp: 1658500271\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: ed2322be\n", " warmup_time: 0.0032410621643066406\n", " \n", "Result for objective_ed217cf2:\n", " date: 2022-07-22_15-31-11\n", " done: false\n", " experiment_id: 52186ede891e429aac44318036e7a7bb\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 7.939249801790311\n", " neg_mean_loss: -7.939249801790311\n", " node_ip: 127.0.0.1\n", " pid: 47162\n", " time_since_restore: 0.10419487953186035\n", " time_this_iter_s: 0.10419487953186035\n", " time_total_s: 0.10419487953186035\n", " timestamp: 1658500271\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: ed217cf2\n", " warmup_time: 0.0031850337982177734\n", " \n", "Result for objective_eb74f122:\n", " date: 2022-07-22_15-31-13\n", " done: false\n", " experiment_id: 0ed1b6ba6d99477dba0632102d1bc531\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 46\n", " iterations_since_restore: 47\n", " mean_loss: 1.9857142857142855\n", " neg_mean_loss: -1.9857142857142855\n", " node_ip: 127.0.0.1\n", " pid: 47156\n", " time_since_restore: 5.155240058898926\n", " time_this_iter_s: 0.11003398895263672\n", " time_total_s: 5.155240058898926\n", " timestamp: 1658500273\n", " timesteps_since_restore: 0\n", " training_iteration: 47\n", " trial_id: eb74f122\n", " warmup_time: 0.002730131149291992\n", " \n", "Result for objective_ed2010ec:\n", " date: 2022-07-22_15-31-16\n", " done: false\n", " experiment_id: 3acf6a8ccf8442a7adcf98cc92362216\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 0.7050505050505051\n", " neg_mean_loss: -0.7050505050505051\n", " node_ip: 127.0.0.1\n", " pid: 47161\n", " time_since_restore: 5.1636271476745605\n", " time_this_iter_s: 0.1071469783782959\n", " time_total_s: 5.1636271476745605\n", " timestamp: 1658500276\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: ed2010ec\n", " warmup_time: 0.0034351348876953125\n", " \n", "Result for objective_ed2322be:\n", " date: 2022-07-22_15-31-16\n", " done: false\n", " experiment_id: e95895ab00b54841933d324c3de8f58e\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 2.1909053484385796\n", " neg_mean_loss: -2.1909053484385796\n", " node_ip: 127.0.0.1\n", " pid: 47163\n", " time_since_restore: 5.168597936630249\n", " time_this_iter_s: 0.10674691200256348\n", " time_total_s: 5.168597936630249\n", " timestamp: 1658500276\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: ed2322be\n", " warmup_time: 0.0032410621643066406\n", " \n", "Result for objective_ed217cf2:\n", " date: 2022-07-22_15-31-16\n", " done: false\n", " experiment_id: 52186ede891e429aac44318036e7a7bb\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -1.9004596703789314\n", " neg_mean_loss: 1.9004596703789314\n", " node_ip: 127.0.0.1\n", " pid: 47162\n", " time_since_restore: 5.14047384262085\n", " time_this_iter_s: 0.10724306106567383\n", " time_total_s: 5.14047384262085\n", " timestamp: 1658500276\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: ed217cf2\n", " warmup_time: 0.0031850337982177734\n", " \n", "Result for objective_eb74f122:\n", " date: 2022-07-22_15-31-18\n", " done: false\n", " experiment_id: 0ed1b6ba6d99477dba0632102d1bc531\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " mean_loss: 1.170873786407767\n", " neg_mean_loss: -1.170873786407767\n", " node_ip: 127.0.0.1\n", " pid: 47156\n", " time_since_restore: 10.253305196762085\n", " time_this_iter_s: 0.10791301727294922\n", " time_total_s: 10.253305196762085\n", " timestamp: 1658500278\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: eb74f122\n", " warmup_time: 0.002730131149291992\n", " \n", "Result for objective_eb74f122:\n", " date: 2022-07-22_15-31-19\n", " done: true\n", " experiment_id: 0ed1b6ba6d99477dba0632102d1bc531\n", " experiment_tag: 1_activation=relu,height=2.0000,steps=100,width=1.0000\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 1.1174311926605505\n", " neg_mean_loss: -1.1174311926605505\n", " node_ip: 127.0.0.1\n", " pid: 47156\n", " time_since_restore: 10.900829076766968\n", " time_this_iter_s: 0.10532379150390625\n", " time_total_s: 10.900829076766968\n", " timestamp: 1658500279\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: eb74f122\n", " warmup_time: 0.002730131149291992\n", " \n", "Result for objective_ed217cf2:\n", " date: 2022-07-22_15-31-21\n", " done: false\n", " experiment_id: 52186ede891e429aac44318036e7a7bb\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 91\n", " iterations_since_restore: 92\n", " mean_loss: -1.9773161428398818\n", " neg_mean_loss: 1.9773161428398818\n", " node_ip: 127.0.0.1\n", " pid: 47162\n", " time_since_restore: 9.86834192276001\n", " time_this_iter_s: 0.10606908798217773\n", " time_total_s: 9.86834192276001\n", " timestamp: 1658500281\n", " timesteps_since_restore: 0\n", " training_iteration: 92\n", " trial_id: ed217cf2\n", " warmup_time: 0.0031850337982177734\n", " \n", "Result for objective_ed2322be:\n", " date: 2022-07-22_15-31-21\n", " done: false\n", " experiment_id: e95895ab00b54841933d324c3de8f58e\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 91\n", " iterations_since_restore: 92\n", " mean_loss: 2.0974537058269864\n", " neg_mean_loss: -2.0974537058269864\n", " node_ip: 127.0.0.1\n", " pid: 47163\n", " time_since_restore: 9.877221822738647\n", " time_this_iter_s: 0.10554194450378418\n", " time_total_s: 9.877221822738647\n", " timestamp: 1658500281\n", " timesteps_since_restore: 0\n", " training_iteration: 92\n", " trial_id: ed2322be\n", " warmup_time: 0.0032410621643066406\n", " \n", "Result for objective_ed2010ec:\n", " date: 2022-07-22_15-31-21\n", " done: false\n", " experiment_id: 3acf6a8ccf8442a7adcf98cc92362216\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 92\n", " iterations_since_restore: 93\n", " mean_loss: 0.46455026455026455\n", " neg_mean_loss: -0.46455026455026455\n", " node_ip: 127.0.0.1\n", " pid: 47161\n", " time_since_restore: 10.010878086090088\n", " time_this_iter_s: 0.12952017784118652\n", " time_total_s: 10.010878086090088\n", " timestamp: 1658500281\n", " timesteps_since_restore: 0\n", " training_iteration: 93\n", " trial_id: ed2010ec\n", " warmup_time: 0.0034351348876953125\n", " \n", "Result for objective_f3a0bef8:\n", " date: 2022-07-22_15-31-22\n", " done: false\n", " experiment_id: 7bbba7b85dbc4029bc8b22b7866d091f\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 9.256084922802092\n", " neg_mean_loss: -9.256084922802092\n", " node_ip: 127.0.0.1\n", " pid: 47180\n", " time_since_restore: 0.10452008247375488\n", " time_this_iter_s: 0.10452008247375488\n", " time_total_s: 0.10452008247375488\n", " timestamp: 1658500282\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: f3a0bef8\n", " warmup_time: 0.003498077392578125\n", " \n", "Result for objective_ed2010ec:\n", " date: 2022-07-22_15-31-22\n", " done: true\n", " experiment_id: 3acf6a8ccf8442a7adcf98cc92362216\n", " experiment_tag: 2_activation=tanh,height=2.0000,steps=100,width=4.0000\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 0.44630541871921187\n", " neg_mean_loss: -0.44630541871921187\n", " node_ip: 127.0.0.1\n", " pid: 47161\n", " time_since_restore: 11.509758949279785\n", " time_this_iter_s: 0.10628509521484375\n", " time_total_s: 11.509758949279785\n", " timestamp: 1658500282\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: ed2010ec\n", " warmup_time: 0.0034351348876953125\n", " \n", "Result for objective_ed217cf2:\n", " date: 2022-07-22_15-31-22\n", " done: true\n", " experiment_id: 52186ede891e429aac44318036e7a7bb\n", " experiment_tag: 3_activation=tanh,height=-20.6075,steps=100,width=13.0610\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -1.9840065470391304\n", " neg_mean_loss: 1.9840065470391304\n", " node_ip: 127.0.0.1\n", " pid: 47162\n", " time_since_restore: 11.562283754348755\n", " time_this_iter_s: 0.10557794570922852\n", " time_total_s: 11.562283754348755\n", " timestamp: 1658500282\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: ed217cf2\n", " warmup_time: 0.0031850337982177734\n", " \n", "Result for objective_ed2322be:\n", " date: 2022-07-22_15-31-22\n", " done: true\n", " experiment_id: e95895ab00b54841933d324c3de8f58e\n", " experiment_tag: 4_activation=tanh,height=19.9564,steps=100,width=10.6836\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 2.0893035832616653\n", " neg_mean_loss: -2.0893035832616653\n", " node_ip: 127.0.0.1\n", " pid: 47163\n", " time_since_restore: 11.605261087417603\n", " time_this_iter_s: 0.10673737525939941\n", " time_total_s: 11.605261087417603\n", " timestamp: 1658500282\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: ed2322be\n", " warmup_time: 0.0032410621643066406\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_f59fe9d6:\n", " date: 2022-07-22_15-31-25\n", " done: false\n", " experiment_id: efbda212c15c4bd38cfc5f39dfae8b73\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 7.250423019956081\n", " neg_mean_loss: -7.250423019956081\n", " node_ip: 127.0.0.1\n", " pid: 47185\n", " time_since_restore: 0.10512709617614746\n", " time_this_iter_s: 0.10512709617614746\n", " time_total_s: 0.10512709617614746\n", " timestamp: 1658500285\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: f59fe9d6\n", " warmup_time: 0.003314971923828125\n", " \n", "Result for objective_f5b1ec08:\n", " date: 2022-07-22_15-31-25\n", " done: false\n", " experiment_id: dadb542868ba4b10adf1ef161c75ea17\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 10.41409792482367\n", " neg_mean_loss: -10.41409792482367\n", " node_ip: 127.0.0.1\n", " pid: 47189\n", " time_since_restore: 0.10109281539916992\n", " time_this_iter_s: 0.10109281539916992\n", " time_total_s: 0.10109281539916992\n", " timestamp: 1658500285\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: f5b1ec08\n", " warmup_time: 0.0031630992889404297\n", " \n", "Result for objective_f5aedf9a:\n", " date: 2022-07-22_15-31-25\n", " done: false\n", " experiment_id: 9f7847b7440648cb86e73bf1be0ccc05\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 14.870604096990785\n", " neg_mean_loss: -14.870604096990785\n", " node_ip: 127.0.0.1\n", " pid: 47188\n", " time_since_restore: 0.10486412048339844\n", " time_this_iter_s: 0.10486412048339844\n", " time_total_s: 0.10486412048339844\n", " timestamp: 1658500285\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: f5aedf9a\n", " warmup_time: 0.002830028533935547\n", " \n", "Result for objective_f3a0bef8:\n", " date: 2022-07-22_15-31-27\n", " done: false\n", " experiment_id: 7bbba7b85dbc4029bc8b22b7866d091f\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 0.12364690126601663\n", " neg_mean_loss: -0.12364690126601663\n", " node_ip: 127.0.0.1\n", " pid: 47180\n", " time_since_restore: 5.149861097335815\n", " time_this_iter_s: 0.1075279712677002\n", " time_total_s: 5.149861097335815\n", " timestamp: 1658500287\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: f3a0bef8\n", " warmup_time: 0.003498077392578125\n", " \n", "Result for objective_f59fe9d6:\n", " date: 2022-07-22_15-31-30\n", " done: false\n", " experiment_id: efbda212c15c4bd38cfc5f39dfae8b73\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -2.3825537636804834\n", " neg_mean_loss: 2.3825537636804834\n", " node_ip: 127.0.0.1\n", " pid: 47185\n", " time_since_restore: 5.170727014541626\n", " time_this_iter_s: 0.10740494728088379\n", " time_total_s: 5.170727014541626\n", " timestamp: 1658500290\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: f59fe9d6\n", " warmup_time: 0.003314971923828125\n", " \n", "Result for objective_f5b1ec08:\n", " date: 2022-07-22_15-31-30\n", " done: false\n", " experiment_id: dadb542868ba4b10adf1ef161c75ea17\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 0.5431509247331814\n", " neg_mean_loss: -0.5431509247331814\n", " node_ip: 127.0.0.1\n", " pid: 47189\n", " time_since_restore: 5.153015613555908\n", " time_this_iter_s: 0.10737276077270508\n", " time_total_s: 5.153015613555908\n", " timestamp: 1658500290\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: f5b1ec08\n", " warmup_time: 0.0031630992889404297\n", " \n", "Result for objective_f5aedf9a:\n", " date: 2022-07-22_15-31-30\n", " done: false\n", " experiment_id: 9f7847b7440648cb86e73bf1be0ccc05\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 4.977264708542528\n", " neg_mean_loss: -4.977264708542528\n", " node_ip: 127.0.0.1\n", " pid: 47188\n", " time_since_restore: 5.183044195175171\n", " time_this_iter_s: 0.10844588279724121\n", " time_total_s: 5.183044195175171\n", " timestamp: 1658500290\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: f5aedf9a\n", " warmup_time: 0.002830028533935547\n", " \n", "Result for objective_f3a0bef8:\n", " date: 2022-07-22_15-31-32\n", " done: false\n", " experiment_id: 7bbba7b85dbc4029bc8b22b7866d091f\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -0.29046425326874115\n", " neg_mean_loss: 0.29046425326874115\n", " node_ip: 127.0.0.1\n", " pid: 47180\n", " time_since_restore: 10.200733184814453\n", " time_this_iter_s: 0.10860228538513184\n", " time_total_s: 10.200733184814453\n", " timestamp: 1658500292\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: f3a0bef8\n", " warmup_time: 0.003498077392578125\n", " \n", "Result for objective_f3a0bef8:\n", " date: 2022-07-22_15-31-32\n", " done: true\n", " experiment_id: 7bbba7b85dbc4029bc8b22b7866d091f\n", " experiment_tag: 5_activation=tanh,height=-7.4392,steps=100,width=2.2397\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -0.3123775218508783\n", " neg_mean_loss: 0.3123775218508783\n", " node_ip: 127.0.0.1\n", " pid: 47180\n", " time_since_restore: 10.737780094146729\n", " time_this_iter_s: 0.10722112655639648\n", " time_total_s: 10.737780094146729\n", " timestamp: 1658500292\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: f3a0bef8\n", " warmup_time: 0.003498077392578125\n", " \n", "Result for objective_fb926d32:\n", " date: 2022-07-22_15-31-35\n", " done: false\n", " experiment_id: c6af4463441e4a3a8ca1298a043e6151\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 14.477795797697073\n", " neg_mean_loss: -14.477795797697073\n", " node_ip: 127.0.0.1\n", " pid: 47205\n", " time_since_restore: 0.10373091697692871\n", " time_this_iter_s: 0.10373091697692871\n", " time_total_s: 0.10373091697692871\n", " timestamp: 1658500295\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: fb926d32\n", " warmup_time: 0.0029900074005126953\n", " \n", "Result for objective_f59fe9d6:\n", " date: 2022-07-22_15-31-35\n", " done: false\n", " experiment_id: efbda212c15c4bd38cfc5f39dfae8b73\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -2.5626347652141552\n", " neg_mean_loss: 2.5626347652141552\n", " node_ip: 127.0.0.1\n", " pid: 47185\n", " time_since_restore: 10.179347038269043\n", " time_this_iter_s: 0.1074531078338623\n", " time_total_s: 10.179347038269043\n", " timestamp: 1658500295\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: f59fe9d6\n", " warmup_time: 0.003314971923828125\n", " \n", "Result for objective_f5b1ec08:\n", " date: 2022-07-22_15-31-35\n", " done: false\n", " experiment_id: dadb542868ba4b10adf1ef161c75ea17\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 0.4790434958168009\n", " neg_mean_loss: -0.4790434958168009\n", " node_ip: 127.0.0.1\n", " pid: 47189\n", " time_since_restore: 10.170033931732178\n", " time_this_iter_s: 0.10643315315246582\n", " time_total_s: 10.170033931732178\n", " timestamp: 1658500295\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: f5b1ec08\n", " warmup_time: 0.0031630992889404297\n", " \n", "Result for objective_f5aedf9a:\n", " date: 2022-07-22_15-31-35\n", " done: false\n", " experiment_id: 9f7847b7440648cb86e73bf1be0ccc05\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 4.924220339829169\n", " neg_mean_loss: -4.924220339829169\n", " node_ip: 127.0.0.1\n", " pid: 47188\n", " time_since_restore: 10.189186096191406\n", " time_this_iter_s: 0.1078641414642334\n", " time_total_s: 10.189186096191406\n", " timestamp: 1658500295\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: f5aedf9a\n", " warmup_time: 0.002830028533935547\n", " \n", "Result for objective_f59fe9d6:\n", " date: 2022-07-22_15-31-36\n", " done: true\n", " experiment_id: efbda212c15c4bd38cfc5f39dfae8b73\n", " experiment_tag: 6_activation=tanh,height=-27.4958,steps=100,width=5.5843\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -2.5719085451008423\n", " neg_mean_loss: 2.5719085451008423\n", " node_ip: 127.0.0.1\n", " pid: 47185\n", " time_since_restore: 10.714513063430786\n", " time_this_iter_s: 0.1071622371673584\n", " time_total_s: 10.714513063430786\n", " timestamp: 1658500296\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: f59fe9d6\n", " warmup_time: 0.003314971923828125\n", " \n", "Result for objective_f5b1ec08:\n", " date: 2022-07-22_15-31-36\n", " done: true\n", " experiment_id: dadb542868ba4b10adf1ef161c75ea17\n", " experiment_tag: 8_activation=tanh,height=4.1410,steps=100,width=16.2739\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 0.4757836498809659\n", " neg_mean_loss: -0.4757836498809659\n", " node_ip: 127.0.0.1\n", " pid: 47189\n", " time_since_restore: 10.703420877456665\n", " time_this_iter_s: 0.10684394836425781\n", " time_total_s: 10.703420877456665\n", " timestamp: 1658500296\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: f5b1ec08\n", " warmup_time: 0.0031630992889404297\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_f5aedf9a:\n", " date: 2022-07-22_15-31-36\n", " done: true\n", " experiment_id: 9f7847b7440648cb86e73bf1be0ccc05\n", " experiment_tag: 7_activation=tanh,height=48.7060,steps=100,width=19.7352\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 4.9215262379377105\n", " neg_mean_loss: -4.9215262379377105\n", " node_ip: 127.0.0.1\n", " pid: 47188\n", " time_since_restore: 10.734119176864624\n", " time_this_iter_s: 0.11568498611450195\n", " time_total_s: 10.734119176864624\n", " timestamp: 1658500296\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: f5aedf9a\n", " warmup_time: 0.002830028533935547\n", " \n", "Result for objective_fd91e28e:\n", " date: 2022-07-22_15-31-38\n", " done: false\n", " experiment_id: 5c3693eeb55b476088ce1de9127c5a39\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 9.70376987633604\n", " neg_mean_loss: -9.70376987633604\n", " node_ip: 127.0.0.1\n", " pid: 47214\n", " time_since_restore: 0.10451984405517578\n", " time_this_iter_s: 0.10451984405517578\n", " time_total_s: 0.10451984405517578\n", " timestamp: 1658500298\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: fd91e28e\n", " warmup_time: 0.00302886962890625\n", " \n", "Result for objective_fb926d32:\n", " date: 2022-07-22_15-31-40\n", " done: false\n", " experiment_id: c6af4463441e4a3a8ca1298a043e6151\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 25\n", " iterations_since_restore: 26\n", " mean_loss: 4.859752718513368\n", " neg_mean_loss: -4.859752718513368\n", " node_ip: 127.0.0.1\n", " pid: 47205\n", " time_since_restore: 5.151448011398315\n", " time_this_iter_s: 0.11450004577636719\n", " time_total_s: 5.151448011398315\n", " timestamp: 1658500300\n", " timesteps_since_restore: 0\n", " training_iteration: 26\n", " trial_id: fb926d32\n", " warmup_time: 0.0029900074005126953\n", " \n", "Result for objective_fd91e28e:\n", " date: 2022-07-22_15-31-43\n", " done: false\n", " experiment_id: 5c3693eeb55b476088ce1de9127c5a39\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 46\n", " iterations_since_restore: 47\n", " mean_loss: -0.11565561363589152\n", " neg_mean_loss: 0.11565561363589152\n", " node_ip: 127.0.0.1\n", " pid: 47214\n", " time_since_restore: 5.126538991928101\n", " time_this_iter_s: 0.10815834999084473\n", " time_total_s: 5.126538991928101\n", " timestamp: 1658500303\n", " timesteps_since_restore: 0\n", " training_iteration: 47\n", " trial_id: fd91e28e\n", " warmup_time: 0.00302886962890625\n", " \n", "Result for objective_fb926d32:\n", " date: 2022-07-22_15-31-45\n", " done: false\n", " experiment_id: c6af4463441e4a3a8ca1298a043e6151\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 72\n", " iterations_since_restore: 73\n", " mean_loss: 4.613811037170093\n", " neg_mean_loss: -4.613811037170093\n", " node_ip: 127.0.0.1\n", " pid: 47205\n", " time_since_restore: 10.208579063415527\n", " time_this_iter_s: 0.10763001441955566\n", " time_total_s: 10.208579063415527\n", " timestamp: 1658500305\n", " timesteps_since_restore: 0\n", " training_iteration: 73\n", " trial_id: fb926d32\n", " warmup_time: 0.0029900074005126953\n", " \n", "Result for objective_fb926d32:\n", " date: 2022-07-22_15-31-48\n", " done: true\n", " experiment_id: c6af4463441e4a3a8ca1298a043e6151\n", " experiment_tag: 9_activation=tanh,height=44.7780,steps=100,width=10.0724\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 4.577084283144497\n", " neg_mean_loss: -4.577084283144497\n", " node_ip: 127.0.0.1\n", " pid: 47205\n", " time_since_restore: 13.110869884490967\n", " time_this_iter_s: 0.10806989669799805\n", " time_total_s: 13.110869884490967\n", " timestamp: 1658500308\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: fb926d32\n", " warmup_time: 0.0029900074005126953\n", " \n", "Result for objective_fd91e28e:\n", " date: 2022-07-22_15-31-48\n", " done: false\n", " experiment_id: 5c3693eeb55b476088ce1de9127c5a39\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " mean_loss: -0.20609110794676003\n", " neg_mean_loss: 0.20609110794676003\n", " node_ip: 127.0.0.1\n", " pid: 47214\n", " time_since_restore: 10.14166784286499\n", " time_this_iter_s: 0.10436320304870605\n", " time_total_s: 10.14166784286499\n", " timestamp: 1658500308\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: fd91e28e\n", " warmup_time: 0.00302886962890625\n", " \n", "Result for objective_fd91e28e:\n", " date: 2022-07-22_15-31-49\n", " done: true\n", " experiment_id: 5c3693eeb55b476088ce1de9127c5a39\n", " experiment_tag: 10_activation=relu,height=-2.9623,steps=100,width=11.8215\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -0.21150779503682235\n", " neg_mean_loss: 0.21150779503682235\n", " node_ip: 127.0.0.1\n", " pid: 47214\n", " time_since_restore: 10.793406963348389\n", " time_this_iter_s: 0.10776925086975098\n", " time_total_s: 10.793406963348389\n", " timestamp: 1658500309\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: fd91e28e\n", " warmup_time: 0.00302886962890625\n", " \n" ] } ], "source": [ "tuner = tune.Tuner(\n", " objective,\n", " tune_config=tune.TuneConfig(\n", " metric=\"mean_loss\",\n", " mode=\"min\",\n", " search_alg=algo,\n", " num_samples=num_samples,\n", " ),\n", " param_space=search_config,\n", ")\n", "results = tuner.fit()" ] }, { "attachments": {}, "cell_type": "markdown", "id": "49be6f01", "metadata": {}, "source": [ "Here are the hyperparamters found to minimize the mean loss of the defined objective." ] }, { "cell_type": "code", "execution_count": 11, "id": "7036798c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best hyperparameters found were: {'steps': 100, 'width': 5.584304853357766, 'height': -27.49576980043919, 'activation': 'tanh'}\n" ] } ], "source": [ "print(\"Best hyperparameters found were: \", results.get_best_result().config)" ] }, { "attachments": {}, "cell_type": "markdown", "id": "504e9d2a", "metadata": {}, "source": [ "## Conditional search spaces\n", "\n", "Sometimes we may want to build a more complicated search space that has conditional dependencies on other hyperparameters. In this case, we pass a nested dictionary to `objective_two`, which has been slightly adjusted from `objective` to deal with the conditional search space." ] }, { "cell_type": "code", "execution_count": 12, "id": "2f7b5449", "metadata": {}, "outputs": [], "source": [ "def evaluation_fn(step, width, height, mult=1):\n", " return (0.1 + width * step / 100) ** (-1) + height * 0.1 * mult" ] }, { "cell_type": "code", "execution_count": 13, "id": "4b83b81c", "metadata": {}, "outputs": [], "source": [ "def objective_two(config):\n", " width, height = config[\"width\"], config[\"height\"]\n", " sub_dict = config[\"activation\"]\n", " mult = sub_dict.get(\"mult\", 1)\n", " \n", " for step in range(config[\"steps\"]):\n", " intermediate_score = evaluation_fn(step, width, height, mult)\n", " train.report({\"iterations\": step, \"mean_loss\": intermediate_score})\n", " time.sleep(0.1)" ] }, { "cell_type": "code", "execution_count": 14, "id": "75cea99e", "metadata": {}, "outputs": [], "source": [ "conditional_space = {\n", " \"activation\": hp.choice(\n", " \"activation\",\n", " [\n", " {\"activation\": \"relu\", \"mult\": hp.uniform(\"mult\", 1, 2)},\n", " {\"activation\": \"tanh\"},\n", " ],\n", " ),\n", " \"width\": hp.uniform(\"width\", 0, 20),\n", " \"height\": hp.uniform(\"height\", -100, 100),\n", " \"steps\": 100,\n", "}" ] }, { "attachments": {}, "cell_type": "markdown", "id": "7df282c1", "metadata": {}, "source": [ "Now we the define the search algorithm built from `HyperOptSearch` constrained by `ConcurrencyLimiter`. When the hyperparameter search space is conditional, we pass it (`conditional_space`) into `HyperOptSearch`." ] }, { "cell_type": "code", "execution_count": 15, "id": "ea2c71a6", "metadata": {}, "outputs": [], "source": [ "algo = HyperOptSearch(space=conditional_space, metric=\"mean_loss\", mode=\"min\")\n", "algo = ConcurrencyLimiter(algo, max_concurrent=4)" ] }, { "attachments": {}, "cell_type": "markdown", "id": "630f84ab", "metadata": {}, "source": [ "Now we run the experiment, this time with an empty `config` because we instead provided `space` to the `HyperOptSearch` `search_alg`." ] }, { "cell_type": "code", "execution_count": 16, "id": "14111e9e", "metadata": {}, "outputs": [ { "data": { "text/html": [ "== Status ==
Current time: 2022-07-22 15:32:33 (running for 00:00:44.21)
Memory usage on this node: 10.7/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.29 GiB heap, 0.0/2.0 GiB objects
Current best trial: 0de7d38c with mean_loss=-9.200364093875208 and parameters={'activation': {'activation': 'relu', 'mult': 1.3982639549501585}, 'height': -66.3136247260571, 'steps': 100, 'width': 13.922128223483856}
Result logdir: /Users/kai/ray_results/objective_two_2022-07-22_15-31-49
Number of trials: 10/10 (10 TERMINATED)
\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Trial name status loc activation/activa... activation/mult height steps width loss iter total time (s) iterations neg_mean_loss
objective_two_05a0ad52TERMINATED127.0.0.1:47229tanh -63.0091 10010.1954 -6.20281 100 10.8547 99 6.20281
objective_two_075f0d78TERMINATED127.0.0.1:47236relu 1.22098 46.4977 10013.6093 5.75095 100 12.0174 99 -5.75095
objective_two_07617f54TERMINATED127.0.0.1:47237tanh -41.2706 100 7.34134 -3.99133 100 11.8677 99 3.99133
objective_two_07636cd8TERMINATED127.0.0.1:47238tanh 49.0313 100 0.828826 5.98945 100 12.0258 99 -5.98945
objective_two_0de7d38cTERMINATED127.0.0.1:47256relu 1.39826-66.3136 10013.9221 -9.20036 100 10.7072 99 9.20036
objective_two_100ebe3cTERMINATED127.0.0.1:47265tanh -38.8555 10015.9966 -3.82281 100 10.6571 99 3.82281
objective_two_101e702aTERMINATED127.0.0.1:47268relu 1.51022 5.84901 100 0.431039 2.78184 100 10.7202 99 -2.78184
objective_two_1022212aTERMINATED127.0.0.1:47269relu 1.32257 71.2885 10011.4466 9.51591 100 10.7072 99 -9.51591
objective_two_15fbf8dcTERMINATED127.0.0.1:47288relu 1.22166-32.1584 100 9.43222 -3.82272 100 10.5991 99 3.82272
objective_two_18090732TERMINATED127.0.0.1:47302tanh 52.7613 10016.7268 5.33616 100 10.6336 99 -5.33616


" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_two_05a0ad52:\n", " date: 2022-07-22_15-31-52\n", " done: false\n", " experiment_id: af7041e0b2c947aa8a30c2e30f94ba83\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 3.699090470918877\n", " neg_mean_loss: -3.699090470918877\n", " node_ip: 127.0.0.1\n", " pid: 47229\n", " time_since_restore: 0.00011491775512695312\n", " time_this_iter_s: 0.00011491775512695312\n", " time_total_s: 0.00011491775512695312\n", " timestamp: 1658500312\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 05a0ad52\n", " warmup_time: 0.0027589797973632812\n", " \n", "Result for objective_two_07636cd8:\n", " date: 2022-07-22_15-31-55\n", " done: false\n", " experiment_id: 5c9d75b036ae410b9fdea303f14fe4cd\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 14.903126957374846\n", " neg_mean_loss: -14.903126957374846\n", " node_ip: 127.0.0.1\n", " pid: 47238\n", " time_since_restore: 0.00011110305786132812\n", " time_this_iter_s: 0.00011110305786132812\n", " time_total_s: 0.00011110305786132812\n", " timestamp: 1658500315\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 07636cd8\n", " warmup_time: 0.0037419795989990234\n", " \n", "Result for objective_two_07617f54:\n", " date: 2022-07-22_15-31-55\n", " done: false\n", " experiment_id: 32155cb3b7f04dc3b7b5b679eecd8c46\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 5.872941741395985\n", " neg_mean_loss: -5.872941741395985\n", " node_ip: 127.0.0.1\n", " pid: 47237\n", " time_since_restore: 0.00015211105346679688\n", " time_this_iter_s: 0.00015211105346679688\n", " time_total_s: 0.00015211105346679688\n", " timestamp: 1658500315\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 07617f54\n", " warmup_time: 0.00436091423034668\n", " \n", "Result for objective_two_075f0d78:\n", " date: 2022-07-22_15-31-55\n", " done: false\n", " experiment_id: 5a46a57ef1164628bac1618e39078e0e\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 15.677277832165048\n", " neg_mean_loss: -15.677277832165048\n", " node_ip: 127.0.0.1\n", " pid: 47236\n", " time_since_restore: 0.00011396408081054688\n", " time_this_iter_s: 0.00011396408081054688\n", " time_total_s: 0.00011396408081054688\n", " timestamp: 1658500315\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 075f0d78\n", " warmup_time: 0.0035758018493652344\n", " \n", "Result for objective_two_05a0ad52:\n", " date: 2022-07-22_15-31-57\n", " done: false\n", " experiment_id: af7041e0b2c947aa8a30c2e30f94ba83\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 45\n", " iterations_since_restore: 46\n", " mean_loss: -6.087595964816368\n", " neg_mean_loss: 6.087595964816368\n", " node_ip: 127.0.0.1\n", " pid: 47229\n", " time_since_restore: 5.055715799331665\n", " time_this_iter_s: 0.10677886009216309\n", " time_total_s: 5.055715799331665\n", " timestamp: 1658500317\n", " timesteps_since_restore: 0\n", " training_iteration: 46\n", " trial_id: 05a0ad52\n", " warmup_time: 0.0027589797973632812\n", " \n", "Result for objective_two_07617f54:\n", " date: 2022-07-22_15-32-00\n", " done: false\n", " experiment_id: 32155cb3b7f04dc3b7b5b679eecd8c46\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -3.8454022145530584\n", " neg_mean_loss: 3.8454022145530584\n", " node_ip: 127.0.0.1\n", " pid: 47237\n", " time_since_restore: 5.048717021942139\n", " time_this_iter_s: 0.10686898231506348\n", " time_total_s: 5.048717021942139\n", " timestamp: 1658500320\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 07617f54\n", " warmup_time: 0.00436091423034668\n", " \n", "Result for objective_two_07636cd8:\n", " date: 2022-07-22_15-32-00\n", " done: false\n", " experiment_id: 5c9d75b036ae410b9fdea303f14fe4cd\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 6.9458266379188185\n", " neg_mean_loss: -6.9458266379188185\n", " node_ip: 127.0.0.1\n", " pid: 47238\n", " time_since_restore: 5.097726821899414\n", " time_this_iter_s: 0.1087799072265625\n", " time_total_s: 5.097726821899414\n", " timestamp: 1658500320\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 07636cd8\n", " warmup_time: 0.0037419795989990234\n", " \n", "Result for objective_two_075f0d78:\n", " date: 2022-07-22_15-32-00\n", " done: false\n", " experiment_id: 5a46a57ef1164628bac1618e39078e0e\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 5.83121027007688\n", " neg_mean_loss: -5.83121027007688\n", " node_ip: 127.0.0.1\n", " pid: 47236\n", " time_since_restore: 5.118211269378662\n", " time_this_iter_s: 0.10918712615966797\n", " time_total_s: 5.118211269378662\n", " timestamp: 1658500320\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 075f0d78\n", " warmup_time: 0.0035758018493652344\n", " \n", "Result for objective_two_05a0ad52:\n", " date: 2022-07-22_15-32-02\n", " done: false\n", " experiment_id: af7041e0b2c947aa8a30c2e30f94ba83\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 92\n", " iterations_since_restore: 93\n", " mean_loss: -6.195421815991769\n", " neg_mean_loss: 6.195421815991769\n", " node_ip: 127.0.0.1\n", " pid: 47229\n", " time_since_restore: 10.10265588760376\n", " time_this_iter_s: 0.10532593727111816\n", " time_total_s: 10.10265588760376\n", " timestamp: 1658500322\n", " timesteps_since_restore: 0\n", " training_iteration: 93\n", " trial_id: 05a0ad52\n", " warmup_time: 0.0027589797973632812\n", " \n", "Result for objective_two_05a0ad52:\n", " date: 2022-07-22_15-32-03\n", " done: true\n", " experiment_id: af7041e0b2c947aa8a30c2e30f94ba83\n", " experiment_tag: 1_activation=tanh,height=-63.0091,steps=100,width=10.1954\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -6.202807371456877\n", " neg_mean_loss: 6.202807371456877\n", " node_ip: 127.0.0.1\n", " pid: 47229\n", " time_since_restore: 10.854680061340332\n", " time_this_iter_s: 0.10699224472045898\n", " time_total_s: 10.854680061340332\n", " timestamp: 1658500323\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 05a0ad52\n", " warmup_time: 0.0027589797973632812\n", " \n", "Result for objective_two_075f0d78:\n", " date: 2022-07-22_15-32-04\n", " done: false\n", " experiment_id: 5a46a57ef1164628bac1618e39078e0e\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 91\n", " iterations_since_restore: 92\n", " mean_loss: 5.757377572321986\n", " neg_mean_loss: -5.757377572321986\n", " node_ip: 127.0.0.1\n", " pid: 47236\n", " time_since_restore: 9.820771932601929\n", " time_this_iter_s: 0.10582685470581055\n", " time_total_s: 9.820771932601929\n", " timestamp: 1658500324\n", " timesteps_since_restore: 0\n", " training_iteration: 92\n", " trial_id: 075f0d78\n", " warmup_time: 0.0035758018493652344\n", " \n", "Result for objective_two_07617f54:\n", " date: 2022-07-22_15-32-04\n", " done: false\n", " experiment_id: 32155cb3b7f04dc3b7b5b679eecd8c46\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 92\n", " iterations_since_restore: 93\n", " mean_loss: -3.981158750752472\n", " neg_mean_loss: 3.981158750752472\n", " node_ip: 127.0.0.1\n", " pid: 47237\n", " time_since_restore: 9.865068197250366\n", " time_this_iter_s: 0.1065821647644043\n", " time_total_s: 9.865068197250366\n", " timestamp: 1658500324\n", " timesteps_since_restore: 0\n", " training_iteration: 93\n", " trial_id: 07617f54\n", " warmup_time: 0.00436091423034668\n", " \n", "Result for objective_two_0de7d38c:\n", " date: 2022-07-22_15-32-06\n", " done: false\n", " experiment_id: 81d62307312a447e97c5d1f1cdad2687\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 0.7276048823462773\n", " neg_mean_loss: -0.7276048823462773\n", " node_ip: 127.0.0.1\n", " pid: 47256\n", " time_since_restore: 0.00012803077697753906\n", " time_this_iter_s: 0.00012803077697753906\n", " time_total_s: 0.00012803077697753906\n", " timestamp: 1658500326\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 0de7d38c\n", " warmup_time: 0.0027358531951904297\n", " \n", "Result for objective_two_07636cd8:\n", " date: 2022-07-22_15-32-04\n", " done: false\n", " experiment_id: 5c9d75b036ae410b9fdea303f14fe4cd\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 91\n", " iterations_since_restore: 92\n", " mean_loss: 6.073769581173668\n", " neg_mean_loss: -6.073769581173668\n", " node_ip: 127.0.0.1\n", " pid: 47238\n", " time_since_restore: 9.813458919525146\n", " time_this_iter_s: 0.10477709770202637\n", " time_total_s: 9.813458919525146\n", " timestamp: 1658500324\n", " timesteps_since_restore: 0\n", " training_iteration: 92\n", " trial_id: 07636cd8\n", " warmup_time: 0.0037419795989990234\n", " \n", "Result for objective_two_07617f54:\n", " date: 2022-07-22_15-32-06\n", " done: true\n", " experiment_id: 32155cb3b7f04dc3b7b5b679eecd8c46\n", " experiment_tag: 3_activation=tanh,height=-41.2706,steps=100,width=7.3413\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -3.9913348635947523\n", " neg_mean_loss: 3.9913348635947523\n", " node_ip: 127.0.0.1\n", " pid: 47237\n", " time_since_restore: 11.867748260498047\n", " time_this_iter_s: 0.10774397850036621\n", " time_total_s: 11.867748260498047\n", " timestamp: 1658500326\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 07617f54\n", " warmup_time: 0.00436091423034668\n", " \n", "Result for objective_two_075f0d78:\n", " date: 2022-07-22_15-32-07\n", " done: true\n", " experiment_id: 5a46a57ef1164628bac1618e39078e0e\n", " experiment_tag: 2_activation=relu,mult=1.2210,height=46.4977,steps=100,width=13.6093\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 5.7509525535298085\n", " neg_mean_loss: -5.7509525535298085\n", " node_ip: 127.0.0.1\n", " pid: 47236\n", " time_since_restore: 12.017354011535645\n", " time_this_iter_s: 0.10736513137817383\n", " time_total_s: 12.017354011535645\n", " timestamp: 1658500327\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 075f0d78\n", " warmup_time: 0.0035758018493652344\n", " \n", "Result for objective_two_07636cd8:\n", " date: 2022-07-22_15-32-07\n", " done: true\n", " experiment_id: 5c9d75b036ae410b9fdea303f14fe4cd\n", " experiment_tag: 4_activation=tanh,height=49.0313,steps=100,width=0.8288\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 5.989448515159168\n", " neg_mean_loss: -5.989448515159168\n", " node_ip: 127.0.0.1\n", " pid: 47238\n", " time_since_restore: 12.025846004486084\n", " time_this_iter_s: 0.10526180267333984\n", " time_total_s: 12.025846004486084\n", " timestamp: 1658500327\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 07636cd8\n", " warmup_time: 0.0037419795989990234\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_two_100ebe3c:\n", " date: 2022-07-22_15-32-09\n", " done: false\n", " experiment_id: f31e00c7c87c4884bef04183585473d1\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 6.11444579492732\n", " neg_mean_loss: -6.11444579492732\n", " node_ip: 127.0.0.1\n", " pid: 47265\n", " time_since_restore: 0.00012183189392089844\n", " time_this_iter_s: 0.00012183189392089844\n", " time_total_s: 0.00012183189392089844\n", " timestamp: 1658500329\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 100ebe3c\n", " warmup_time: 0.002948284149169922\n", " \n", "Result for objective_two_101e702a:\n", " date: 2022-07-22_15-32-09\n", " done: false\n", " experiment_id: 8c013d0ff6434dfe9719fa317f95feb8\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 10.883328850459264\n", " neg_mean_loss: -10.883328850459264\n", " node_ip: 127.0.0.1\n", " pid: 47268\n", " time_since_restore: 0.00011587142944335938\n", " time_this_iter_s: 0.00011587142944335938\n", " time_total_s: 0.00011587142944335938\n", " timestamp: 1658500329\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 101e702a\n", " warmup_time: 0.003309965133666992\n", " \n", "Result for objective_two_1022212a:\n", " date: 2022-07-22_15-32-09\n", " done: false\n", " experiment_id: c3c0bdb8b68146c68f2c34e08fa2f184\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 19.42843735686162\n", " neg_mean_loss: -19.42843735686162\n", " node_ip: 127.0.0.1\n", " pid: 47269\n", " time_since_restore: 0.00011324882507324219\n", " time_this_iter_s: 0.00011324882507324219\n", " time_total_s: 0.00011324882507324219\n", " timestamp: 1658500329\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 1022212a\n", " warmup_time: 0.0029020309448242188\n", " \n", "Result for objective_two_0de7d38c:\n", " date: 2022-07-22_15-32-11\n", " done: false\n", " experiment_id: 81d62307312a447e97c5d1f1cdad2687\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -9.121869790245293\n", " neg_mean_loss: 9.121869790245293\n", " node_ip: 127.0.0.1\n", " pid: 47256\n", " time_since_restore: 5.047628164291382\n", " time_this_iter_s: 0.10473132133483887\n", " time_total_s: 5.047628164291382\n", " timestamp: 1658500331\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 0de7d38c\n", " warmup_time: 0.0027358531951904297\n", " \n", "Result for objective_two_100ebe3c:\n", " date: 2022-07-22_15-32-14\n", " done: false\n", " experiment_id: f31e00c7c87c4884bef04183585473d1\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -3.7542934113649324\n", " neg_mean_loss: 3.7542934113649324\n", " node_ip: 127.0.0.1\n", " pid: 47265\n", " time_since_restore: 5.057713985443115\n", " time_this_iter_s: 0.10337996482849121\n", " time_total_s: 5.057713985443115\n", " timestamp: 1658500334\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 100ebe3c\n", " warmup_time: 0.002948284149169922\n", " \n", "Result for objective_two_101e702a:\n", " date: 2022-07-22_15-32-14\n", " done: false\n", " experiment_id: 8c013d0ff6434dfe9719fa317f95feb8\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 4.188150411466976\n", " neg_mean_loss: -4.188150411466976\n", " node_ip: 127.0.0.1\n", " pid: 47268\n", " time_since_restore: 5.101155757904053\n", " time_this_iter_s: 0.10826706886291504\n", " time_total_s: 5.101155757904053\n", " timestamp: 1658500334\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 101e702a\n", " warmup_time: 0.003309965133666992\n", " \n", "Result for objective_two_1022212a:\n", " date: 2022-07-22_15-32-14\n", " done: false\n", " experiment_id: c3c0bdb8b68146c68f2c34e08fa2f184\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 9.610922254324745\n", " neg_mean_loss: -9.610922254324745\n", " node_ip: 127.0.0.1\n", " pid: 47269\n", " time_since_restore: 5.09367823600769\n", " time_this_iter_s: 0.11145401000976562\n", " time_total_s: 5.09367823600769\n", " timestamp: 1658500334\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 1022212a\n", " warmup_time: 0.0029020309448242188\n", " \n", "Result for objective_two_0de7d38c:\n", " date: 2022-07-22_15-32-16\n", " done: false\n", " experiment_id: 81d62307312a447e97c5d1f1cdad2687\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 93\n", " iterations_since_restore: 94\n", " mean_loss: -9.195752548100788\n", " neg_mean_loss: 9.195752548100788\n", " node_ip: 127.0.0.1\n", " pid: 47256\n", " time_since_restore: 10.068511009216309\n", " time_this_iter_s: 0.10715794563293457\n", " time_total_s: 10.068511009216309\n", " timestamp: 1658500336\n", " timesteps_since_restore: 0\n", " training_iteration: 94\n", " trial_id: 0de7d38c\n", " warmup_time: 0.0027358531951904297\n", " \n", "Result for objective_two_0de7d38c:\n", " date: 2022-07-22_15-32-16\n", " done: true\n", " experiment_id: 81d62307312a447e97c5d1f1cdad2687\n", " experiment_tag: 5_activation=relu,mult=1.3983,height=-66.3136,steps=100,width=13.9221\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -9.200364093875208\n", " neg_mean_loss: 9.200364093875208\n", " node_ip: 127.0.0.1\n", " pid: 47256\n", " time_since_restore: 10.707159996032715\n", " time_this_iter_s: 0.1063990592956543\n", " time_total_s: 10.707159996032715\n", " timestamp: 1658500336\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 0de7d38c\n", " warmup_time: 0.0027358531951904297\n", " \n", "Result for objective_two_100ebe3c:\n", " date: 2022-07-22_15-32-19\n", " done: false\n", " experiment_id: f31e00c7c87c4884bef04183585473d1\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -3.8194902277136236\n", " neg_mean_loss: 3.8194902277136236\n", " node_ip: 127.0.0.1\n", " pid: 47265\n", " time_since_restore: 10.118366956710815\n", " time_this_iter_s: 0.10590505599975586\n", " time_total_s: 10.118366956710815\n", " timestamp: 1658500339\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 100ebe3c\n", " warmup_time: 0.002948284149169922\n", " \n", "Result for objective_two_15fbf8dc:\n", " date: 2022-07-22_15-32-19\n", " done: false\n", " experiment_id: 825ceb15d1134810ba19f09bb90c2b35\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 6.071327317808535\n", " neg_mean_loss: -6.071327317808535\n", " node_ip: 127.0.0.1\n", " pid: 47288\n", " time_since_restore: 0.0002067089080810547\n", " time_this_iter_s: 0.0002067089080810547\n", " time_total_s: 0.0002067089080810547\n", " timestamp: 1658500339\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: 15fbf8dc\n", " warmup_time: 0.005033969879150391\n", " \n", "Result for objective_two_101e702a:\n", " date: 2022-07-22_15-32-19\n", " done: false\n", " experiment_id: 8c013d0ff6434dfe9719fa317f95feb8\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 2.86283543260466\n", " neg_mean_loss: -2.86283543260466\n", " node_ip: 127.0.0.1\n", " pid: 47268\n", " time_since_restore: 10.181840896606445\n", " time_this_iter_s: 0.10625004768371582\n", " time_total_s: 10.181840896606445\n", " timestamp: 1658500339\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 101e702a\n", " warmup_time: 0.003309965133666992\n", " \n", "Result for objective_two_1022212a:\n", " date: 2022-07-22_15-32-19\n", " done: false\n", " experiment_id: c3c0bdb8b68146c68f2c34e08fa2f184\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 9.520519990087298\n", " neg_mean_loss: -9.520519990087298\n", " node_ip: 127.0.0.1\n", " pid: 47269\n", " time_since_restore: 10.166353225708008\n", " time_this_iter_s: 0.10583710670471191\n", " time_total_s: 10.166353225708008\n", " timestamp: 1658500339\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 1022212a\n", " warmup_time: 0.0029020309448242188\n", " \n", "Result for objective_two_100ebe3c:\n", " date: 2022-07-22_15-32-20\n", " done: true\n", " experiment_id: f31e00c7c87c4884bef04183585473d1\n", " experiment_tag: 6_activation=tanh,height=-38.8555,steps=100,width=15.9966\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -3.8228058558351754\n", " neg_mean_loss: 3.8228058558351754\n", " node_ip: 127.0.0.1\n", " pid: 47265\n", " time_since_restore: 10.657065153121948\n", " time_this_iter_s: 0.10721731185913086\n", " time_total_s: 10.657065153121948\n", " timestamp: 1658500340\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 100ebe3c\n", " warmup_time: 0.002948284149169922\n", " \n", "Result for objective_two_101e702a:\n", " date: 2022-07-22_15-32-20\n", " done: true\n", " experiment_id: 8c013d0ff6434dfe9719fa317f95feb8\n", " experiment_tag: 7_activation=relu,mult=1.5102,height=5.8490,steps=100,width=0.4310\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 2.781840740489214\n", " neg_mean_loss: -2.781840740489214\n", " node_ip: 127.0.0.1\n", " pid: 47268\n", " time_since_restore: 10.72019100189209\n", " time_this_iter_s: 0.1079869270324707\n", " time_total_s: 10.72019100189209\n", " timestamp: 1658500340\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 101e702a\n", " warmup_time: 0.003309965133666992\n", " \n", "Result for objective_two_1022212a:\n", " date: 2022-07-22_15-32-20\n", " done: true\n", " experiment_id: c3c0bdb8b68146c68f2c34e08fa2f184\n", " experiment_tag: 8_activation=relu,mult=1.3226,height=71.2885,steps=100,width=11.4466\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 9.515910032420853\n", " neg_mean_loss: -9.515910032420853\n", " node_ip: 127.0.0.1\n", " pid: 47269\n", " time_since_restore: 10.707203149795532\n", " time_this_iter_s: 0.10514593124389648\n", " time_total_s: 10.707203149795532\n", " timestamp: 1658500340\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 1022212a\n", " warmup_time: 0.0029020309448242188\n", " \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Result for objective_two_18090732:\n", " date: 2022-07-22_15-32-23\n", " done: false\n", " experiment_id: 7cb1145f46214bc4a5dd35e796969e53\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 0\n", " iterations_since_restore: 1\n", " mean_loss: 15.276134004304078\n", " neg_mean_loss: -15.276134004304078\n", " node_ip: 127.0.0.1\n", " pid: 47302\n", " time_since_restore: 0.00011110305786132812\n", " time_this_iter_s: 0.00011110305786132812\n", " time_total_s: 0.00011110305786132812\n", " timestamp: 1658500343\n", " timesteps_since_restore: 0\n", " training_iteration: 1\n", " trial_id: '18090732'\n", " warmup_time: 0.0028650760650634766\n", " \n", "Result for objective_two_15fbf8dc:\n", " date: 2022-07-22_15-32-24\n", " done: false\n", " experiment_id: 825ceb15d1134810ba19f09bb90c2b35\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: -3.708075105876967\n", " neg_mean_loss: 3.708075105876967\n", " node_ip: 127.0.0.1\n", " pid: 47288\n", " time_since_restore: 5.015958786010742\n", " time_this_iter_s: 0.10610795021057129\n", " time_total_s: 5.015958786010742\n", " timestamp: 1658500344\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: 15fbf8dc\n", " warmup_time: 0.005033969879150391\n", " \n", "Result for objective_two_18090732:\n", " date: 2022-07-22_15-32-28\n", " done: false\n", " experiment_id: 7cb1145f46214bc4a5dd35e796969e53\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 47\n", " iterations_since_restore: 48\n", " mean_loss: 5.4017373166361775\n", " neg_mean_loss: -5.4017373166361775\n", " node_ip: 127.0.0.1\n", " pid: 47302\n", " time_since_restore: 5.047500133514404\n", " time_this_iter_s: 0.10670995712280273\n", " time_total_s: 5.047500133514404\n", " timestamp: 1658500348\n", " timesteps_since_restore: 0\n", " training_iteration: 48\n", " trial_id: '18090732'\n", " warmup_time: 0.0028650760650634766\n", " \n", "Result for objective_two_15fbf8dc:\n", " date: 2022-07-22_15-32-29\n", " done: false\n", " experiment_id: 825ceb15d1134810ba19f09bb90c2b35\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: -3.8171437433543964\n", " neg_mean_loss: 3.8171437433543964\n", " node_ip: 127.0.0.1\n", " pid: 47288\n", " time_since_restore: 10.057979822158813\n", " time_this_iter_s: 0.10869002342224121\n", " time_total_s: 10.057979822158813\n", " timestamp: 1658500349\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: 15fbf8dc\n", " warmup_time: 0.005033969879150391\n", " \n", "Result for objective_two_15fbf8dc:\n", " date: 2022-07-22_15-32-30\n", " done: true\n", " experiment_id: 825ceb15d1134810ba19f09bb90c2b35\n", " experiment_tag: 9_activation=relu,mult=1.2217,height=-32.1584,steps=100,width=9.4322\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: -3.8227168355020016\n", " neg_mean_loss: 3.8227168355020016\n", " node_ip: 127.0.0.1\n", " pid: 47288\n", " time_since_restore: 10.599114894866943\n", " time_this_iter_s: 0.10727405548095703\n", " time_total_s: 10.599114894866943\n", " timestamp: 1658500350\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: 15fbf8dc\n", " warmup_time: 0.005033969879150391\n", " \n", "Result for objective_two_18090732:\n", " date: 2022-07-22_15-32-33\n", " done: false\n", " experiment_id: 7cb1145f46214bc4a5dd35e796969e53\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 94\n", " iterations_since_restore: 95\n", " mean_loss: 5.339332557853146\n", " neg_mean_loss: -5.339332557853146\n", " node_ip: 127.0.0.1\n", " pid: 47302\n", " time_since_restore: 10.092173099517822\n", " time_this_iter_s: 0.1067051887512207\n", " time_total_s: 10.092173099517822\n", " timestamp: 1658500353\n", " timesteps_since_restore: 0\n", " training_iteration: 95\n", " trial_id: '18090732'\n", " warmup_time: 0.0028650760650634766\n", " \n", "Result for objective_two_18090732:\n", " date: 2022-07-22_15-32-33\n", " done: true\n", " experiment_id: 7cb1145f46214bc4a5dd35e796969e53\n", " experiment_tag: 10_activation=tanh,height=52.7613,steps=100,width=16.7268\n", " hostname: Kais-MacBook-Pro.local\n", " iterations: 99\n", " iterations_since_restore: 100\n", " mean_loss: 5.336159871047403\n", " neg_mean_loss: -5.336159871047403\n", " node_ip: 127.0.0.1\n", " pid: 47302\n", " time_since_restore: 10.633639097213745\n", " time_this_iter_s: 0.1067359447479248\n", " time_total_s: 10.633639097213745\n", " timestamp: 1658500353\n", " timesteps_since_restore: 0\n", " training_iteration: 100\n", " trial_id: '18090732'\n", " warmup_time: 0.0028650760650634766\n", " \n" ] } ], "source": [ "tuner = tune.Tuner(\n", " objective_two,\n", " tune_config=tune.TuneConfig(\n", " metric=\"mean_loss\",\n", " mode=\"min\",\n", " search_alg=algo,\n", " num_samples=num_samples,\n", " ),\n", ")\n", "results = tuner.fit()" ] }, { "attachments": {}, "cell_type": "markdown", "id": "e6172afa", "metadata": {}, "source": [ "Finally, we again show the hyperparameters that minimize the mean loss defined by the score of the objective function above. " ] }, { "cell_type": "code", "execution_count": 17, "id": "03c3fc49", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best hyperparameters found were: {'activation': {'activation': 'relu', 'mult': 1.3982639549501585}, 'height': -66.3136247260571, 'steps': 100, 'width': 13.922128223483856}\n" ] } ], "source": [ "print(\"Best hyperparameters found were: \", results.get_best_result().config)" ] }, { "cell_type": "code", "execution_count": 18, "id": "2f7b72d3", "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "ray.shutdown()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.7" }, "orphan": true }, "nbformat": 4, "nbformat_minor": 5 }