{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "a587ce4e",
"metadata": {},
"source": [
"# Running Tune experiments with Optuna\n",
"\n",
"In this tutorial we introduce Optuna, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with Optuna and, as a result, allow you to seamlessly scale up a Optuna optimization process - without sacrificing performance.\n",
"\n",
"Similar to Ray Tune, Optuna is an automatic hyperparameter optimization software framework, particularly designed for machine learning. It features an imperative (\"how\" over \"what\" emphasis), define-by-run style user API. With Optuna, a user has the ability to dynamically construct the search spaces for the hyperparameters. Optuna falls in the domain of \"derivative-free optimization\" and \"black-box optimization\".\n",
"\n",
"In this example we minimize a simple objective to briefly demonstrate the usage of Optuna with Ray Tune via `OptunaSearch`, including examples of conditional search spaces (string together relationships between hyperparameters), and the multi-objective problem (measure trade-offs among all important metrics). It's useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume `optuna==2.9.1` library is installed. To learn more, please refer to [Optuna website](https://optuna.org/).\n",
"\n",
"Please note that sophisticated schedulers, such as `AsyncHyperBandScheduler`, may not work correctly with multi-objective optimization, since they typically expect a scalar score to compare fitness among trials.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "aeaf9ff0",
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: optuna==2.9.1 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (2.9.1)\n",
"Requirement already satisfied: alembic in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (1.7.6)\n",
"Requirement already satisfied: colorlog in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (6.6.0)\n",
"Requirement already satisfied: PyYAML in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (5.4.1)\n",
"Requirement already satisfied: scipy!=1.4.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (1.4.1)\n",
"Requirement already satisfied: tqdm in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (4.64.0)\n",
"Requirement already satisfied: numpy in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (1.21.6)\n",
"Requirement already satisfied: cliff in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (3.10.1)\n",
"Requirement already satisfied: sqlalchemy>=1.1.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (1.4.32)\n",
"Requirement already satisfied: packaging>=20.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (21.3)\n",
"Requirement already satisfied: cmaes>=0.8.2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from optuna==2.9.1) (0.8.2)\n",
"Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from packaging>=20.0->optuna==2.9.1) (2.4.7)\n",
"Requirement already satisfied: importlib-metadata in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from sqlalchemy>=1.1.0->optuna==2.9.1) (4.10.1)\n",
"Requirement already satisfied: greenlet!=0.4.17 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from sqlalchemy>=1.1.0->optuna==2.9.1) (1.1.2)\n",
"Requirement already satisfied: importlib-resources in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from alembic->optuna==2.9.1) (5.4.0)\n",
"Requirement already satisfied: Mako in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from alembic->optuna==2.9.1) (1.1.6)\n",
"Requirement already satisfied: autopage>=0.4.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cliff->optuna==2.9.1) (0.5.0)\n",
"Requirement already satisfied: cmd2>=1.0.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cliff->optuna==2.9.1) (2.4.0)\n",
"Requirement already satisfied: PrettyTable>=0.7.2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cliff->optuna==2.9.1) (3.2.0)\n",
"Requirement already satisfied: stevedore>=2.0.1 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cliff->optuna==2.9.1) (3.5.0)\n",
"Requirement already satisfied: pbr!=2.1.0,>=2.0.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cliff->optuna==2.9.1) (5.8.1)\n",
"Requirement already satisfied: typing-extensions in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cmd2>=1.0.0->cliff->optuna==2.9.1) (4.1.1)\n",
"Requirement already satisfied: attrs>=16.3.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cmd2>=1.0.0->cliff->optuna==2.9.1) (21.2.0)\n",
"Requirement already satisfied: pyperclip>=1.6 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cmd2>=1.0.0->cliff->optuna==2.9.1) (1.8.2)\n",
"Requirement already satisfied: wcwidth>=0.1.7 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from cmd2>=1.0.0->cliff->optuna==2.9.1) (0.2.5)\n",
"Requirement already satisfied: zipp>=0.5 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from importlib-metadata->sqlalchemy>=1.1.0->optuna==2.9.1) (3.7.0)\n",
"Requirement already satisfied: MarkupSafe>=0.9.2 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from Mako->alembic->optuna==2.9.1) (2.0.1)\n",
"\u001b[33mWARNING: There was an error checking the latest version of pip.\u001b[0m\u001b[33m\n",
"\u001b[0m"
]
}
],
"source": [
"# !pip install ray[tune]\n",
"!pip install optuna==2.9.1"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "467466a3",
"metadata": {},
"source": [
"Click below to see all the imports we need for this example.\n",
"You can also launch directly into a Binder instance to run this notebook yourself.\n",
"Just click on the rocket symbol at the top of the navigation."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "fb1ad624",
"metadata": {
"tags": [
"hide-input"
]
},
"outputs": [],
"source": [
"import time\n",
"from typing import Dict, Optional, Any\n",
"\n",
"import ray\n",
"from ray import train, tune\n",
"from ray.tune.search import ConcurrencyLimiter\n",
"from ray.tune.search.optuna import OptunaSearch"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e64f0b44",
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [],
"source": [
"ray.init(configure_logging=False)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "56b0c685",
"metadata": {},
"source": [
"Let's start by defining a simple evaluation function.\n",
"An explicit math formula is queried here for demonstration, yet in practice this is typically a black-box function-- e.g. the performance results after training an ML model.\n",
"We artificially sleep for a bit (`0.1` seconds) to simulate a long-running ML experiment.\n",
"This setup assumes that we're running multiple `step`s of an experiment while tuning three hyperparameters,\n",
"namely `width`, `height`, and `activation`."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "90a11f98",
"metadata": {},
"outputs": [],
"source": [
"def evaluate(step, width, height, activation):\n",
" time.sleep(0.1)\n",
" activation_boost = 10 if activation==\"relu\" else 0\n",
" return (0.1 + width * step / 100) ** (-1) + height * 0.1 + activation_boost"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "bc579b83",
"metadata": {},
"source": [
"Next, our `objective` function to be optimized takes a Tune `config`, evaluates the `score` of your experiment in a training loop,\n",
"and uses `train.report` to report the `score` back to Tune."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "3a11d0e0",
"metadata": {
"lines_to_next_cell": 0
},
"outputs": [],
"source": [
"def objective(config):\n",
" for step in range(config[\"steps\"]):\n",
" score = evaluate(step, config[\"width\"], config[\"height\"], config[\"activation\"])\n",
" train.report({\"iterations\": step, \"mean_loss\": score})"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "c58bd20b",
"metadata": {},
"source": [
"Next we define a search space. The critical assumption is that the optimal hyperparamters live within this space. Yet, if the space is very large, then those hyperparamters may be difficult to find in a short amount of time.\n",
"\n",
"The simplest case is a search space with independent dimensions. In this case, a config dictionary will suffice."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "c3e4eecb",
"metadata": {},
"outputs": [],
"source": [
"search_space = {\n",
" \"steps\": 100,\n",
" \"width\": tune.uniform(0, 20),\n",
" \"height\": tune.uniform(-100, 100),\n",
" \"activation\": tune.choice([\"relu\", \"tanh\"]),\n",
"}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "ef0c666d",
"metadata": {},
"source": [
"Here we define the Optuna search algorithm:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "f23cadc8",
"metadata": {},
"outputs": [],
"source": [
"algo = OptunaSearch()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "4287fa79",
"metadata": {},
"source": [
"We also constrain the number of concurrent trials to `4` with a `ConcurrencyLimiter`."
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "68022ea4",
"metadata": {},
"outputs": [],
"source": [
"algo = ConcurrencyLimiter(algo, max_concurrent=4)\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "4c250f74",
"metadata": {},
"source": [
"The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to `1000` samples.\n",
"(you can decrease this if it takes too long on your machine)."
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "f6c21314",
"metadata": {},
"outputs": [],
"source": [
"num_samples = 1000"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "9533aabf",
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [],
"source": [
"# We override here for our smoke tests.\n",
"num_samples = 10"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "92942b88",
"metadata": {},
"source": [
"Finally, we run the experiment to `\"min\"`imize the \"mean_loss\" of the `objective` by searching `search_space` via `algo`, `num_samples` times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute `tuner.fit()`."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "4e224bb2",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n",
"\u001b[32m[I 2022-07-22 15:21:47,769]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n"
]
},
{
"data": {
"text/html": [
"== Status ==
Current time: 2022-07-22 15:22:32 (running for 00:00:43.89)
Memory usage on this node: 10.1/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects
Current best trial: b1bad2fe with mean_loss=-1.5166144695235813 and parameters={'steps': 100, 'width': 7.688465886501621, 'height': -16.46289560854555, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/optuna_example
Number of trials: 10/10 (10 TERMINATED)
\n",
"\n",
"Trial name | status | loc | activation | height | width | loss | iter | total time (s) | iterations | neg_mean_loss |
\n",
"\n",
"\n",
"objective_9f689668 | TERMINATED | 127.0.0.1:46141 | tanh | 32.9806 | 16.0406 | 3.36064 | 100 | 11.0409 | 99 | -3.36064 |
\n",
"objective_a11d2104 | TERMINATED | 127.0.0.1:46149 | relu | 72.2627 | 0.763881 | 18.3942 | 100 | 11.7018 | 99 | -18.3942 |
\n",
"objective_a11eef8e | TERMINATED | 127.0.0.1:46150 | relu | -32.8474 | 13.1144 | 6.79169 | 100 | 11.8071 | 99 | -6.79169 |
\n",
"objective_a1209e56 | TERMINATED | 127.0.0.1:46151 | tanh | 75.6408 | 10.4415 | 7.65989 | 100 | 11.6987 | 99 | -7.65989 |
\n",
"objective_a7b383e6 | TERMINATED | 127.0.0.1:46174 | relu | 50.1501 | 3.4612 | 15.2986 | 100 | 10.7015 | 99 | -15.2986 |
\n",
"objective_a9c844c8 | TERMINATED | 127.0.0.1:46183 | relu | -40.3931 | 14.0525 | 6.03205 | 100 | 10.6687 | 99 | -6.03205 |
\n",
"objective_a9cb308e | TERMINATED | 127.0.0.1:46184 | tanh | 48.4802 | 12.7746 | 4.92647 | 100 | 10.7122 | 99 | -4.92647 |
\n",
"objective_a9d8332e | TERMINATED | 127.0.0.1:46189 | tanh | -1.26682 | 10.9788 | -0.0355157 | 100 | 10.7089 | 99 | 0.0355157 |
\n",
"objective_af948a1a | TERMINATED | 127.0.0.1:46204 | tanh | 61.6381 | 11.0105 | 6.25472 | 100 | 10.744 | 99 | -6.25472 |
\n",
"objective_b1bad2fe | TERMINATED | 127.0.0.1:46209 | tanh | -16.4629 | 7.68847 | -1.51661 | 100 | 10.7274 | 99 | 1.51661 |
\n",
"\n",
"
"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_9f689668:\n",
" date: 2022-07-22_15-21-51\n",
" done: false\n",
" experiment_id: 7d29e5c95aa44d77becc24b23a326b9b\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 13.298063134872326\n",
" neg_mean_loss: -13.298063134872326\n",
" node_ip: 127.0.0.1\n",
" pid: 46141\n",
" time_since_restore: 0.10468196868896484\n",
" time_this_iter_s: 0.10468196868896484\n",
" time_total_s: 0.10468196868896484\n",
" timestamp: 1658499711\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 9f689668\n",
" warmup_time: 0.003930091857910156\n",
" \n",
"Result for objective_a1209e56:\n",
" date: 2022-07-22_15-21-54\n",
" done: false\n",
" experiment_id: 70d9788c327c43fb94450e5084416b81\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 17.564083106485064\n",
" neg_mean_loss: -17.564083106485064\n",
" node_ip: 127.0.0.1\n",
" pid: 46151\n",
" time_since_restore: 0.10416483879089355\n",
" time_this_iter_s: 0.10416483879089355\n",
" time_total_s: 0.10416483879089355\n",
" timestamp: 1658499714\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: a1209e56\n",
" warmup_time: 0.0026938915252685547\n",
" \n",
"Result for objective_a11d2104:\n",
" date: 2022-07-22_15-21-54\n",
" done: false\n",
" experiment_id: af2598364a004cd28f37ede19afd1ffb\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 27.2262748487915\n",
" neg_mean_loss: -27.2262748487915\n",
" node_ip: 127.0.0.1\n",
" pid: 46149\n",
" time_since_restore: 0.10507392883300781\n",
" time_this_iter_s: 0.10507392883300781\n",
" time_total_s: 0.10507392883300781\n",
" timestamp: 1658499714\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: a11d2104\n",
" warmup_time: 0.0031232833862304688\n",
" \n",
"Result for objective_a11eef8e:\n",
" date: 2022-07-22_15-21-54\n",
" done: false\n",
" experiment_id: e4a5922603d44b459c23607a42d5e574\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 16.71525618736072\n",
" neg_mean_loss: -16.71525618736072\n",
" node_ip: 127.0.0.1\n",
" pid: 46150\n",
" time_since_restore: 0.10363006591796875\n",
" time_this_iter_s: 0.10363006591796875\n",
" time_total_s: 0.10363006591796875\n",
" timestamp: 1658499714\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: a11eef8e\n",
" warmup_time: 0.0028891563415527344\n",
" \n",
"Result for objective_9f689668:\n",
" date: 2022-07-22_15-21-56\n",
" done: false\n",
" experiment_id: 7d29e5c95aa44d77becc24b23a326b9b\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 45\n",
" iterations_since_restore: 46\n",
" mean_loss: 3.434707064844201\n",
" neg_mean_loss: -3.434707064844201\n",
" node_ip: 127.0.0.1\n",
" pid: 46141\n",
" time_since_restore: 5.202661752700806\n",
" time_this_iter_s: 0.10527586936950684\n",
" time_total_s: 5.202661752700806\n",
" timestamp: 1658499716\n",
" timesteps_since_restore: 0\n",
" training_iteration: 46\n",
" trial_id: 9f689668\n",
" warmup_time: 0.003930091857910156\n",
" \n",
"Result for objective_a1209e56:\n",
" date: 2022-07-22_15-21-59\n",
" done: false\n",
" experiment_id: 70d9788c327c43fb94450e5084416b81\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 7.763782433167051\n",
" neg_mean_loss: -7.763782433167051\n",
" node_ip: 127.0.0.1\n",
" pid: 46151\n",
" time_since_restore: 5.146381139755249\n",
" time_this_iter_s: 0.10691022872924805\n",
" time_total_s: 5.146381139755249\n",
" timestamp: 1658499719\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: a1209e56\n",
" warmup_time: 0.0026938915252685547\n",
" \n",
"Result for objective_a11eef8e:\n",
" date: 2022-07-22_15-21-59\n",
" done: false\n",
" experiment_id: e4a5922603d44b459c23607a42d5e574\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 6.874905099790281\n",
" neg_mean_loss: -6.874905099790281\n",
" node_ip: 127.0.0.1\n",
" pid: 46150\n",
" time_since_restore: 5.1618921756744385\n",
" time_this_iter_s: 0.10776615142822266\n",
" time_total_s: 5.1618921756744385\n",
" timestamp: 1658499719\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: a11eef8e\n",
" warmup_time: 0.0028891563415527344\n",
" \n",
"Result for objective_a11d2104:\n",
" date: 2022-07-22_15-21-59\n",
" done: false\n",
" experiment_id: af2598364a004cd28f37ede19afd1ffb\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 19.4048102011043\n",
" neg_mean_loss: -19.4048102011043\n",
" node_ip: 127.0.0.1\n",
" pid: 46149\n",
" time_since_restore: 5.170748233795166\n",
" time_this_iter_s: 0.11556220054626465\n",
" time_total_s: 5.170748233795166\n",
" timestamp: 1658499719\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: a11d2104\n",
" warmup_time: 0.0031232833862304688\n",
" \n",
"Result for objective_9f689668:\n",
" date: 2022-07-22_15-22-01\n",
" done: false\n",
" experiment_id: 7d29e5c95aa44d77becc24b23a326b9b\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 92\n",
" iterations_since_restore: 93\n",
" mean_loss: 3.3653696881211697\n",
" neg_mean_loss: -3.3653696881211697\n",
" node_ip: 127.0.0.1\n",
" pid: 46141\n",
" time_since_restore: 10.248206853866577\n",
" time_this_iter_s: 0.10745501518249512\n",
" time_total_s: 10.248206853866577\n",
" timestamp: 1658499721\n",
" timesteps_since_restore: 0\n",
" training_iteration: 93\n",
" trial_id: 9f689668\n",
" warmup_time: 0.003930091857910156\n",
" \n",
"Result for objective_9f689668:\n",
" date: 2022-07-22_15-22-02\n",
" done: true\n",
" experiment_id: 7d29e5c95aa44d77becc24b23a326b9b\n",
" experiment_tag: 1_activation=tanh,height=32.9806,steps=100,width=16.0406\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 3.3606404197394255\n",
" neg_mean_loss: -3.3606404197394255\n",
" node_ip: 127.0.0.1\n",
" pid: 46141\n",
" time_since_restore: 11.04089903831482\n",
" time_this_iter_s: 0.10824704170227051\n",
" time_total_s: 11.04089903831482\n",
" timestamp: 1658499722\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 9f689668\n",
" warmup_time: 0.003930091857910156\n",
" \n",
"Result for objective_a1209e56:\n",
" date: 2022-07-22_15-22-04\n",
" done: false\n",
" experiment_id: 70d9788c327c43fb94450e5084416b81\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 90\n",
" iterations_since_restore: 91\n",
" mean_loss: 7.669375143981996\n",
" neg_mean_loss: -7.669375143981996\n",
" node_ip: 127.0.0.1\n",
" pid: 46151\n",
" time_since_restore: 9.724387168884277\n",
" time_this_iter_s: 0.10670638084411621\n",
" time_total_s: 9.724387168884277\n",
" timestamp: 1658499724\n",
" timesteps_since_restore: 0\n",
" training_iteration: 91\n",
" trial_id: a1209e56\n",
" warmup_time: 0.0026938915252685547\n",
" \n",
"Result for objective_a11d2104:\n",
" date: 2022-07-22_15-22-04\n",
" done: false\n",
" experiment_id: af2598364a004cd28f37ede19afd1ffb\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 90\n",
" iterations_since_restore: 91\n",
" mean_loss: 18.49612779999574\n",
" neg_mean_loss: -18.49612779999574\n",
" node_ip: 127.0.0.1\n",
" pid: 46149\n",
" time_since_restore: 9.818917274475098\n",
" time_this_iter_s: 0.10725140571594238\n",
" time_total_s: 9.818917274475098\n",
" timestamp: 1658499724\n",
" timesteps_since_restore: 0\n",
" training_iteration: 91\n",
" trial_id: a11d2104\n",
" warmup_time: 0.0031232833862304688\n",
" \n",
"Result for objective_a11eef8e:\n",
" date: 2022-07-22_15-22-03\n",
" done: false\n",
" experiment_id: e4a5922603d44b459c23607a42d5e574\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 89\n",
" iterations_since_restore: 90\n",
" mean_loss: 6.800205168670805\n",
" neg_mean_loss: -6.800205168670805\n",
" node_ip: 127.0.0.1\n",
" pid: 46150\n",
" time_since_restore: 9.714852094650269\n",
" time_this_iter_s: 0.10286402702331543\n",
" time_total_s: 9.714852094650269\n",
" timestamp: 1658499723\n",
" timesteps_since_restore: 0\n",
" training_iteration: 90\n",
" trial_id: a11eef8e\n",
" warmup_time: 0.0028891563415527344\n",
" \n",
"Result for objective_a7b383e6:\n",
" date: 2022-07-22_15-22-05\n",
" done: false\n",
" experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 25.015010793208496\n",
" neg_mean_loss: -25.015010793208496\n",
" node_ip: 127.0.0.1\n",
" pid: 46174\n",
" time_since_restore: 0.10384607315063477\n",
" time_this_iter_s: 0.10384607315063477\n",
" time_total_s: 0.10384607315063477\n",
" timestamp: 1658499725\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: a7b383e6\n",
" warmup_time: 0.002817869186401367\n",
" \n",
"Result for objective_a1209e56:\n",
" date: 2022-07-22_15-22-05\n",
" done: true\n",
" experiment_id: 70d9788c327c43fb94450e5084416b81\n",
" experiment_tag: 4_activation=tanh,height=75.6408,steps=100,width=10.4415\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 7.659894851608276\n",
" neg_mean_loss: -7.659894851608276\n",
" node_ip: 127.0.0.1\n",
" pid: 46151\n",
" time_since_restore: 11.698678970336914\n",
" time_this_iter_s: 0.10874581336975098\n",
" time_total_s: 11.698678970336914\n",
" timestamp: 1658499725\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: a1209e56\n",
" warmup_time: 0.0026938915252685547\n",
" \n",
"Result for objective_a11d2104:\n",
" date: 2022-07-22_15-22-05\n",
" done: true\n",
" experiment_id: af2598364a004cd28f37ede19afd1ffb\n",
" experiment_tag: 2_activation=relu,height=72.2627,steps=100,width=0.7639\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 18.394168929385685\n",
" neg_mean_loss: -18.394168929385685\n",
" node_ip: 127.0.0.1\n",
" pid: 46149\n",
" time_since_restore: 11.701792001724243\n",
" time_this_iter_s: 0.10734391212463379\n",
" time_total_s: 11.701792001724243\n",
" timestamp: 1658499725\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: a11d2104\n",
" warmup_time: 0.0031232833862304688\n",
" \n",
"Result for objective_a11eef8e:\n",
" date: 2022-07-22_15-22-06\n",
" done: true\n",
" experiment_id: e4a5922603d44b459c23607a42d5e574\n",
" experiment_tag: 3_activation=relu,height=-32.8474,steps=100,width=13.1144\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 6.791690049131352\n",
" neg_mean_loss: -6.791690049131352\n",
" node_ip: 127.0.0.1\n",
" pid: 46150\n",
" time_since_restore: 11.807135105133057\n",
" time_this_iter_s: 0.10672497749328613\n",
" time_total_s: 11.807135105133057\n",
" timestamp: 1658499726\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: a11eef8e\n",
" warmup_time: 0.0028891563415527344\n",
" \n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_a9cb308e:\n",
" date: 2022-07-22_15-22-08\n",
" done: false\n",
" experiment_id: fcc8b8f4880c471c8e8545228d7e591a\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 14.84802392036395\n",
" neg_mean_loss: -14.84802392036395\n",
" node_ip: 127.0.0.1\n",
" pid: 46184\n",
" time_since_restore: 0.1039888858795166\n",
" time_this_iter_s: 0.1039888858795166\n",
" time_total_s: 0.1039888858795166\n",
" timestamp: 1658499728\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: a9cb308e\n",
" warmup_time: 0.0027310848236083984\n",
" \n",
"Result for objective_a9c844c8:\n",
" date: 2022-07-22_15-22-08\n",
" done: false\n",
" experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 15.960686077942514\n",
" neg_mean_loss: -15.960686077942514\n",
" node_ip: 127.0.0.1\n",
" pid: 46183\n",
" time_since_restore: 0.1004641056060791\n",
" time_this_iter_s: 0.1004641056060791\n",
" time_total_s: 0.1004641056060791\n",
" timestamp: 1658499728\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: a9c844c8\n",
" warmup_time: 0.002599954605102539\n",
" \n",
"Result for objective_a9d8332e:\n",
" date: 2022-07-22_15-22-08\n",
" done: false\n",
" experiment_id: 9c2f8aa7e4ed45c8867025673b146195\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 9.873318211243287\n",
" neg_mean_loss: -9.873318211243287\n",
" node_ip: 127.0.0.1\n",
" pid: 46189\n",
" time_since_restore: 0.10463905334472656\n",
" time_this_iter_s: 0.10463905334472656\n",
" time_total_s: 0.10463905334472656\n",
" timestamp: 1658499728\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: a9d8332e\n",
" warmup_time: 0.002753019332885742\n",
" \n",
"Result for objective_a7b383e6:\n",
" date: 2022-07-22_15-22-10\n",
" done: false\n",
" experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 15.594128340598196\n",
" neg_mean_loss: -15.594128340598196\n",
" node_ip: 127.0.0.1\n",
" pid: 46174\n",
" time_since_restore: 5.12034797668457\n",
" time_this_iter_s: 0.10577988624572754\n",
" time_total_s: 5.12034797668457\n",
" timestamp: 1658499730\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: a7b383e6\n",
" warmup_time: 0.002817869186401367\n",
" \n",
"Result for objective_a9cb308e:\n",
" date: 2022-07-22_15-22-13\n",
" done: false\n",
" experiment_id: fcc8b8f4880c471c8e8545228d7e591a\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 5.011849709111575\n",
" neg_mean_loss: -5.011849709111575\n",
" node_ip: 127.0.0.1\n",
" pid: 46184\n",
" time_since_restore: 5.150436162948608\n",
" time_this_iter_s: 0.10827016830444336\n",
" time_total_s: 5.150436162948608\n",
" timestamp: 1658499733\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: a9cb308e\n",
" warmup_time: 0.0027310848236083984\n",
" \n",
"Result for objective_a9c844c8:\n",
" date: 2022-07-22_15-22-13\n",
" done: false\n",
" experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 6.109835222238134\n",
" neg_mean_loss: -6.109835222238134\n",
" node_ip: 127.0.0.1\n",
" pid: 46183\n",
" time_since_restore: 5.136260032653809\n",
" time_this_iter_s: 0.10730218887329102\n",
" time_total_s: 5.136260032653809\n",
" timestamp: 1658499733\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: a9c844c8\n",
" warmup_time: 0.002599954605102539\n",
" \n",
"Result for objective_a9d8332e:\n",
" date: 2022-07-22_15-22-13\n",
" done: false\n",
" experiment_id: 9c2f8aa7e4ed45c8867025673b146195\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 0.06343129971586822\n",
" neg_mean_loss: -0.06343129971586822\n",
" node_ip: 127.0.0.1\n",
" pid: 46189\n",
" time_since_restore: 5.141700983047485\n",
" time_this_iter_s: 0.1074991226196289\n",
" time_total_s: 5.141700983047485\n",
" timestamp: 1658499733\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: a9d8332e\n",
" warmup_time: 0.002753019332885742\n",
" \n",
"Result for objective_a7b383e6:\n",
" date: 2022-07-22_15-22-15\n",
" done: false\n",
" experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 15.313204013214245\n",
" neg_mean_loss: -15.313204013214245\n",
" node_ip: 127.0.0.1\n",
" pid: 46174\n",
" time_since_restore: 10.163710832595825\n",
" time_this_iter_s: 0.10754895210266113\n",
" time_total_s: 10.163710832595825\n",
" timestamp: 1658499735\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: a7b383e6\n",
" warmup_time: 0.002817869186401367\n",
" \n",
"Result for objective_a7b383e6:\n",
" date: 2022-07-22_15-22-15\n",
" done: true\n",
" experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa\n",
" experiment_tag: 5_activation=relu,height=50.1501,steps=100,width=3.4612\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 15.298570798421121\n",
" neg_mean_loss: -15.298570798421121\n",
" node_ip: 127.0.0.1\n",
" pid: 46174\n",
" time_since_restore: 10.701509952545166\n",
" time_this_iter_s: 0.10684585571289062\n",
" time_total_s: 10.701509952545166\n",
" timestamp: 1658499735\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: a7b383e6\n",
" warmup_time: 0.002817869186401367\n",
" \n",
"Result for objective_af948a1a:\n",
" date: 2022-07-22_15-22-18\n",
" done: false\n",
" experiment_id: d27f58d6527a4d47ae4c12496f5275cf\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 16.16381214127525\n",
" neg_mean_loss: -16.16381214127525\n",
" node_ip: 127.0.0.1\n",
" pid: 46204\n",
" time_since_restore: 0.10407209396362305\n",
" time_this_iter_s: 0.10407209396362305\n",
" time_total_s: 0.10407209396362305\n",
" timestamp: 1658499738\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: af948a1a\n",
" warmup_time: 0.002669095993041992\n",
" \n",
"Result for objective_a9cb308e:\n",
" date: 2022-07-22_15-22-18\n",
" done: false\n",
" experiment_id: fcc8b8f4880c471c8e8545228d7e591a\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 4.930613328484136\n",
" neg_mean_loss: -4.930613328484136\n",
" node_ip: 127.0.0.1\n",
" pid: 46184\n",
" time_since_restore: 10.175731897354126\n",
" time_this_iter_s: 0.10618400573730469\n",
" time_total_s: 10.175731897354126\n",
" timestamp: 1658499738\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: a9cb308e\n",
" warmup_time: 0.0027310848236083984\n",
" \n",
"Result for objective_a9d8332e:\n",
" date: 2022-07-22_15-22-18\n",
" done: false\n",
" experiment_id: 9c2f8aa7e4ed45c8867025673b146195\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: -0.030712998363490673\n",
" neg_mean_loss: 0.030712998363490673\n",
" node_ip: 127.0.0.1\n",
" pid: 46189\n",
" time_since_restore: 10.171820163726807\n",
" time_this_iter_s: 0.10807609558105469\n",
" time_total_s: 10.171820163726807\n",
" timestamp: 1658499738\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: a9d8332e\n",
" warmup_time: 0.002753019332885742\n",
" \n",
"Result for objective_a9c844c8:\n",
" date: 2022-07-22_15-22-18\n",
" done: false\n",
" experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 95\n",
" iterations_since_restore: 96\n",
" mean_loss: 6.035035952030598\n",
" neg_mean_loss: -6.035035952030598\n",
" node_ip: 127.0.0.1\n",
" pid: 46183\n",
" time_since_restore: 10.23954701423645\n",
" time_this_iter_s: 0.10863399505615234\n",
" time_total_s: 10.23954701423645\n",
" timestamp: 1658499738\n",
" timesteps_since_restore: 0\n",
" training_iteration: 96\n",
" trial_id: a9c844c8\n",
" warmup_time: 0.002599954605102539\n",
" \n",
"Result for objective_a9cb308e:\n",
" date: 2022-07-22_15-22-19\n",
" done: true\n",
" experiment_id: fcc8b8f4880c471c8e8545228d7e591a\n",
" experiment_tag: 7_activation=tanh,height=48.4802,steps=100,width=12.7746\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 4.926474869576359\n",
" neg_mean_loss: -4.926474869576359\n",
" node_ip: 127.0.0.1\n",
" pid: 46184\n",
" time_since_restore: 10.712197065353394\n",
" time_this_iter_s: 0.10726809501647949\n",
" time_total_s: 10.712197065353394\n",
" timestamp: 1658499739\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: a9cb308e\n",
" warmup_time: 0.0027310848236083984\n",
" \n",
"Result for objective_a9c844c8:\n",
" date: 2022-07-22_15-22-19\n",
" done: true\n",
" experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e\n",
" experiment_tag: 6_activation=relu,height=-40.3931,steps=100,width=14.0525\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 6.03205335569619\n",
" neg_mean_loss: -6.03205335569619\n",
" node_ip: 127.0.0.1\n",
" pid: 46183\n",
" time_since_restore: 10.668684959411621\n",
" time_this_iter_s: 0.1074681282043457\n",
" time_total_s: 10.668684959411621\n",
" timestamp: 1658499739\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: a9c844c8\n",
" warmup_time: 0.002599954605102539\n",
" \n",
"Result for objective_a9d8332e:\n",
" date: 2022-07-22_15-22-19\n",
" done: true\n",
" experiment_id: 9c2f8aa7e4ed45c8867025673b146195\n",
" experiment_tag: 8_activation=tanh,height=-1.2668,steps=100,width=10.9788\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: -0.035515719608698634\n",
" neg_mean_loss: 0.035515719608698634\n",
" node_ip: 127.0.0.1\n",
" pid: 46189\n",
" time_since_restore: 10.708936929702759\n",
" time_this_iter_s: 0.10948991775512695\n",
" time_total_s: 10.708936929702759\n",
" timestamp: 1658499739\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: a9d8332e\n",
" warmup_time: 0.002753019332885742\n",
" \n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_b1bad2fe:\n",
" date: 2022-07-22_15-22-21\n",
" done: false\n",
" experiment_id: f58c8e48e2324dd69c28f526bf96497c\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 8.353710439145445\n",
" neg_mean_loss: -8.353710439145445\n",
" node_ip: 127.0.0.1\n",
" pid: 46209\n",
" time_since_restore: 0.10369992256164551\n",
" time_this_iter_s: 0.10369992256164551\n",
" time_total_s: 0.10369992256164551\n",
" timestamp: 1658499741\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: b1bad2fe\n",
" warmup_time: 0.0026597976684570312\n",
" \n",
"Result for objective_af948a1a:\n",
" date: 2022-07-22_15-22-23\n",
" done: false\n",
" experiment_id: d27f58d6527a4d47ae4c12496f5275cf\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 6.353387644558537\n",
" neg_mean_loss: -6.353387644558537\n",
" node_ip: 127.0.0.1\n",
" pid: 46204\n",
" time_since_restore: 5.161527872085571\n",
" time_this_iter_s: 0.10784792900085449\n",
" time_total_s: 5.161527872085571\n",
" timestamp: 1658499743\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: af948a1a\n",
" warmup_time: 0.002669095993041992\n",
" \n",
"Result for objective_b1bad2fe:\n",
" date: 2022-07-22_15-22-26\n",
" done: false\n",
" experiment_id: f58c8e48e2324dd69c28f526bf96497c\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: -1.3770075531258914\n",
" neg_mean_loss: 1.3770075531258914\n",
" node_ip: 127.0.0.1\n",
" pid: 46209\n",
" time_since_restore: 5.158898115158081\n",
" time_this_iter_s: 0.10701417922973633\n",
" time_total_s: 5.158898115158081\n",
" timestamp: 1658499746\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: b1bad2fe\n",
" warmup_time: 0.0026597976684570312\n",
" \n",
"Result for objective_af948a1a:\n",
" date: 2022-07-22_15-22-28\n",
" done: false\n",
" experiment_id: d27f58d6527a4d47ae4c12496f5275cf\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 6.259506962612728\n",
" neg_mean_loss: -6.259506962612728\n",
" node_ip: 127.0.0.1\n",
" pid: 46204\n",
" time_since_restore: 10.20361876487732\n",
" time_this_iter_s: 0.10824179649353027\n",
" time_total_s: 10.20361876487732\n",
" timestamp: 1658499748\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: af948a1a\n",
" warmup_time: 0.002669095993041992\n",
" \n",
"Result for objective_af948a1a:\n",
" date: 2022-07-22_15-22-28\n",
" done: true\n",
" experiment_id: d27f58d6527a4d47ae4c12496f5275cf\n",
" experiment_tag: 9_activation=tanh,height=61.6381,steps=100,width=11.0105\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 6.254717826198094\n",
" neg_mean_loss: -6.254717826198094\n",
" node_ip: 127.0.0.1\n",
" pid: 46204\n",
" time_since_restore: 10.744001865386963\n",
" time_this_iter_s: 0.10536718368530273\n",
" time_total_s: 10.744001865386963\n",
" timestamp: 1658499748\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: af948a1a\n",
" warmup_time: 0.002669095993041992\n",
" \n",
"Result for objective_b1bad2fe:\n",
" date: 2022-07-22_15-22-31\n",
" done: false\n",
" experiment_id: f58c8e48e2324dd69c28f526bf96497c\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: -1.5098109958909571\n",
" neg_mean_loss: 1.5098109958909571\n",
" node_ip: 127.0.0.1\n",
" pid: 46209\n",
" time_since_restore: 10.190011024475098\n",
" time_this_iter_s: 0.10639405250549316\n",
" time_total_s: 10.190011024475098\n",
" timestamp: 1658499751\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: b1bad2fe\n",
" warmup_time: 0.0026597976684570312\n",
" \n",
"Result for objective_b1bad2fe:\n",
" date: 2022-07-22_15-22-32\n",
" done: true\n",
" experiment_id: f58c8e48e2324dd69c28f526bf96497c\n",
" experiment_tag: 10_activation=tanh,height=-16.4629,steps=100,width=7.6885\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: -1.5166144695235813\n",
" neg_mean_loss: 1.5166144695235813\n",
" node_ip: 127.0.0.1\n",
" pid: 46209\n",
" time_since_restore: 10.727396011352539\n",
" time_this_iter_s: 0.10742807388305664\n",
" time_total_s: 10.727396011352539\n",
" timestamp: 1658499752\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: b1bad2fe\n",
" warmup_time: 0.0026597976684570312\n",
" \n"
]
}
],
"source": [
"tuner = tune.Tuner(\n",
" objective,\n",
" tune_config=tune.TuneConfig(\n",
" metric=\"mean_loss\",\n",
" mode=\"min\",\n",
" search_alg=algo,\n",
" num_samples=num_samples,\n",
" ),\n",
" param_space=search_space,\n",
")\n",
"results = tuner.fit()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "b66aab6a",
"metadata": {},
"source": [
"And now we have the hyperparameters found to minimize the mean loss."
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "e69db02e",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best hyperparameters found were: {'steps': 100, 'width': 7.688465886501621, 'height': -16.46289560854555, 'activation': 'tanh'}\n"
]
}
],
"source": [
"print(\"Best hyperparameters found were: \", results.get_best_result().config)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "d545d30b",
"metadata": {},
"source": [
"## Providing an initial set of hyperparameters\n",
"\n",
"While defining the search algorithm, we may choose to provide an initial set of hyperparameters that we believe are especially promising or informative, and\n",
"pass this information as a helpful starting point for the `OptunaSearch` object."
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "7596b7f4",
"metadata": {},
"outputs": [],
"source": [
"initial_params = [\n",
" {\"width\": 1, \"height\": 2, \"activation\": \"relu\"},\n",
" {\"width\": 4, \"height\": 2, \"activation\": \"relu\"},\n",
"]"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "f84bbff0",
"metadata": {},
"source": [
"Now the `search_alg` built using `OptunaSearch` takes `points_to_evaluate`."
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "320d1935",
"metadata": {
"lines_to_next_cell": 0
},
"outputs": [],
"source": [
"searcher = OptunaSearch(points_to_evaluate=initial_params)\n",
"algo = ConcurrencyLimiter(searcher, max_concurrent=4)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "9147d9a2",
"metadata": {},
"source": [
"And run the experiment with initial hyperparameter evaluations:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "ee442efd",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[32m[I 2022-07-22 15:22:32,644]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n",
"/Users/kai/coding/ray/python/ray/tune/search/optuna/optuna_search.py:389: ExperimentalWarning: enqueue_trial is experimental (supported from v1.2.0). The interface can change in the future.\n",
" self._ot_study.enqueue_trial(point)\n",
"/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/optuna/study/study.py:857: ExperimentalWarning: create_trial is experimental (supported from v2.0.0). The interface can change in the future.\n",
" create_trial(state=TrialState.WAITING, system_attrs={\"fixed_params\": params})\n",
"/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/optuna/study/study.py:857: ExperimentalWarning: add_trial is experimental (supported from v2.0.0). The interface can change in the future.\n",
" create_trial(state=TrialState.WAITING, system_attrs={\"fixed_params\": params})\n"
]
},
{
"data": {
"text/html": [
"== Status ==
Current time: 2022-07-22 15:23:15 (running for 00:00:42.97)
Memory usage on this node: 10.1/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects
Current best trial: c3762426 with mean_loss=-8.621481572769046 and parameters={'steps': 100, 'width': 0.7236705053153614, 'height': -98.46320622400306, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/optuna_example_2
Number of trials: 10/10 (10 TERMINATED)
\n",
"\n",
"Trial name | status | loc | activation | height | width | loss | iter | total time (s) | iterations | neg_mean_loss |
\n",
"\n",
"\n",
"objective_b9acff32 | TERMINATED | 127.0.0.1:46220 | relu | 2 | 1 | 11.1174 | 100 | 10.6837 | 99 | -11.1174 |
\n",
"objective_bb3b2702 | TERMINATED | 127.0.0.1:46227 | relu | 2 | 4 | 10.4463 | 100 | 11.2816 | 99 | -10.4463 |
\n",
"objective_bb3c8714 | TERMINATED | 127.0.0.1:46228 | relu | -90.7016 | 0.566597 | 2.44285 | 100 | 11.2766 | 99 | -2.44285 |
\n",
"objective_bb3e0210 | TERMINATED | 127.0.0.1:46229 | tanh | -81.4501 | 14.783 | -8.07715 | 100 | 11.2196 | 99 | 8.07715 |
\n",
"objective_c19b9dde | TERMINATED | 127.0.0.1:46248 | relu | 86.466 | 0.308659 | 21.1122 | 100 | 10.7226 | 99 | -21.1122 |
\n",
"objective_c3762426 | TERMINATED | 127.0.0.1:46253 | tanh | -98.4632 | 0.723671 | -8.62148 | 100 | 10.7271 | 99 | 8.62148 |
\n",
"objective_c385528e | TERMINATED | 127.0.0.1:46256 | relu | -54.8368 | 11.0765 | 4.60669 | 100 | 10.7498 | 99 | -4.60669 |
\n",
"objective_c387a7c8 | TERMINATED | 127.0.0.1:46257 | tanh | 42.9466 | 8.01708 | 4.41909 | 100 | 10.7214 | 99 | -4.41909 |
\n",
"objective_c97678f8 | TERMINATED | 127.0.0.1:46272 | tanh | 9.25214 | 15.5288 | 0.989841 | 100 | 12.4591 | 99 | -0.989841 |
\n",
"objective_cb6180fe | TERMINATED | 127.0.0.1:46277 | relu | 41.6647 | 8.0585 | 14.2903 | 100 | 10.752 | 99 | -14.2903 |
\n",
"\n",
"
"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_b9acff32:\n",
" date: 2022-07-22_15-22-35\n",
" done: false\n",
" experiment_id: ec3c260f0ad348fa9b188f9c93c01eba\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 20.2\n",
" neg_mean_loss: -20.2\n",
" node_ip: 127.0.0.1\n",
" pid: 46220\n",
" time_since_restore: 0.10049605369567871\n",
" time_this_iter_s: 0.10049605369567871\n",
" time_total_s: 0.10049605369567871\n",
" timestamp: 1658499755\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: b9acff32\n",
" warmup_time: 0.0029239654541015625\n",
" \n",
"Result for objective_bb3e0210:\n",
" date: 2022-07-22_15-22-37\n",
" done: false\n",
" experiment_id: 08a7bc47c140405296e58a049cd2f6b4\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 1.8549886153795079\n",
" neg_mean_loss: -1.8549886153795079\n",
" node_ip: 127.0.0.1\n",
" pid: 46229\n",
" time_since_restore: 0.10361814498901367\n",
" time_this_iter_s: 0.10361814498901367\n",
" time_total_s: 0.10361814498901367\n",
" timestamp: 1658499757\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: bb3e0210\n",
" warmup_time: 0.002833843231201172\n",
" \n",
"Result for objective_bb3b2702:\n",
" date: 2022-07-22_15-22-37\n",
" done: false\n",
" experiment_id: 2f984120624b4e9aa3392fcc1b16b811\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 20.2\n",
" neg_mean_loss: -20.2\n",
" node_ip: 127.0.0.1\n",
" pid: 46227\n",
" time_since_restore: 0.10420107841491699\n",
" time_this_iter_s: 0.10420107841491699\n",
" time_total_s: 0.10420107841491699\n",
" timestamp: 1658499757\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: bb3b2702\n",
" warmup_time: 0.002672910690307617\n",
" \n",
"Result for objective_bb3c8714:\n",
" date: 2022-07-22_15-22-37\n",
" done: false\n",
" experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 10.929835533284928\n",
" neg_mean_loss: -10.929835533284928\n",
" node_ip: 127.0.0.1\n",
" pid: 46228\n",
" time_since_restore: 0.1048588752746582\n",
" time_this_iter_s: 0.1048588752746582\n",
" time_total_s: 0.1048588752746582\n",
" timestamp: 1658499757\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: bb3c8714\n",
" warmup_time: 0.0027511119842529297\n",
" \n",
"Result for objective_b9acff32:\n",
" date: 2022-07-22_15-22-40\n",
" done: false\n",
" experiment_id: ec3c260f0ad348fa9b188f9c93c01eba\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 48\n",
" iterations_since_restore: 49\n",
" mean_loss: 11.924137931034483\n",
" neg_mean_loss: -11.924137931034483\n",
" node_ip: 127.0.0.1\n",
" pid: 46220\n",
" time_since_restore: 5.207761287689209\n",
" time_this_iter_s: 0.10753417015075684\n",
" time_total_s: 5.207761287689209\n",
" timestamp: 1658499760\n",
" timesteps_since_restore: 0\n",
" training_iteration: 49\n",
" trial_id: b9acff32\n",
" warmup_time: 0.0029239654541015625\n",
" \n",
"Result for objective_bb3e0210:\n",
" date: 2022-07-22_15-22-43\n",
" done: false\n",
" experiment_id: 08a7bc47c140405296e58a049cd2f6b4\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: -8.003127819283387\n",
" neg_mean_loss: 8.003127819283387\n",
" node_ip: 127.0.0.1\n",
" pid: 46229\n",
" time_since_restore: 5.188760042190552\n",
" time_this_iter_s: 0.10853385925292969\n",
" time_total_s: 5.188760042190552\n",
" timestamp: 1658499763\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: bb3e0210\n",
" warmup_time: 0.002833843231201172\n",
" \n",
"Result for objective_bb3b2702:\n",
" date: 2022-07-22_15-22-43\n",
" done: false\n",
" experiment_id: 2f984120624b4e9aa3392fcc1b16b811\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 10.705050505050504\n",
" neg_mean_loss: -10.705050505050504\n",
" node_ip: 127.0.0.1\n",
" pid: 46227\n",
" time_since_restore: 5.1574530601501465\n",
" time_this_iter_s: 0.10719585418701172\n",
" time_total_s: 5.1574530601501465\n",
" timestamp: 1658499763\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: bb3b2702\n",
" warmup_time: 0.002672910690307617\n",
" \n",
"Result for objective_bb3c8714:\n",
" date: 2022-07-22_15-22-43\n",
" done: false\n",
" experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 3.6598337799021596\n",
" neg_mean_loss: -3.6598337799021596\n",
" node_ip: 127.0.0.1\n",
" pid: 46228\n",
" time_since_restore: 5.163677930831909\n",
" time_this_iter_s: 0.10699272155761719\n",
" time_total_s: 5.163677930831909\n",
" timestamp: 1658499763\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: bb3c8714\n",
" warmup_time: 0.0027511119842529297\n",
" \n",
"Result for objective_b9acff32:\n",
" date: 2022-07-22_15-22-45\n",
" done: false\n",
" experiment_id: ec3c260f0ad348fa9b188f9c93c01eba\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 95\n",
" iterations_since_restore: 96\n",
" mean_loss: 11.152380952380952\n",
" neg_mean_loss: -11.152380952380952\n",
" node_ip: 127.0.0.1\n",
" pid: 46220\n",
" time_since_restore: 10.250982284545898\n",
" time_this_iter_s: 0.1073312759399414\n",
" time_total_s: 10.250982284545898\n",
" timestamp: 1658499765\n",
" timesteps_since_restore: 0\n",
" training_iteration: 96\n",
" trial_id: b9acff32\n",
" warmup_time: 0.0029239654541015625\n",
" \n",
"Result for objective_b9acff32:\n",
" date: 2022-07-22_15-22-45\n",
" done: true\n",
" experiment_id: ec3c260f0ad348fa9b188f9c93c01eba\n",
" experiment_tag: 1_activation=relu,height=2,steps=100,width=1\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 11.11743119266055\n",
" neg_mean_loss: -11.11743119266055\n",
" node_ip: 127.0.0.1\n",
" pid: 46220\n",
" time_since_restore: 10.68371295928955\n",
" time_this_iter_s: 0.10785365104675293\n",
" time_total_s: 10.68371295928955\n",
" timestamp: 1658499765\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: b9acff32\n",
" warmup_time: 0.0029239654541015625\n",
" \n",
"Result for objective_bb3e0210:\n",
" date: 2022-07-22_15-22-47\n",
" done: false\n",
" experiment_id: 08a7bc47c140405296e58a049cd2f6b4\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 93\n",
" iterations_since_restore: 94\n",
" mean_loss: -8.072800015147221\n",
" neg_mean_loss: 8.072800015147221\n",
" node_ip: 127.0.0.1\n",
" pid: 46229\n",
" time_since_restore: 10.099772214889526\n",
" time_this_iter_s: 0.10250687599182129\n",
" time_total_s: 10.099772214889526\n",
" timestamp: 1658499767\n",
" timesteps_since_restore: 0\n",
" training_iteration: 94\n",
" trial_id: bb3e0210\n",
" warmup_time: 0.002833843231201172\n",
" \n",
"Result for objective_bb3c8714:\n",
" date: 2022-07-22_15-22-47\n",
" done: false\n",
" experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 92\n",
" iterations_since_restore: 93\n",
" mean_loss: 2.53944357306219\n",
" neg_mean_loss: -2.53944357306219\n",
" node_ip: 127.0.0.1\n",
" pid: 46228\n",
" time_since_restore: 9.980368852615356\n",
" time_this_iter_s: 0.10658979415893555\n",
" time_total_s: 9.980368852615356\n",
" timestamp: 1658499767\n",
" timesteps_since_restore: 0\n",
" training_iteration: 93\n",
" trial_id: bb3c8714\n",
" warmup_time: 0.0027511119842529297\n",
" \n",
"Result for objective_bb3b2702:\n",
" date: 2022-07-22_15-22-47\n",
" done: false\n",
" experiment_id: 2f984120624b4e9aa3392fcc1b16b811\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 92\n",
" iterations_since_restore: 93\n",
" mean_loss: 10.464550264550265\n",
" neg_mean_loss: -10.464550264550265\n",
" node_ip: 127.0.0.1\n",
" pid: 46227\n",
" time_since_restore: 9.964121103286743\n",
" time_this_iter_s: 0.10547995567321777\n",
" time_total_s: 9.964121103286743\n",
" timestamp: 1658499767\n",
" timesteps_since_restore: 0\n",
" training_iteration: 93\n",
" trial_id: bb3b2702\n",
" warmup_time: 0.002672910690307617\n",
" \n",
"Result for objective_c19b9dde:\n",
" date: 2022-07-22_15-22-48\n",
" done: false\n",
" experiment_id: fab35a2f68c347c4bb815c268e9500a6\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 28.646596578422436\n",
" neg_mean_loss: -28.646596578422436\n",
" node_ip: 127.0.0.1\n",
" pid: 46248\n",
" time_since_restore: 0.10230708122253418\n",
" time_this_iter_s: 0.10230708122253418\n",
" time_total_s: 0.10230708122253418\n",
" timestamp: 1658499768\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: c19b9dde\n",
" warmup_time: 0.0026540756225585938\n",
" \n",
"Result for objective_bb3e0210:\n",
" date: 2022-07-22_15-22-49\n",
" done: true\n",
" experiment_id: 08a7bc47c140405296e58a049cd2f6b4\n",
" experiment_tag: 4_activation=tanh,height=-81.4501,steps=100,width=14.7830\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: -8.077146761191683\n",
" neg_mean_loss: 8.077146761191683\n",
" node_ip: 127.0.0.1\n",
" pid: 46229\n",
" time_since_restore: 11.219618082046509\n",
" time_this_iter_s: 0.1108238697052002\n",
" time_total_s: 11.219618082046509\n",
" timestamp: 1658499769\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: bb3e0210\n",
" warmup_time: 0.002833843231201172\n",
" \n",
"Result for objective_bb3c8714:\n",
" date: 2022-07-22_15-22-49\n",
" done: true\n",
" experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf\n",
" experiment_tag: 3_activation=relu,height=-90.7016,steps=100,width=0.5666\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 2.442852652712544\n",
" neg_mean_loss: -2.442852652712544\n",
" node_ip: 127.0.0.1\n",
" pid: 46228\n",
" time_since_restore: 11.276630163192749\n",
" time_this_iter_s: 0.10824012756347656\n",
" time_total_s: 11.276630163192749\n",
" timestamp: 1658499769\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: bb3c8714\n",
" warmup_time: 0.0027511119842529297\n",
" \n",
"Result for objective_bb3b2702:\n",
" date: 2022-07-22_15-22-49\n",
" done: true\n",
" experiment_id: 2f984120624b4e9aa3392fcc1b16b811\n",
" experiment_tag: 2_activation=relu,height=2,steps=100,width=4\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 10.446305418719211\n",
" neg_mean_loss: -10.446305418719211\n",
" node_ip: 127.0.0.1\n",
" pid: 46227\n",
" time_since_restore: 11.281576871871948\n",
" time_this_iter_s: 0.1068716049194336\n",
" time_total_s: 11.281576871871948\n",
" timestamp: 1658499769\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: bb3b2702\n",
" warmup_time: 0.002672910690307617\n",
" \n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_c3762426:\n",
" date: 2022-07-22_15-22-51\n",
" done: false\n",
" experiment_id: 034596724b2a4c9f86726e9a15fb386b\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 0.15367937759969408\n",
" neg_mean_loss: -0.15367937759969408\n",
" node_ip: 127.0.0.1\n",
" pid: 46253\n",
" time_since_restore: 0.1050560474395752\n",
" time_this_iter_s: 0.1050560474395752\n",
" time_total_s: 0.1050560474395752\n",
" timestamp: 1658499771\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: c3762426\n",
" warmup_time: 0.0028579235076904297\n",
" \n",
"Result for objective_c385528e:\n",
" date: 2022-07-22_15-22-51\n",
" done: false\n",
" experiment_id: 3044e8f1500f4b16aba04b72105c67be\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 14.516317633001044\n",
" neg_mean_loss: -14.516317633001044\n",
" node_ip: 127.0.0.1\n",
" pid: 46256\n",
" time_since_restore: 0.10402488708496094\n",
" time_this_iter_s: 0.10402488708496094\n",
" time_total_s: 0.10402488708496094\n",
" timestamp: 1658499771\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: c385528e\n",
" warmup_time: 0.0027010440826416016\n",
" \n",
"Result for objective_c387a7c8:\n",
" date: 2022-07-22_15-22-51\n",
" done: false\n",
" experiment_id: fc7f3d46b136437cab526c2fa6ea2944\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 14.294664350689382\n",
" neg_mean_loss: -14.294664350689382\n",
" node_ip: 127.0.0.1\n",
" pid: 46257\n",
" time_since_restore: 0.10461091995239258\n",
" time_this_iter_s: 0.10461091995239258\n",
" time_total_s: 0.10461091995239258\n",
" timestamp: 1658499771\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: c387a7c8\n",
" warmup_time: 0.0025899410247802734\n",
" \n",
"Result for objective_c19b9dde:\n",
" date: 2022-07-22_15-22-53\n",
" done: false\n",
" experiment_id: fab35a2f68c347c4bb815c268e9500a6\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 22.72706638821166\n",
" neg_mean_loss: -22.72706638821166\n",
" node_ip: 127.0.0.1\n",
" pid: 46248\n",
" time_since_restore: 5.118642330169678\n",
" time_this_iter_s: 0.10826611518859863\n",
" time_total_s: 5.118642330169678\n",
" timestamp: 1658499773\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: c19b9dde\n",
" warmup_time: 0.0026540756225585938\n",
" \n",
"Result for objective_c3762426:\n",
" date: 2022-07-22_15-22-56\n",
" done: false\n",
" experiment_id: 034596724b2a4c9f86726e9a15fb386b\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: -7.5742395372701825\n",
" neg_mean_loss: 7.5742395372701825\n",
" node_ip: 127.0.0.1\n",
" pid: 46253\n",
" time_since_restore: 5.14322304725647\n",
" time_this_iter_s: 0.10359907150268555\n",
" time_total_s: 5.14322304725647\n",
" timestamp: 1658499776\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: c3762426\n",
" warmup_time: 0.0028579235076904297\n",
" \n",
"Result for objective_c385528e:\n",
" date: 2022-07-22_15-22-56\n",
" done: false\n",
" experiment_id: 3044e8f1500f4b16aba04b72105c67be\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 4.704784733826337\n",
" neg_mean_loss: -4.704784733826337\n",
" node_ip: 127.0.0.1\n",
" pid: 46256\n",
" time_since_restore: 5.16935396194458\n",
" time_this_iter_s: 0.10719513893127441\n",
" time_total_s: 5.16935396194458\n",
" timestamp: 1658499776\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: c385528e\n",
" warmup_time: 0.0027010440826416016\n",
" \n",
"Result for objective_c387a7c8:\n",
" date: 2022-07-22_15-22-56\n",
" done: false\n",
" experiment_id: fc7f3d46b136437cab526c2fa6ea2944\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 4.553194134463865\n",
" neg_mean_loss: -4.553194134463865\n",
" node_ip: 127.0.0.1\n",
" pid: 46257\n",
" time_since_restore: 5.1458728313446045\n",
" time_this_iter_s: 0.10374999046325684\n",
" time_total_s: 5.1458728313446045\n",
" timestamp: 1658499776\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: c387a7c8\n",
" warmup_time: 0.0025899410247802734\n",
" \n",
"Result for objective_c19b9dde:\n",
" date: 2022-07-22_15-22-58\n",
" done: false\n",
" experiment_id: fab35a2f68c347c4bb815c268e9500a6\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 21.20978139671461\n",
" neg_mean_loss: -21.20978139671461\n",
" node_ip: 127.0.0.1\n",
" pid: 46248\n",
" time_since_restore: 10.183105230331421\n",
" time_this_iter_s: 0.10718917846679688\n",
" time_total_s: 10.183105230331421\n",
" timestamp: 1658499778\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: c19b9dde\n",
" warmup_time: 0.0026540756225585938\n",
" \n",
"Result for objective_c19b9dde:\n",
" date: 2022-07-22_15-22-59\n",
" done: true\n",
" experiment_id: fab35a2f68c347c4bb815c268e9500a6\n",
" experiment_tag: 5_activation=relu,height=86.4660,steps=100,width=0.3087\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 21.112246386473707\n",
" neg_mean_loss: -21.112246386473707\n",
" node_ip: 127.0.0.1\n",
" pid: 46248\n",
" time_since_restore: 10.722583055496216\n",
" time_this_iter_s: 0.10788774490356445\n",
" time_total_s: 10.722583055496216\n",
" timestamp: 1658499779\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: c19b9dde\n",
" warmup_time: 0.0026540756225585938\n",
" \n",
"Result for objective_c97678f8:\n",
" date: 2022-07-22_15-23-01\n",
" done: false\n",
" experiment_id: 7c4187f99b9246bdbae534ce20f33bfc\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 10.925214276212655\n",
" neg_mean_loss: -10.925214276212655\n",
" node_ip: 127.0.0.1\n",
" pid: 46272\n",
" time_since_restore: 0.1039581298828125\n",
" time_this_iter_s: 0.1039581298828125\n",
" time_total_s: 0.1039581298828125\n",
" timestamp: 1658499781\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: c97678f8\n",
" warmup_time: 0.002711057662963867\n",
" \n",
"Result for objective_c3762426:\n",
" date: 2022-07-22_15-23-01\n",
" done: false\n",
" experiment_id: 034596724b2a4c9f86726e9a15fb386b\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: -8.564680574270156\n",
" neg_mean_loss: 8.564680574270156\n",
" node_ip: 127.0.0.1\n",
" pid: 46253\n",
" time_since_restore: 10.190189123153687\n",
" time_this_iter_s: 0.10848307609558105\n",
" time_total_s: 10.190189123153687\n",
" timestamp: 1658499781\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: c3762426\n",
" warmup_time: 0.0028579235076904297\n",
" \n",
"Result for objective_c387a7c8:\n",
" date: 2022-07-22_15-23-01\n",
" done: false\n",
" experiment_id: fc7f3d46b136437cab526c2fa6ea2944\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 4.425622066068654\n",
" neg_mean_loss: -4.425622066068654\n",
" node_ip: 127.0.0.1\n",
" pid: 46257\n",
" time_since_restore: 10.181554794311523\n",
" time_this_iter_s: 0.10594582557678223\n",
" time_total_s: 10.181554794311523\n",
" timestamp: 1658499781\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: c387a7c8\n",
" warmup_time: 0.0025899410247802734\n",
" \n",
"Result for objective_c385528e:\n",
" date: 2022-07-22_15-23-01\n",
" done: false\n",
" experiment_id: 3044e8f1500f4b16aba04b72105c67be\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 4.611447627123378\n",
" neg_mean_loss: -4.611447627123378\n",
" node_ip: 127.0.0.1\n",
" pid: 46256\n",
" time_since_restore: 10.214791059494019\n",
" time_this_iter_s: 0.10866808891296387\n",
" time_total_s: 10.214791059494019\n",
" timestamp: 1658499781\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: c385528e\n",
" warmup_time: 0.0027010440826416016\n",
" \n",
"Result for objective_c3762426:\n",
" date: 2022-07-22_15-23-02\n",
" done: true\n",
" experiment_id: 034596724b2a4c9f86726e9a15fb386b\n",
" experiment_tag: 6_activation=tanh,height=-98.4632,steps=100,width=0.7237\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: -8.621481572769046\n",
" neg_mean_loss: 8.621481572769046\n",
" node_ip: 127.0.0.1\n",
" pid: 46253\n",
" time_since_restore: 10.72708010673523\n",
" time_this_iter_s: 0.10719490051269531\n",
" time_total_s: 10.72708010673523\n",
" timestamp: 1658499782\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: c3762426\n",
" warmup_time: 0.0028579235076904297\n",
" \n",
"Result for objective_c387a7c8:\n",
" date: 2022-07-22_15-23-02\n",
" done: true\n",
" experiment_id: fc7f3d46b136437cab526c2fa6ea2944\n",
" experiment_tag: 8_activation=tanh,height=42.9466,steps=100,width=8.0171\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 4.419090335709762\n",
" neg_mean_loss: -4.419090335709762\n",
" node_ip: 127.0.0.1\n",
" pid: 46257\n",
" time_since_restore: 10.721372842788696\n",
" time_this_iter_s: 0.10873699188232422\n",
" time_total_s: 10.721372842788696\n",
" timestamp: 1658499782\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: c387a7c8\n",
" warmup_time: 0.0025899410247802734\n",
" \n",
"Result for objective_c385528e:\n",
" date: 2022-07-22_15-23-02\n",
" done: true\n",
" experiment_id: 3044e8f1500f4b16aba04b72105c67be\n",
" experiment_tag: 7_activation=relu,height=-54.8368,steps=100,width=11.0765\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 4.606686500095965\n",
" neg_mean_loss: -4.606686500095965\n",
" node_ip: 127.0.0.1\n",
" pid: 46256\n",
" time_since_restore: 10.749844074249268\n",
" time_this_iter_s: 0.10710501670837402\n",
" time_total_s: 10.749844074249268\n",
" timestamp: 1658499782\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: c385528e\n",
" warmup_time: 0.0027010440826416016\n",
" \n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_cb6180fe:\n",
" date: 2022-07-22_15-23-04\n",
" done: false\n",
" experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 24.166473702046588\n",
" neg_mean_loss: -24.166473702046588\n",
" node_ip: 127.0.0.1\n",
" pid: 46277\n",
" time_since_restore: 0.10389995574951172\n",
" time_this_iter_s: 0.10389995574951172\n",
" time_total_s: 0.10389995574951172\n",
" timestamp: 1658499784\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: cb6180fe\n",
" warmup_time: 0.0027260780334472656\n",
" \n",
"Result for objective_c97678f8:\n",
" date: 2022-07-22_15-23-06\n",
" done: false\n",
" experiment_id: 7c4187f99b9246bdbae534ce20f33bfc\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 31\n",
" iterations_since_restore: 32\n",
" mean_loss: 1.1287174007116527\n",
" neg_mean_loss: -1.1287174007116527\n",
" node_ip: 127.0.0.1\n",
" pid: 46272\n",
" time_since_restore: 5.140314340591431\n",
" time_this_iter_s: 0.10663819313049316\n",
" time_total_s: 5.140314340591431\n",
" timestamp: 1658499786\n",
" timesteps_since_restore: 0\n",
" training_iteration: 32\n",
" trial_id: c97678f8\n",
" warmup_time: 0.002711057662963867\n",
" \n",
"Result for objective_cb6180fe:\n",
" date: 2022-07-22_15-23-09\n",
" done: false\n",
" experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 14.42370878193022\n",
" neg_mean_loss: -14.42370878193022\n",
" node_ip: 127.0.0.1\n",
" pid: 46277\n",
" time_since_restore: 5.139087915420532\n",
" time_this_iter_s: 0.10686802864074707\n",
" time_total_s: 5.139087915420532\n",
" timestamp: 1658499789\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: cb6180fe\n",
" warmup_time: 0.0027260780334472656\n",
" \n",
"Result for objective_c97678f8:\n",
" date: 2022-07-22_15-23-11\n",
" done: false\n",
" experiment_id: 7c4187f99b9246bdbae534ce20f33bfc\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 78\n",
" iterations_since_restore: 79\n",
" mean_loss: 1.0070978091228147\n",
" neg_mean_loss: -1.0070978091228147\n",
" node_ip: 127.0.0.1\n",
" pid: 46272\n",
" time_since_restore: 10.206932067871094\n",
" time_this_iter_s: 0.10774993896484375\n",
" time_total_s: 10.206932067871094\n",
" timestamp: 1658499791\n",
" timesteps_since_restore: 0\n",
" training_iteration: 79\n",
" trial_id: c97678f8\n",
" warmup_time: 0.002711057662963867\n",
" \n",
"Result for objective_c97678f8:\n",
" date: 2022-07-22_15-23-14\n",
" done: true\n",
" experiment_id: 7c4187f99b9246bdbae534ce20f33bfc\n",
" experiment_tag: 9_activation=tanh,height=9.2521,steps=100,width=15.5288\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 0.9898408262377837\n",
" neg_mean_loss: -0.9898408262377837\n",
" node_ip: 127.0.0.1\n",
" pid: 46272\n",
" time_since_restore: 12.459095239639282\n",
" time_this_iter_s: 0.10785508155822754\n",
" time_total_s: 12.459095239639282\n",
" timestamp: 1658499794\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: c97678f8\n",
" warmup_time: 0.002711057662963867\n",
" \n",
"Result for objective_cb6180fe:\n",
" date: 2022-07-22_15-23-15\n",
" done: false\n",
" experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 14.296767042885353\n",
" neg_mean_loss: -14.296767042885353\n",
" node_ip: 127.0.0.1\n",
" pid: 46277\n",
" time_since_restore: 10.220713138580322\n",
" time_this_iter_s: 0.10750126838684082\n",
" time_total_s: 10.220713138580322\n",
" timestamp: 1658499795\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: cb6180fe\n",
" warmup_time: 0.0027260780334472656\n",
" \n",
"Result for objective_cb6180fe:\n",
" date: 2022-07-22_15-23-15\n",
" done: true\n",
" experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea\n",
" experiment_tag: 10_activation=relu,height=41.6647,steps=100,width=8.0585\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 14.29026803363612\n",
" neg_mean_loss: -14.29026803363612\n",
" node_ip: 127.0.0.1\n",
" pid: 46277\n",
" time_since_restore: 10.752035856246948\n",
" time_this_iter_s: 0.1066899299621582\n",
" time_total_s: 10.752035856246948\n",
" timestamp: 1658499795\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: cb6180fe\n",
" warmup_time: 0.0027260780334472656\n",
" \n"
]
}
],
"source": [
"tuner = tune.Tuner(\n",
" objective,\n",
" tune_config=tune.TuneConfig(\n",
" metric=\"mean_loss\",\n",
" mode=\"min\",\n",
" search_alg=algo,\n",
" num_samples=num_samples,\n",
" ),\n",
" param_space=search_space,\n",
")\n",
"results = tuner.fit()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "ccfe15e2",
"metadata": {},
"source": [
"We take another look at the optimal hyperparamters."
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "fcfa0c2e",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best hyperparameters found were: {'steps': 100, 'width': 0.7236705053153614, 'height': -98.46320622400306, 'activation': 'tanh'}\n"
]
}
],
"source": [
"print(\"Best hyperparameters found were: \", results.get_best_result().config)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "88080576",
"metadata": {},
"source": [
"## Conditional search spaces \n",
"\n",
"Sometimes we may want to build a more complicated search space that has conditional dependencies on other hyperparameters. In this case, we pass a define-by-run function to the `search_alg` argument in `ray.tune()`."
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "f0acc2fc",
"metadata": {
"lines_to_next_cell": 0
},
"outputs": [],
"source": [
"def define_by_run_func(trial) -> Optional[Dict[str, Any]]:\n",
" \"\"\"Define-by-run function to create the search space.\n",
"\n",
" Ensure no actual computation takes place here. That should go into\n",
" the trainable passed to ``Tuner()`` (in this example, that's\n",
" ``objective``).\n",
"\n",
" For more information, see https://optuna.readthedocs.io/en/stable\\\n",
" /tutorial/10_key_features/002_configurations.html\n",
"\n",
" This function should either return None or a dict with constant values.\n",
" \"\"\"\n",
"\n",
" activation = trial.suggest_categorical(\"activation\", [\"relu\", \"tanh\"])\n",
"\n",
" # Define-by-run allows for conditional search spaces.\n",
" if activation == \"relu\":\n",
" trial.suggest_float(\"width\", 0, 20)\n",
" trial.suggest_float(\"height\", -100, 100)\n",
" else:\n",
" trial.suggest_float(\"width\", -1, 21)\n",
" trial.suggest_float(\"height\", -101, 101)\n",
" \n",
" # Return all constants in a dictionary.\n",
" return {\"steps\": 100}"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "4c9d0945",
"metadata": {},
"source": [
"As before, we create the `search_alg` from `OptunaSearch` and `ConcurrencyLimiter`, this time we define the scope of search via the `space` argument and provide no initialization. We also must specific metric and mode when using `space`. "
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "906f9ffc",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[32m[I 2022-07-22 15:23:15,784]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n"
]
}
],
"source": [
"searcher = OptunaSearch(space=define_by_run_func, metric=\"mean_loss\", mode=\"min\")\n",
"algo = ConcurrencyLimiter(searcher, max_concurrent=4)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "fea9399c",
"metadata": {},
"source": [
"Running the experiment with a define-by-run search space:"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "bf0ee932",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"== Status ==
Current time: 2022-07-22 15:23:58 (running for 00:00:43.15)
Memory usage on this node: 10.4/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects
Result logdir: /Users/kai/ray_results/optuna_example_3
Number of trials: 10/10 (10 TERMINATED)
\n",
"\n",
"Trial name | status | loc | activation | height | steps | width | loss | iter | total time (s) | iterations | neg_mean_loss |
\n",
"\n",
"\n",
"objective_d363eed6 | TERMINATED | 127.0.0.1:46290 | relu | 96.9804 | 100 | 6.47293 | 19.8517 | 100 | 10.735 | 99 | -19.8517 |
\n",
"objective_d4f3a700 | TERMINATED | 127.0.0.1:46298 | tanh | -76.4387 | 100 | 17.7358 | -7.58724 | 100 | 11.396 | 99 | 7.58724 |
\n",
"objective_d4f50ce4 | TERMINATED | 127.0.0.1:46299 | relu | -89.8609 | 100 | 6.28321 | 1.17212 | 100 | 11.4182 | 99 | -1.17212 |
\n",
"objective_d4f67908 | TERMINATED | 127.0.0.1:46300 | relu | 67.9266 | 100 | 14.0599 | 16.864 | 100 | 11.341 | 99 | -16.864 |
\n",
"objective_db5c0402 | TERMINATED | 127.0.0.1:46317 | tanh | -54.13 | 100 | 11.4712 | -5.32571 | 100 | 10.705 | 99 | 5.32571 |
\n",
"objective_dd487b56 | TERMINATED | 127.0.0.1:46322 | relu | -25.9818 | 100 | 0.758497 | 8.57703 | 100 | 10.6916 | 99 | -8.57703 |
\n",
"objective_dd4b4e94 | TERMINATED | 127.0.0.1:46323 | relu | 54.6085 | 100 | 1.2361 | 16.2163 | 100 | 10.6711 | 99 | -16.2163 |
\n",
"objective_dd5a8bca | TERMINATED | 127.0.0.1:46328 | tanh | 5.22131 | 100 | 17.0952 | 0.580871 | 100 | 10.7017 | 99 | -0.580871 |
\n",
"objective_e3482178 | TERMINATED | 127.0.0.1:46341 | tanh | 78.844 | 100 | 15.5079 | 7.94912 | 100 | 12.6417 | 99 | -7.94912 |
\n",
"objective_e532d6e0 | TERMINATED | 127.0.0.1:46353 | tanh | -66.9988 | 100 | 7.15087 | -6.56059 | 100 | 10.7486 | 99 | 6.56059 |
\n",
"\n",
"
"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_d363eed6:\n",
" date: 2022-07-22_15-23-18\n",
" done: false\n",
" experiment_id: af8f846e60254bc794827e76909df4f0\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 29.698044266420816\n",
" neg_mean_loss: -29.698044266420816\n",
" node_ip: 127.0.0.1\n",
" pid: 46290\n",
" time_since_restore: 0.10497498512268066\n",
" time_this_iter_s: 0.10497498512268066\n",
" time_total_s: 0.10497498512268066\n",
" timestamp: 1658499798\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: d363eed6\n",
" warmup_time: 0.0027971267700195312\n",
" \n",
"Result for objective_d4f3a700:\n",
" date: 2022-07-22_15-23-21\n",
" done: false\n",
" experiment_id: 52e5f2a557cf4466939b6c80bb9ff905\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 2.3561338711040003\n",
" neg_mean_loss: -2.3561338711040003\n",
" node_ip: 127.0.0.1\n",
" pid: 46298\n",
" time_since_restore: 0.10145115852355957\n",
" time_this_iter_s: 0.10145115852355957\n",
" time_total_s: 0.10145115852355957\n",
" timestamp: 1658499801\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: d4f3a700\n",
" warmup_time: 0.0033049583435058594\n",
" \n",
"Result for objective_d4f67908:\n",
" date: 2022-07-22_15-23-21\n",
" done: false\n",
" experiment_id: 230c6da41a834386be865bb19780b3a5\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 26.792658178391687\n",
" neg_mean_loss: -26.792658178391687\n",
" node_ip: 127.0.0.1\n",
" pid: 46300\n",
" time_since_restore: 0.10428690910339355\n",
" time_this_iter_s: 0.10428690910339355\n",
" time_total_s: 0.10428690910339355\n",
" timestamp: 1658499801\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: d4f67908\n",
" warmup_time: 0.0030798912048339844\n",
" \n",
"Result for objective_d4f50ce4:\n",
" date: 2022-07-22_15-23-21\n",
" done: false\n",
" experiment_id: f2396d75f50b417f9fae16d1c9ddcda4\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 11.013906266700227\n",
" neg_mean_loss: -11.013906266700227\n",
" node_ip: 127.0.0.1\n",
" pid: 46299\n",
" time_since_restore: 0.1020050048828125\n",
" time_this_iter_s: 0.1020050048828125\n",
" time_total_s: 0.1020050048828125\n",
" timestamp: 1658499801\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: d4f50ce4\n",
" warmup_time: 0.0027348995208740234\n",
" \n",
"Result for objective_d363eed6:\n",
" date: 2022-07-22_15-23-23\n",
" done: false\n",
" experiment_id: af8f846e60254bc794827e76909df4f0\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 20.016284953435374\n",
" neg_mean_loss: -20.016284953435374\n",
" node_ip: 127.0.0.1\n",
" pid: 46290\n",
" time_since_restore: 5.138504981994629\n",
" time_this_iter_s: 0.10720086097717285\n",
" time_total_s: 5.138504981994629\n",
" timestamp: 1658499803\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: d363eed6\n",
" warmup_time: 0.0027971267700195312\n",
" \n",
"Result for objective_d4f3a700:\n",
" date: 2022-07-22_15-23-26\n",
" done: false\n",
" experiment_id: 52e5f2a557cf4466939b6c80bb9ff905\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: -7.525324371061028\n",
" neg_mean_loss: 7.525324371061028\n",
" node_ip: 127.0.0.1\n",
" pid: 46298\n",
" time_since_restore: 5.153075218200684\n",
" time_this_iter_s: 0.10515189170837402\n",
" time_total_s: 5.153075218200684\n",
" timestamp: 1658499806\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: d4f3a700\n",
" warmup_time: 0.0033049583435058594\n",
" \n",
"Result for objective_d4f67908:\n",
" date: 2022-07-22_15-23-26\n",
" done: false\n",
" experiment_id: 230c6da41a834386be865bb19780b3a5\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 16.94173032408916\n",
" neg_mean_loss: -16.94173032408916\n",
" node_ip: 127.0.0.1\n",
" pid: 46300\n",
" time_since_restore: 5.153704881668091\n",
" time_this_iter_s: 0.10537075996398926\n",
" time_total_s: 5.153704881668091\n",
" timestamp: 1658499806\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: d4f67908\n",
" warmup_time: 0.0030798912048339844\n",
" \n",
"Result for objective_d4f50ce4:\n",
" date: 2022-07-22_15-23-26\n",
" done: false\n",
" experiment_id: f2396d75f50b417f9fae16d1c9ddcda4\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 1.3414410369095506\n",
" neg_mean_loss: -1.3414410369095506\n",
" node_ip: 127.0.0.1\n",
" pid: 46299\n",
" time_since_restore: 5.1622138023376465\n",
" time_this_iter_s: 0.10337996482849121\n",
" time_total_s: 5.1622138023376465\n",
" timestamp: 1658499806\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: d4f50ce4\n",
" warmup_time: 0.0027348995208740234\n",
" \n",
"Result for objective_d363eed6:\n",
" date: 2022-07-22_15-23-28\n",
" done: false\n",
" experiment_id: af8f846e60254bc794827e76909df4f0\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 19.85973747786284\n",
" neg_mean_loss: -19.85973747786284\n",
" node_ip: 127.0.0.1\n",
" pid: 46290\n",
" time_since_restore: 10.1943039894104\n",
" time_this_iter_s: 0.12109899520874023\n",
" time_total_s: 10.1943039894104\n",
" timestamp: 1658499808\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: d363eed6\n",
" warmup_time: 0.0027971267700195312\n",
" \n",
"Result for objective_d363eed6:\n",
" date: 2022-07-22_15-23-29\n",
" done: true\n",
" experiment_id: af8f846e60254bc794827e76909df4f0\n",
" experiment_tag: 1_activation=relu,height=96.9804,steps=100,width=6.4729\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 19.851696631549423\n",
" neg_mean_loss: -19.851696631549423\n",
" node_ip: 127.0.0.1\n",
" pid: 46290\n",
" time_since_restore: 10.734967947006226\n",
" time_this_iter_s: 0.10534882545471191\n",
" time_total_s: 10.734967947006226\n",
" timestamp: 1658499809\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: d363eed6\n",
" warmup_time: 0.0027971267700195312\n",
" \n",
"Result for objective_d4f67908:\n",
" date: 2022-07-22_15-23-31\n",
" done: false\n",
" experiment_id: 230c6da41a834386be865bb19780b3a5\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 93\n",
" iterations_since_restore: 94\n",
" mean_loss: 16.86855533914186\n",
" neg_mean_loss: -16.86855533914186\n",
" node_ip: 127.0.0.1\n",
" pid: 46300\n",
" time_since_restore: 10.096334218978882\n",
" time_this_iter_s: 0.10534310340881348\n",
" time_total_s: 10.096334218978882\n",
" timestamp: 1658499811\n",
" timesteps_since_restore: 0\n",
" training_iteration: 94\n",
" trial_id: d4f67908\n",
" warmup_time: 0.0030798912048339844\n",
" \n",
"Result for objective_d4f50ce4:\n",
" date: 2022-07-22_15-23-31\n",
" done: false\n",
" experiment_id: f2396d75f50b417f9fae16d1c9ddcda4\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 92\n",
" iterations_since_restore: 93\n",
" mean_loss: 1.183958166642718\n",
" neg_mean_loss: -1.183958166642718\n",
" node_ip: 127.0.0.1\n",
" pid: 46299\n",
" time_since_restore: 9.976859092712402\n",
" time_this_iter_s: 0.1057121753692627\n",
" time_total_s: 9.976859092712402\n",
" timestamp: 1658499811\n",
" timesteps_since_restore: 0\n",
" training_iteration: 93\n",
" trial_id: d4f50ce4\n",
" warmup_time: 0.0027348995208740234\n",
" \n",
"Result for objective_d4f3a700:\n",
" date: 2022-07-22_15-23-31\n",
" done: false\n",
" experiment_id: 52e5f2a557cf4466939b6c80bb9ff905\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 93\n",
" iterations_since_restore: 94\n",
" mean_loss: -7.583604593881161\n",
" neg_mean_loss: 7.583604593881161\n",
" node_ip: 127.0.0.1\n",
" pid: 46298\n",
" time_since_restore: 10.091833114624023\n",
" time_this_iter_s: 0.10643887519836426\n",
" time_total_s: 10.091833114624023\n",
" timestamp: 1658499811\n",
" timesteps_since_restore: 0\n",
" training_iteration: 94\n",
" trial_id: d4f3a700\n",
" warmup_time: 0.0033049583435058594\n",
" \n",
"Result for objective_db5c0402:\n",
" date: 2022-07-22_15-23-31\n",
" done: false\n",
" experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 4.587004175501654\n",
" neg_mean_loss: -4.587004175501654\n",
" node_ip: 127.0.0.1\n",
" pid: 46317\n",
" time_since_restore: 0.10449004173278809\n",
" time_this_iter_s: 0.10449004173278809\n",
" time_total_s: 0.10449004173278809\n",
" timestamp: 1658499811\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: db5c0402\n",
" warmup_time: 0.003406047821044922\n",
" \n",
"Result for objective_d4f67908:\n",
" date: 2022-07-22_15-23-32\n",
" done: true\n",
" experiment_id: 230c6da41a834386be865bb19780b3a5\n",
" experiment_tag: 4_activation=relu,height=67.9266,steps=100,width=14.0599\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 16.86398832185618\n",
" neg_mean_loss: -16.86398832185618\n",
" node_ip: 127.0.0.1\n",
" pid: 46300\n",
" time_since_restore: 11.341041088104248\n",
" time_this_iter_s: 0.10513019561767578\n",
" time_total_s: 11.341041088104248\n",
" timestamp: 1658499812\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: d4f67908\n",
" warmup_time: 0.0030798912048339844\n",
" \n",
"Result for objective_d4f3a700:\n",
" date: 2022-07-22_15-23-32\n",
" done: true\n",
" experiment_id: 52e5f2a557cf4466939b6c80bb9ff905\n",
" experiment_tag: 2_activation=tanh,height=-76.4387,steps=100,width=17.7358\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: -7.587236125634034\n",
" neg_mean_loss: 7.587236125634034\n",
" node_ip: 127.0.0.1\n",
" pid: 46298\n",
" time_since_restore: 11.396040916442871\n",
" time_this_iter_s: 0.10666394233703613\n",
" time_total_s: 11.396040916442871\n",
" timestamp: 1658499812\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: d4f3a700\n",
" warmup_time: 0.0033049583435058594\n",
" \n",
"Result for objective_d4f50ce4:\n",
" date: 2022-07-22_15-23-32\n",
" done: true\n",
" experiment_id: f2396d75f50b417f9fae16d1c9ddcda4\n",
" experiment_tag: 3_activation=relu,height=-89.8609,steps=100,width=6.2832\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 1.1721245345451354\n",
" neg_mean_loss: -1.1721245345451354\n",
" node_ip: 127.0.0.1\n",
" pid: 46299\n",
" time_since_restore: 11.418162822723389\n",
" time_this_iter_s: 0.11828303337097168\n",
" time_total_s: 11.418162822723389\n",
" timestamp: 1658499812\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: d4f50ce4\n",
" warmup_time: 0.0027348995208740234\n",
" \n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_dd4b4e94:\n",
" date: 2022-07-22_15-23-35\n",
" done: false\n",
" experiment_id: c6036d4a26674f86946fd7c8c895e85c\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 25.460850093028082\n",
" neg_mean_loss: -25.460850093028082\n",
" node_ip: 127.0.0.1\n",
" pid: 46323\n",
" time_since_restore: 0.10411691665649414\n",
" time_this_iter_s: 0.10411691665649414\n",
" time_total_s: 0.10411691665649414\n",
" timestamp: 1658499815\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: dd4b4e94\n",
" warmup_time: 0.003142118453979492\n",
" \n",
"Result for objective_dd487b56:\n",
" date: 2022-07-22_15-23-35\n",
" done: false\n",
" experiment_id: 75e118a5733e454fab54f4a20c036852\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 17.401819651158622\n",
" neg_mean_loss: -17.401819651158622\n",
" node_ip: 127.0.0.1\n",
" pid: 46322\n",
" time_since_restore: 0.1048891544342041\n",
" time_this_iter_s: 0.1048891544342041\n",
" time_total_s: 0.1048891544342041\n",
" timestamp: 1658499815\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: dd487b56\n",
" warmup_time: 0.0028679370880126953\n",
" \n",
"Result for objective_dd5a8bca:\n",
" date: 2022-07-22_15-23-35\n",
" done: false\n",
" experiment_id: 775683e6d95a4705846a37e809b474d7\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 10.522131438315931\n",
" neg_mean_loss: -10.522131438315931\n",
" node_ip: 127.0.0.1\n",
" pid: 46328\n",
" time_since_restore: 0.10494732856750488\n",
" time_this_iter_s: 0.10494732856750488\n",
" time_total_s: 0.10494732856750488\n",
" timestamp: 1658499815\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: dd5a8bca\n",
" warmup_time: 0.002931833267211914\n",
" \n",
"Result for objective_db5c0402:\n",
" date: 2022-07-22_15-23-36\n",
" done: false\n",
" experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: -5.230895562358142\n",
" neg_mean_loss: 5.230895562358142\n",
" node_ip: 127.0.0.1\n",
" pid: 46317\n",
" time_since_restore: 5.120880842208862\n",
" time_this_iter_s: 0.1076059341430664\n",
" time_total_s: 5.120880842208862\n",
" timestamp: 1658499816\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: db5c0402\n",
" warmup_time: 0.003406047821044922\n",
" \n",
"Result for objective_dd4b4e94:\n",
" date: 2022-07-22_15-23-40\n",
" done: false\n",
" experiment_id: c6036d4a26674f86946fd7c8c895e85c\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 16.929346028466988\n",
" neg_mean_loss: -16.929346028466988\n",
" node_ip: 127.0.0.1\n",
" pid: 46323\n",
" time_since_restore: 5.137623071670532\n",
" time_this_iter_s: 0.10670709609985352\n",
" time_total_s: 5.137623071670532\n",
" timestamp: 1658499820\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: dd4b4e94\n",
" warmup_time: 0.003142118453979492\n",
" \n",
"Result for objective_dd487b56:\n",
" date: 2022-07-22_15-23-40\n",
" done: false\n",
" experiment_id: 75e118a5733e454fab54f4a20c036852\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 9.592430453048937\n",
" neg_mean_loss: -9.592430453048937\n",
" node_ip: 127.0.0.1\n",
" pid: 46322\n",
" time_since_restore: 5.140692949295044\n",
" time_this_iter_s: 0.1086587905883789\n",
" time_total_s: 5.140692949295044\n",
" timestamp: 1658499820\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: dd487b56\n",
" warmup_time: 0.0028679370880126953\n",
" \n",
"Result for objective_dd5a8bca:\n",
" date: 2022-07-22_15-23-40\n",
" done: false\n",
" experiment_id: 775683e6d95a4705846a37e809b474d7\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" mean_loss: 0.6450610360030851\n",
" neg_mean_loss: -0.6450610360030851\n",
" node_ip: 127.0.0.1\n",
" pid: 46328\n",
" time_since_restore: 5.138496160507202\n",
" time_this_iter_s: 0.10809993743896484\n",
" time_total_s: 5.138496160507202\n",
" timestamp: 1658499820\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: dd5a8bca\n",
" warmup_time: 0.002931833267211914\n",
" \n",
"Result for objective_db5c0402:\n",
" date: 2022-07-22_15-23-41\n",
" done: false\n",
" experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: -5.321109063262834\n",
" neg_mean_loss: 5.321109063262834\n",
" node_ip: 127.0.0.1\n",
" pid: 46317\n",
" time_since_restore: 10.162945985794067\n",
" time_this_iter_s: 0.10671615600585938\n",
" time_total_s: 10.162945985794067\n",
" timestamp: 1658499821\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: db5c0402\n",
" warmup_time: 0.003406047821044922\n",
" \n",
"Result for objective_db5c0402:\n",
" date: 2022-07-22_15-23-42\n",
" done: true\n",
" experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8\n",
" experiment_tag: 5_activation=tanh,height=-54.1300,steps=100,width=11.4712\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: -5.325709301325668\n",
" neg_mean_loss: 5.325709301325668\n",
" node_ip: 127.0.0.1\n",
" pid: 46317\n",
" time_since_restore: 10.704953908920288\n",
" time_this_iter_s: 0.1082158088684082\n",
" time_total_s: 10.704953908920288\n",
" timestamp: 1658499822\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: db5c0402\n",
" warmup_time: 0.003406047821044922\n",
" \n",
"Result for objective_dd4b4e94:\n",
" date: 2022-07-22_15-23-45\n",
" done: false\n",
" experiment_id: c6036d4a26674f86946fd7c8c895e85c\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 16.253282229309562\n",
" neg_mean_loss: -16.253282229309562\n",
" node_ip: 127.0.0.1\n",
" pid: 46323\n",
" time_since_restore: 10.140441179275513\n",
" time_this_iter_s: 0.10501408576965332\n",
" time_total_s: 10.140441179275513\n",
" timestamp: 1658499825\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: dd4b4e94\n",
" warmup_time: 0.003142118453979492\n",
" \n",
"Result for objective_dd487b56:\n",
" date: 2022-07-22_15-23-45\n",
" done: false\n",
" experiment_id: 75e118a5733e454fab54f4a20c036852\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 8.631851056235014\n",
" neg_mean_loss: -8.631851056235014\n",
" node_ip: 127.0.0.1\n",
" pid: 46322\n",
" time_since_restore: 10.156053066253662\n",
" time_this_iter_s: 0.10682201385498047\n",
" time_total_s: 10.156053066253662\n",
" timestamp: 1658499825\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: dd487b56\n",
" warmup_time: 0.0028679370880126953\n",
" \n",
"Result for objective_e3482178:\n",
" date: 2022-07-22_15-23-45\n",
" done: false\n",
" experiment_id: 6709b686be234eb996457210f3efbe08\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 17.884403838898738\n",
" neg_mean_loss: -17.884403838898738\n",
" node_ip: 127.0.0.1\n",
" pid: 46341\n",
" time_since_restore: 0.10499191284179688\n",
" time_this_iter_s: 0.10499191284179688\n",
" time_total_s: 0.10499191284179688\n",
" timestamp: 1658499825\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: e3482178\n",
" warmup_time: 0.002877950668334961\n",
" \n",
"Result for objective_dd5a8bca:\n",
" date: 2022-07-22_15-23-45\n",
" done: false\n",
" experiment_id: 775683e6d95a4705846a37e809b474d7\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" mean_loss: 0.583976365762005\n",
" neg_mean_loss: -0.583976365762005\n",
" node_ip: 127.0.0.1\n",
" pid: 46328\n",
" time_since_restore: 10.165286302566528\n",
" time_this_iter_s: 0.10989904403686523\n",
" time_total_s: 10.165286302566528\n",
" timestamp: 1658499825\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: dd5a8bca\n",
" warmup_time: 0.002931833267211914\n",
" \n",
"Result for objective_dd4b4e94:\n",
" date: 2022-07-22_15-23-45\n",
" done: true\n",
" experiment_id: c6036d4a26674f86946fd7c8c895e85c\n",
" experiment_tag: 7_activation=relu,height=54.6085,steps=100,width=1.2361\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 16.216283787762706\n",
" neg_mean_loss: -16.216283787762706\n",
" node_ip: 127.0.0.1\n",
" pid: 46323\n",
" time_since_restore: 10.67109203338623\n",
" time_this_iter_s: 0.1058969497680664\n",
" time_total_s: 10.67109203338623\n",
" timestamp: 1658499825\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: dd4b4e94\n",
" warmup_time: 0.003142118453979492\n",
" \n",
"Result for objective_dd487b56:\n",
" date: 2022-07-22_15-23-45\n",
" done: true\n",
" experiment_id: 75e118a5733e454fab54f4a20c036852\n",
" experiment_tag: 6_activation=relu,height=-25.9818,steps=100,width=0.7585\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 8.577028987245935\n",
" neg_mean_loss: -8.577028987245935\n",
" node_ip: 127.0.0.1\n",
" pid: 46322\n",
" time_since_restore: 10.691627979278564\n",
" time_this_iter_s: 0.10821914672851562\n",
" time_total_s: 10.691627979278564\n",
" timestamp: 1658499825\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: dd487b56\n",
" warmup_time: 0.0028679370880126953\n",
" \n",
"Result for objective_dd5a8bca:\n",
" date: 2022-07-22_15-23-45\n",
" done: true\n",
" experiment_id: 775683e6d95a4705846a37e809b474d7\n",
" experiment_tag: 8_activation=tanh,height=5.2213,steps=100,width=17.0952\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 0.5808712318418436\n",
" neg_mean_loss: -0.5808712318418436\n",
" node_ip: 127.0.0.1\n",
" pid: 46328\n",
" time_since_restore: 10.701741218566895\n",
" time_this_iter_s: 0.11321616172790527\n",
" time_total_s: 10.701741218566895\n",
" timestamp: 1658499825\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: dd5a8bca\n",
" warmup_time: 0.002931833267211914\n",
" \n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for objective_e532d6e0:\n",
" date: 2022-07-22_15-23-48\n",
" done: false\n",
" experiment_id: 34d05fd425474b82a450c22c2a063843\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" mean_loss: 3.30012197692201\n",
" neg_mean_loss: -3.30012197692201\n",
" node_ip: 127.0.0.1\n",
" pid: 46353\n",
" time_since_restore: 0.10400509834289551\n",
" time_this_iter_s: 0.10400509834289551\n",
" time_total_s: 0.10400509834289551\n",
" timestamp: 1658499828\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: e532d6e0\n",
" warmup_time: 0.0028901100158691406\n",
" \n",
"Result for objective_e3482178:\n",
" date: 2022-07-22_15-23-50\n",
" done: false\n",
" experiment_id: 6709b686be234eb996457210f3efbe08\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 29\n",
" iterations_since_restore: 30\n",
" mean_loss: 8.10192352359083\n",
" neg_mean_loss: -8.10192352359083\n",
" node_ip: 127.0.0.1\n",
" pid: 46341\n",
" time_since_restore: 5.183844089508057\n",
" time_this_iter_s: 0.10574007034301758\n",
" time_total_s: 5.183844089508057\n",
" timestamp: 1658499830\n",
" timesteps_since_restore: 0\n",
" training_iteration: 30\n",
" trial_id: e3482178\n",
" warmup_time: 0.002877950668334961\n",
" \n",
"Result for objective_e532d6e0:\n",
" date: 2022-07-22_15-23-53\n",
" done: false\n",
" experiment_id: 34d05fd425474b82a450c22c2a063843\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 48\n",
" iterations_since_restore: 49\n",
" mean_loss: -6.416785967274913\n",
" neg_mean_loss: 6.416785967274913\n",
" node_ip: 127.0.0.1\n",
" pid: 46353\n",
" time_since_restore: 5.212116003036499\n",
" time_this_iter_s: 0.10714387893676758\n",
" time_total_s: 5.212116003036499\n",
" timestamp: 1658499833\n",
" timesteps_since_restore: 0\n",
" training_iteration: 49\n",
" trial_id: e532d6e0\n",
" warmup_time: 0.0028901100158691406\n",
" \n",
"Result for objective_e3482178:\n",
" date: 2022-07-22_15-23-55\n",
" done: false\n",
" experiment_id: 6709b686be234eb996457210f3efbe08\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 76\n",
" iterations_since_restore: 77\n",
" mean_loss: 7.968536513457016\n",
" neg_mean_loss: -7.968536513457016\n",
" node_ip: 127.0.0.1\n",
" pid: 46341\n",
" time_since_restore: 10.190052032470703\n",
" time_this_iter_s: 0.10843491554260254\n",
" time_total_s: 10.190052032470703\n",
" timestamp: 1658499835\n",
" timesteps_since_restore: 0\n",
" training_iteration: 77\n",
" trial_id: e3482178\n",
" warmup_time: 0.002877950668334961\n",
" \n",
"Result for objective_e3482178:\n",
" date: 2022-07-22_15-23-57\n",
" done: true\n",
" experiment_id: 6709b686be234eb996457210f3efbe08\n",
" experiment_tag: 9_activation=tanh,height=78.8440,steps=100,width=15.5079\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: 7.9491170267942515\n",
" neg_mean_loss: -7.9491170267942515\n",
" node_ip: 127.0.0.1\n",
" pid: 46341\n",
" time_since_restore: 12.64172887802124\n",
" time_this_iter_s: 0.10761594772338867\n",
" time_total_s: 12.64172887802124\n",
" timestamp: 1658499837\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: e3482178\n",
" warmup_time: 0.002877950668334961\n",
" \n",
"Result for objective_e532d6e0:\n",
" date: 2022-07-22_15-23-58\n",
" done: false\n",
" experiment_id: 34d05fd425474b82a450c22c2a063843\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 95\n",
" iterations_since_restore: 96\n",
" mean_loss: -6.554810275125985\n",
" neg_mean_loss: 6.554810275125985\n",
" node_ip: 127.0.0.1\n",
" pid: 46353\n",
" time_since_restore: 10.305040121078491\n",
" time_this_iter_s: 0.10702300071716309\n",
" time_total_s: 10.305040121078491\n",
" timestamp: 1658499838\n",
" timesteps_since_restore: 0\n",
" training_iteration: 96\n",
" trial_id: e532d6e0\n",
" warmup_time: 0.0028901100158691406\n",
" \n",
"Result for objective_e532d6e0:\n",
" date: 2022-07-22_15-23-58\n",
" done: true\n",
" experiment_id: 34d05fd425474b82a450c22c2a063843\n",
" experiment_tag: 10_activation=tanh,height=-66.9988,steps=100,width=7.1509\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" mean_loss: -6.560589957038967\n",
" neg_mean_loss: 6.560589957038967\n",
" node_ip: 127.0.0.1\n",
" pid: 46353\n",
" time_since_restore: 10.74857211112976\n",
" time_this_iter_s: 0.1075899600982666\n",
" time_total_s: 10.74857211112976\n",
" timestamp: 1658499838\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: e532d6e0\n",
" warmup_time: 0.0028901100158691406\n",
" \n"
]
}
],
"source": [
"tuner = tune.Tuner(\n",
" objective,\n",
" tune_config=tune.TuneConfig(\n",
" search_alg=algo,\n",
" num_samples=num_samples,\n",
" ),\n",
")\n",
"results = tuner.fit()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "11e1ee04",
"metadata": {},
"source": [
"We take a look again at the optimal hyperparameters."
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "13e4ce18",
"metadata": {
"lines_to_next_cell": 0
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best hyperparameters for loss found were: {'activation': 'tanh', 'width': 17.73584230792165, 'height': -76.43866128895999, 'steps': 100}\n"
]
}
],
"source": [
"print(\"Best hyperparameters for loss found were: \", results.get_best_result(\"mean_loss\", \"min\").config)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "34bbd066",
"metadata": {},
"source": [
"## Multi-objective optimization\n",
"\n",
"Finally, let's take a look at the multi-objective case."
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "b233cbea",
"metadata": {},
"outputs": [],
"source": [
"def multi_objective(config):\n",
" # Hyperparameters\n",
" width, height = config[\"width\"], config[\"height\"]\n",
"\n",
" for step in range(config[\"steps\"]):\n",
" # Iterative training function - can be any arbitrary training procedure\n",
" intermediate_score = evaluate(step, config[\"width\"], config[\"height\"], config[\"activation\"])\n",
" # Feed the score back back to Tune.\n",
" train.report({\n",
" \"iterations\": step, \"loss\": intermediate_score, \"gain\": intermediate_score * width\n",
" })"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "338e5108",
"metadata": {},
"source": [
"We define the `OptunaSearch` object this time with metric and mode as list arguments."
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "624d0bc8",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[32m[I 2022-07-22 15:26:50,680]\u001b[0m A new study created in memory with name: optuna\u001b[0m\n"
]
},
{
"data": {
"text/html": [
"== Status ==
Current time: 2022-07-22 15:27:34 (running for 00:00:43.55)
Memory usage on this node: 8.2/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects
Result logdir: /Users/kai/ray_results/multi_objective_2022-07-22_15-26-50
Number of trials: 10/10 (10 TERMINATED)
\n",
"\n",
"Trial name | status | loc | activation | height | width | iter | total time (s) | iterations | loss | gain |
\n",
"\n",
"\n",
"multi_objective_5378f1e8 | TERMINATED | 127.0.0.1:46621 | tanh | -47.5236 | 10.9256 | 100 | 10.7064 | 99 | -4.66076 | -50.9216 |
\n",
"multi_objective_550962f4 | TERMINATED | 127.0.0.1:46631 | relu | -40.6828 | 4.34953 | 100 | 11.3948 | 99 | 6.15869 | 26.7874 |
\n",
"multi_objective_550ad9a4 | TERMINATED | 127.0.0.1:46632 | tanh | -42.0563 | 2.02124 | 100 | 11.2755 | 99 | -3.72967 | -7.53857 |
\n",
"multi_objective_550c4faa | TERMINATED | 127.0.0.1:46633 | relu | -56.2838 | 5.65237 | 100 | 11.407 | 99 | 4.54719 | 25.7024 |
\n",
"multi_objective_5b6c4ba2 | TERMINATED | 127.0.0.1:46679 | relu | 54.9133 | 16.9039 | 100 | 10.6698 | 99 | 15.5507 | 262.867 |
\n",
"multi_objective_5d51f2d2 | TERMINATED | 127.0.0.1:46686 | tanh | 16.5793 | 5.16709 | 100 | 10.7451 | 99 | 1.84967 | 9.55744 |
\n",
"multi_objective_5d636760 | TERMINATED | 127.0.0.1:46689 | tanh | -13.8895 | 13.6043 | 100 | 10.6897 | 99 | -1.31525 | -17.8931 |
\n",
"multi_objective_5d66093e | TERMINATED | 127.0.0.1:46690 | relu | -51.6714 | 19.8211 | 100 | 10.7246 | 99 | 4.88357 | 96.7974 |
\n",
"multi_objective_634c9444 | TERMINATED | 127.0.0.1:46710 | relu | -12.3384 | 15.1097 | 100 | 13.1986 | 99 | 8.83256 | 133.457 |
\n",
"multi_objective_65469a2e | TERMINATED | 127.0.0.1:46719 | tanh | 19.6774 | 13.6172 | 100 | 10.7279 | 99 | 2.04137 | 27.7978 |
\n",
"\n",
"
"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for multi_objective_5378f1e8:\n",
" date: 2022-07-22_15-26-53\n",
" done: false\n",
" experiment_id: 4ce03cab420a449a9f4827c3b6b76c38\n",
" gain: 57.33359096413655\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" loss: 5.247638036988612\n",
" node_ip: 127.0.0.1\n",
" pid: 46621\n",
" time_since_restore: 0.10418081283569336\n",
" time_this_iter_s: 0.10418081283569336\n",
" time_total_s: 0.10418081283569336\n",
" timestamp: 1658500013\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 5378f1e8\n",
" warmup_time: 0.002785921096801758\n",
" \n",
"Result for multi_objective_550ad9a4:\n",
" date: 2022-07-22_15-26-56\n",
" done: false\n",
" experiment_id: 102a4b06449b428b9870a75d02812be6\n",
" gain: 11.711822825618256\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" loss: 5.794371166191576\n",
" node_ip: 127.0.0.1\n",
" pid: 46632\n",
" time_since_restore: 0.10288000106811523\n",
" time_this_iter_s: 0.10288000106811523\n",
" time_total_s: 0.10288000106811523\n",
" timestamp: 1658500016\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 550ad9a4\n",
" warmup_time: 0.0030989646911621094\n",
" \n",
"Result for multi_objective_550c4faa:\n",
" date: 2022-07-22_15-26-56\n",
" done: false\n",
" experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d\n",
" gain: 81.23378320141829\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" loss: 14.371624912716872\n",
" node_ip: 127.0.0.1\n",
" pid: 46633\n",
" time_since_restore: 0.10215306282043457\n",
" time_this_iter_s: 0.10215306282043457\n",
" time_total_s: 0.10215306282043457\n",
" timestamp: 1658500016\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 550c4faa\n",
" warmup_time: 0.0036542415618896484\n",
" \n",
"Result for multi_objective_550962f4:\n",
" date: 2022-07-22_15-26-56\n",
" done: false\n",
" experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64\n",
" gain: 69.29543163212045\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" loss: 15.93172358423837\n",
" node_ip: 127.0.0.1\n",
" pid: 46631\n",
" time_since_restore: 0.10244321823120117\n",
" time_this_iter_s: 0.10244321823120117\n",
" time_total_s: 0.10244321823120117\n",
" timestamp: 1658500016\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 550962f4\n",
" warmup_time: 0.003384828567504883\n",
" \n",
"Result for multi_objective_5378f1e8:\n",
" date: 2022-07-22_15-26-58\n",
" done: false\n",
" experiment_id: 4ce03cab420a449a9f4827c3b6b76c38\n",
" gain: -49.83538614085625\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" loss: -4.561341152769549\n",
" node_ip: 127.0.0.1\n",
" pid: 46621\n",
" time_since_restore: 5.1368408203125\n",
" time_this_iter_s: 0.1180417537689209\n",
" time_total_s: 5.1368408203125\n",
" timestamp: 1658500018\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 5378f1e8\n",
" warmup_time: 0.002785921096801758\n",
" \n",
"Result for multi_objective_550ad9a4:\n",
" date: 2022-07-22_15-27-01\n",
" done: false\n",
" experiment_id: 102a4b06449b428b9870a75d02812be6\n",
" gain: -6.575568930702184\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" loss: -3.2532328725144097\n",
" node_ip: 127.0.0.1\n",
" pid: 46632\n",
" time_since_restore: 5.146106719970703\n",
" time_this_iter_s: 0.10509181022644043\n",
" time_total_s: 5.146106719970703\n",
" timestamp: 1658500021\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 550ad9a4\n",
" warmup_time: 0.0030989646911621094\n",
" \n",
"Result for multi_objective_550c4faa:\n",
" date: 2022-07-22_15-27-01\n",
" done: false\n",
" experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d\n",
" gain: 26.760529995257745\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" loss: 4.73438862995905\n",
" node_ip: 127.0.0.1\n",
" pid: 46633\n",
" time_since_restore: 5.1661481857299805\n",
" time_this_iter_s: 0.10656309127807617\n",
" time_total_s: 5.1661481857299805\n",
" timestamp: 1658500021\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 550c4faa\n",
" warmup_time: 0.0036542415618896484\n",
" \n",
"Result for multi_objective_550962f4:\n",
" date: 2022-07-22_15-27-01\n",
" done: false\n",
" experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64\n",
" gain: 27.828615153405433\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" loss: 6.398081286367129\n",
" node_ip: 127.0.0.1\n",
" pid: 46631\n",
" time_since_restore: 5.169018983840942\n",
" time_this_iter_s: 0.10616683959960938\n",
" time_total_s: 5.169018983840942\n",
" timestamp: 1658500021\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 550962f4\n",
" warmup_time: 0.003384828567504883\n",
" \n",
"Result for multi_objective_5378f1e8:\n",
" date: 2022-07-22_15-27-03\n",
" done: false\n",
" experiment_id: 4ce03cab420a449a9f4827c3b6b76c38\n",
" gain: -50.86883186480195\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" loss: -4.65593053743818\n",
" node_ip: 127.0.0.1\n",
" pid: 46621\n",
" time_since_restore: 10.17048692703247\n",
" time_this_iter_s: 0.10630917549133301\n",
" time_total_s: 10.17048692703247\n",
" timestamp: 1658500023\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: 5378f1e8\n",
" warmup_time: 0.002785921096801758\n",
" \n",
"Result for multi_objective_5378f1e8:\n",
" date: 2022-07-22_15-27-04\n",
" done: true\n",
" experiment_id: 4ce03cab420a449a9f4827c3b6b76c38\n",
" experiment_tag: 1_activation=tanh,height=-47.5236,steps=100,width=10.9256\n",
" gain: -50.92155508732405\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" loss: -4.66075619693878\n",
" node_ip: 127.0.0.1\n",
" pid: 46621\n",
" time_since_restore: 10.706397771835327\n",
" time_this_iter_s: 0.10694766044616699\n",
" time_total_s: 10.706397771835327\n",
" timestamp: 1658500024\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 5378f1e8\n",
" warmup_time: 0.002785921096801758\n",
" \n",
"Result for multi_objective_550ad9a4:\n",
" date: 2022-07-22_15-27-05\n",
" done: false\n",
" experiment_id: 102a4b06449b428b9870a75d02812be6\n",
" gain: -7.479635489825483\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 93\n",
" iterations_since_restore: 94\n",
" loss: -3.700515697784228\n",
" node_ip: 127.0.0.1\n",
" pid: 46632\n",
" time_since_restore: 10.056212902069092\n",
" time_this_iter_s: 0.10589313507080078\n",
" time_total_s: 10.056212902069092\n",
" timestamp: 1658500025\n",
" timesteps_since_restore: 0\n",
" training_iteration: 94\n",
" trial_id: 550ad9a4\n",
" warmup_time: 0.0030989646911621094\n",
" \n",
"Result for multi_objective_550c4faa:\n",
" date: 2022-07-22_15-27-05\n",
" done: false\n",
" experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d\n",
" gain: 25.776502806907246\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 92\n",
" iterations_since_restore: 93\n",
" loss: 4.5602976409942295\n",
" node_ip: 127.0.0.1\n",
" pid: 46633\n",
" time_since_restore: 9.99274206161499\n",
" time_this_iter_s: 0.10583209991455078\n",
" time_total_s: 9.99274206161499\n",
" timestamp: 1658500025\n",
" timesteps_since_restore: 0\n",
" training_iteration: 93\n",
" trial_id: 550c4faa\n",
" warmup_time: 0.0036542415618896484\n",
" \n",
"Result for multi_objective_550962f4:\n",
" date: 2022-07-22_15-27-05\n",
" done: false\n",
" experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64\n",
" gain: 26.860636112316993\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 92\n",
" iterations_since_restore: 93\n",
" loss: 6.175533072801938\n",
" node_ip: 127.0.0.1\n",
" pid: 46631\n",
" time_since_restore: 9.971143007278442\n",
" time_this_iter_s: 0.1054527759552002\n",
" time_total_s: 9.971143007278442\n",
" timestamp: 1658500025\n",
" timesteps_since_restore: 0\n",
" training_iteration: 93\n",
" trial_id: 550962f4\n",
" warmup_time: 0.003384828567504883\n",
" \n",
"Result for multi_objective_5b6c4ba2:\n",
" date: 2022-07-22_15-27-06\n",
" done: false\n",
" experiment_id: f143c125a0734166ac832270ac08dbb6\n",
" gain: 430.90195437776714\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" loss: 25.491329340230966\n",
" node_ip: 127.0.0.1\n",
" pid: 46679\n",
" time_since_restore: 0.10215902328491211\n",
" time_this_iter_s: 0.10215902328491211\n",
" time_total_s: 0.10215902328491211\n",
" timestamp: 1658500026\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 5b6c4ba2\n",
" warmup_time: 0.002663135528564453\n",
" \n",
"Result for multi_objective_550ad9a4:\n",
" date: 2022-07-22_15-27-07\n",
" done: true\n",
" experiment_id: 102a4b06449b428b9870a75d02812be6\n",
" experiment_tag: 3_activation=tanh,height=-42.0563,steps=100,width=2.0212\n",
" gain: -7.538566547680232\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" loss: -3.729671571085163\n",
" node_ip: 127.0.0.1\n",
" pid: 46632\n",
" time_since_restore: 11.27550482749939\n",
" time_this_iter_s: 0.10642290115356445\n",
" time_total_s: 11.27550482749939\n",
" timestamp: 1658500027\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 550ad9a4\n",
" warmup_time: 0.0030989646911621094\n",
" \n",
"Result for multi_objective_550962f4:\n",
" date: 2022-07-22_15-27-07\n",
" done: true\n",
" experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64\n",
" experiment_tag: 2_activation=relu,height=-40.6828,steps=100,width=4.3495\n",
" gain: 26.787356208375382\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" loss: 6.15868527856241\n",
" node_ip: 127.0.0.1\n",
" pid: 46631\n",
" time_since_restore: 11.394789934158325\n",
" time_this_iter_s: 0.10491585731506348\n",
" time_total_s: 11.394789934158325\n",
" timestamp: 1658500027\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 550962f4\n",
" warmup_time: 0.003384828567504883\n",
" \n",
"Result for multi_objective_550c4faa:\n",
" date: 2022-07-22_15-27-07\n",
" done: true\n",
" experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d\n",
" experiment_tag: 4_activation=relu,height=-56.2838,steps=100,width=5.6524\n",
" gain: 25.702421216002982\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" loss: 4.547191359402326\n",
" node_ip: 127.0.0.1\n",
" pid: 46633\n",
" time_since_restore: 11.406961917877197\n",
" time_this_iter_s: 0.12177181243896484\n",
" time_total_s: 11.406961917877197\n",
" timestamp: 1658500027\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 550c4faa\n",
" warmup_time: 0.0036542415618896484\n",
" \n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for multi_objective_5d51f2d2:\n",
" date: 2022-07-22_15-27-09\n",
" done: false\n",
" experiment_id: 1433194530b14db4b77ce40a69d65407\n",
" gain: 60.23765295445784\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" loss: 11.657934184433229\n",
" node_ip: 127.0.0.1\n",
" pid: 46686\n",
" time_since_restore: 0.10274410247802734\n",
" time_this_iter_s: 0.10274410247802734\n",
" time_total_s: 0.10274410247802734\n",
" timestamp: 1658500029\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 5d51f2d2\n",
" warmup_time: 0.0028769969940185547\n",
" \n",
"Result for multi_objective_5d636760:\n",
" date: 2022-07-22_15-27-10\n",
" done: false\n",
" experiment_id: b8dccac5d8914569b1c2ca061ad47a56\n",
" gain: 117.14725547845826\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" loss: 8.611046741256526\n",
" node_ip: 127.0.0.1\n",
" pid: 46689\n",
" time_since_restore: 0.10460686683654785\n",
" time_this_iter_s: 0.10460686683654785\n",
" time_total_s: 0.10460686683654785\n",
" timestamp: 1658500030\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 5d636760\n",
" warmup_time: 0.0028090476989746094\n",
" \n",
"Result for multi_objective_5d66093e:\n",
" date: 2022-07-22_15-27-10\n",
" done: false\n",
" experiment_id: e7259e986278458899b46b485e069ec9\n",
" gain: 294.00300634729234\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" loss: 14.832864636325777\n",
" node_ip: 127.0.0.1\n",
" pid: 46690\n",
" time_since_restore: 0.10294318199157715\n",
" time_this_iter_s: 0.10294318199157715\n",
" time_total_s: 0.10294318199157715\n",
" timestamp: 1658500030\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 5d66093e\n",
" warmup_time: 0.0027511119842529297\n",
" \n",
"Result for multi_objective_5b6c4ba2:\n",
" date: 2022-07-22_15-27-11\n",
" done: false\n",
" experiment_id: f143c125a0734166ac832270ac08dbb6\n",
" gain: 263.92128797150474\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 48\n",
" iterations_since_restore: 49\n",
" loss: 15.6130748612977\n",
" node_ip: 127.0.0.1\n",
" pid: 46679\n",
" time_since_restore: 5.2078468799591064\n",
" time_this_iter_s: 0.10803008079528809\n",
" time_total_s: 5.2078468799591064\n",
" timestamp: 1658500031\n",
" timesteps_since_restore: 0\n",
" training_iteration: 49\n",
" trial_id: 5b6c4ba2\n",
" warmup_time: 0.002663135528564453\n",
" \n",
"Result for multi_objective_5d51f2d2:\n",
" date: 2022-07-22_15-27-14\n",
" done: false\n",
" experiment_id: 1433194530b14db4b77ce40a69d65407\n",
" gain: 10.610216987384797\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" loss: 2.053420165872076\n",
" node_ip: 127.0.0.1\n",
" pid: 46686\n",
" time_since_restore: 5.178762912750244\n",
" time_this_iter_s: 0.10710978507995605\n",
" time_total_s: 5.178762912750244\n",
" timestamp: 1658500034\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 5d51f2d2\n",
" warmup_time: 0.0028769969940185547\n",
" \n",
"Result for multi_objective_5d636760:\n",
" date: 2022-07-22_15-27-15\n",
" done: false\n",
" experiment_id: b8dccac5d8914569b1c2ca061ad47a56\n",
" gain: -16.800839364465826\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" loss: -1.234965449842409\n",
" node_ip: 127.0.0.1\n",
" pid: 46689\n",
" time_since_restore: 5.146665811538696\n",
" time_this_iter_s: 0.10526585578918457\n",
" time_total_s: 5.146665811538696\n",
" timestamp: 1658500035\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 5d636760\n",
" warmup_time: 0.0028090476989746094\n",
" \n",
"Result for multi_objective_5d66093e:\n",
" date: 2022-07-22_15-27-15\n",
" done: false\n",
" experiment_id: e7259e986278458899b46b485e069ec9\n",
" gain: 97.89753254610552\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" loss: 4.939068026982622\n",
" node_ip: 127.0.0.1\n",
" pid: 46690\n",
" time_since_restore: 5.1532862186431885\n",
" time_this_iter_s: 0.10752320289611816\n",
" time_total_s: 5.1532862186431885\n",
" timestamp: 1658500035\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 5d66093e\n",
" warmup_time: 0.0027511119842529297\n",
" \n",
"Result for multi_objective_5b6c4ba2:\n",
" date: 2022-07-22_15-27-16\n",
" done: false\n",
" experiment_id: f143c125a0734166ac832270ac08dbb6\n",
" gain: 262.90943552209455\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 95\n",
" iterations_since_restore: 96\n",
" loss: 15.553215620072208\n",
" node_ip: 127.0.0.1\n",
" pid: 46679\n",
" time_since_restore: 10.240519762039185\n",
" time_this_iter_s: 0.10443782806396484\n",
" time_total_s: 10.240519762039185\n",
" timestamp: 1658500036\n",
" timesteps_since_restore: 0\n",
" training_iteration: 96\n",
" trial_id: 5b6c4ba2\n",
" warmup_time: 0.002663135528564453\n",
" \n",
"Result for multi_objective_5b6c4ba2:\n",
" date: 2022-07-22_15-27-17\n",
" done: true\n",
" experiment_id: f143c125a0734166ac832270ac08dbb6\n",
" experiment_tag: 5_activation=relu,height=54.9133,steps=100,width=16.9039\n",
" gain: 262.86741923011425\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" loss: 15.550730017197484\n",
" node_ip: 127.0.0.1\n",
" pid: 46679\n",
" time_since_restore: 10.669770956039429\n",
" time_this_iter_s: 0.10598993301391602\n",
" time_total_s: 10.669770956039429\n",
" timestamp: 1658500037\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 5b6c4ba2\n",
" warmup_time: 0.002663135528564453\n",
" \n",
"Result for multi_objective_634c9444:\n",
" date: 2022-07-22_15-27-19\n",
" done: false\n",
" experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee\n",
" gain: 283.55076351521427\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" loss: 18.766157678691147\n",
" node_ip: 127.0.0.1\n",
" pid: 46710\n",
" time_since_restore: 0.10465264320373535\n",
" time_this_iter_s: 0.10465264320373535\n",
" time_total_s: 0.10465264320373535\n",
" timestamp: 1658500039\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 634c9444\n",
" warmup_time: 0.004335880279541016\n",
" \n",
"Result for multi_objective_5d51f2d2:\n",
" date: 2022-07-22_15-27-20\n",
" done: false\n",
" experiment_id: 1433194530b14db4b77ce40a69d65407\n",
" gain: 9.60907229177291\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" loss: 1.8596662860626851\n",
" node_ip: 127.0.0.1\n",
" pid: 46686\n",
" time_since_restore: 10.218497037887573\n",
" time_this_iter_s: 0.10553288459777832\n",
" time_total_s: 10.218497037887573\n",
" timestamp: 1658500040\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: 5d51f2d2\n",
" warmup_time: 0.0028769969940185547\n",
" \n",
"Result for multi_objective_5d636760:\n",
" date: 2022-07-22_15-27-20\n",
" done: false\n",
" experiment_id: b8dccac5d8914569b1c2ca061ad47a56\n",
" gain: -17.840160182810624\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" loss: -1.311361948500235\n",
" node_ip: 127.0.0.1\n",
" pid: 46689\n",
" time_since_restore: 10.16199278831482\n",
" time_this_iter_s: 0.10700488090515137\n",
" time_total_s: 10.16199278831482\n",
" timestamp: 1658500040\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: 5d636760\n",
" warmup_time: 0.0028090476989746094\n",
" \n",
"Result for multi_objective_5d66093e:\n",
" date: 2022-07-22_15-27-20\n",
" done: false\n",
" experiment_id: e7259e986278458899b46b485e069ec9\n",
" gain: 96.85061995053432\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" loss: 4.886249816008858\n",
" node_ip: 127.0.0.1\n",
" pid: 46690\n",
" time_since_restore: 10.197196006774902\n",
" time_this_iter_s: 0.10463690757751465\n",
" time_total_s: 10.197196006774902\n",
" timestamp: 1658500040\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: 5d66093e\n",
" warmup_time: 0.0027511119842529297\n",
" \n",
"Result for multi_objective_5d51f2d2:\n",
" date: 2022-07-22_15-27-20\n",
" done: true\n",
" experiment_id: 1433194530b14db4b77ce40a69d65407\n",
" experiment_tag: 6_activation=tanh,height=16.5793,steps=100,width=5.1671\n",
" gain: 9.557436804323405\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" loss: 1.8496731491334866\n",
" node_ip: 127.0.0.1\n",
" pid: 46686\n",
" time_since_restore: 10.745142936706543\n",
" time_this_iter_s: 0.10509181022644043\n",
" time_total_s: 10.745142936706543\n",
" timestamp: 1658500040\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 5d51f2d2\n",
" warmup_time: 0.0028769969940185547\n",
" \n",
"Result for multi_objective_5d636760:\n",
" date: 2022-07-22_15-27-20\n",
" done: true\n",
" experiment_id: b8dccac5d8914569b1c2ca061ad47a56\n",
" experiment_tag: 7_activation=tanh,height=-13.8895,steps=100,width=13.6043\n",
" gain: -17.893079155032332\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" loss: -1.3152518197690004\n",
" node_ip: 127.0.0.1\n",
" pid: 46689\n",
" time_since_restore: 10.689707040786743\n",
" time_this_iter_s: 0.10322403907775879\n",
" time_total_s: 10.689707040786743\n",
" timestamp: 1658500040\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 5d636760\n",
" warmup_time: 0.0028090476989746094\n",
" \n",
"Result for multi_objective_5d66093e:\n",
" date: 2022-07-22_15-27-20\n",
" done: true\n",
" experiment_id: e7259e986278458899b46b485e069ec9\n",
" experiment_tag: 8_activation=relu,height=-51.6714,steps=100,width=19.8211\n",
" gain: 96.79744897015512\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" loss: 4.8835672653630535\n",
" node_ip: 127.0.0.1\n",
" pid: 46690\n",
" time_since_restore: 10.72464108467102\n",
" time_this_iter_s: 0.10328292846679688\n",
" time_total_s: 10.72464108467102\n",
" timestamp: 1658500040\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 5d66093e\n",
" warmup_time: 0.0027511119842529297\n",
" \n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for multi_objective_65469a2e:\n",
" date: 2022-07-22_15-27-23\n",
" done: false\n",
" experiment_id: 015ee29ddb8b4c239d85a022a23d4093\n",
" gain: 162.96737941207124\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 0\n",
" iterations_since_restore: 1\n",
" loss: 11.96774070054979\n",
" node_ip: 127.0.0.1\n",
" pid: 46719\n",
" time_since_restore: 0.10403180122375488\n",
" time_this_iter_s: 0.10403180122375488\n",
" time_total_s: 0.10403180122375488\n",
" timestamp: 1658500043\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 65469a2e\n",
" warmup_time: 0.0033850669860839844\n",
" \n",
"Result for multi_objective_634c9444:\n",
" date: 2022-07-22_15-27-24\n",
" done: false\n",
" experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee\n",
" gain: 136.50874589912112\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 24\n",
" iterations_since_restore: 25\n",
" loss: 9.034518610724309\n",
" node_ip: 127.0.0.1\n",
" pid: 46710\n",
" time_since_restore: 5.125189781188965\n",
" time_this_iter_s: 0.11187601089477539\n",
" time_total_s: 5.125189781188965\n",
" timestamp: 1658500044\n",
" timesteps_since_restore: 0\n",
" training_iteration: 25\n",
" trial_id: 634c9444\n",
" warmup_time: 0.004335880279541016\n",
" \n",
"Result for multi_objective_65469a2e:\n",
" date: 2022-07-22_15-27-28\n",
" done: false\n",
" experiment_id: 015ee29ddb8b4c239d85a022a23d4093\n",
" gain: 28.890088364068774\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 47\n",
" iterations_since_restore: 48\n",
" loss: 2.1215846238952016\n",
" node_ip: 127.0.0.1\n",
" pid: 46719\n",
" time_since_restore: 5.174077749252319\n",
" time_this_iter_s: 0.10734677314758301\n",
" time_total_s: 5.174077749252319\n",
" timestamp: 1658500048\n",
" timesteps_since_restore: 0\n",
" training_iteration: 48\n",
" trial_id: 65469a2e\n",
" warmup_time: 0.0033850669860839844\n",
" \n",
"Result for multi_objective_634c9444:\n",
" date: 2022-07-22_15-27-30\n",
" done: false\n",
" experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee\n",
" gain: 133.84933935751317\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 71\n",
" iterations_since_restore: 72\n",
" loss: 8.85851188137237\n",
" node_ip: 127.0.0.1\n",
" pid: 46710\n",
" time_since_restore: 10.204095840454102\n",
" time_this_iter_s: 0.10942888259887695\n",
" time_total_s: 10.204095840454102\n",
" timestamp: 1658500050\n",
" timesteps_since_restore: 0\n",
" training_iteration: 72\n",
" trial_id: 634c9444\n",
" warmup_time: 0.004335880279541016\n",
" \n",
"Result for multi_objective_634c9444:\n",
" date: 2022-07-22_15-27-33\n",
" done: true\n",
" experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee\n",
" experiment_tag: 9_activation=relu,height=-12.3384,steps=100,width=15.1097\n",
" gain: 133.45728949146132\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" loss: 8.83256495915982\n",
" node_ip: 127.0.0.1\n",
" pid: 46710\n",
" time_since_restore: 13.198630809783936\n",
" time_this_iter_s: 0.1067969799041748\n",
" time_total_s: 13.198630809783936\n",
" timestamp: 1658500053\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 634c9444\n",
" warmup_time: 0.004335880279541016\n",
" \n",
"Result for multi_objective_65469a2e:\n",
" date: 2022-07-22_15-27-33\n",
" done: false\n",
" experiment_id: 015ee29ddb8b4c239d85a022a23d4093\n",
" gain: 27.850744704436202\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 94\n",
" iterations_since_restore: 95\n",
" loss: 2.0452589477867855\n",
" node_ip: 127.0.0.1\n",
" pid: 46719\n",
" time_since_restore: 10.19201397895813\n",
" time_this_iter_s: 0.10735702514648438\n",
" time_total_s: 10.19201397895813\n",
" timestamp: 1658500053\n",
" timesteps_since_restore: 0\n",
" training_iteration: 95\n",
" trial_id: 65469a2e\n",
" warmup_time: 0.0033850669860839844\n",
" \n",
"Result for multi_objective_65469a2e:\n",
" date: 2022-07-22_15-27-34\n",
" done: true\n",
" experiment_id: 015ee29ddb8b4c239d85a022a23d4093\n",
" experiment_tag: 10_activation=tanh,height=19.6774,steps=100,width=13.6172\n",
" gain: 27.797824972416166\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations: 99\n",
" iterations_since_restore: 100\n",
" loss: 2.041372712191397\n",
" node_ip: 127.0.0.1\n",
" pid: 46719\n",
" time_since_restore: 10.727944135665894\n",
" time_this_iter_s: 0.10885024070739746\n",
" time_total_s: 10.727944135665894\n",
" timestamp: 1658500054\n",
" timesteps_since_restore: 0\n",
" training_iteration: 100\n",
" trial_id: 65469a2e\n",
" warmup_time: 0.0033850669860839844\n",
" \n"
]
}
],
"source": [
"searcher = OptunaSearch(metric=[\"loss\", \"gain\"], mode=[\"min\", \"max\"])\n",
"algo = ConcurrencyLimiter(searcher, max_concurrent=4)\n",
"\n",
"tuner = tune.Tuner(\n",
" multi_objective,\n",
" tune_config=tune.TuneConfig(\n",
" search_alg=algo,\n",
" num_samples=num_samples,\n",
" ),\n",
" param_space=search_space\n",
")\n",
"results = tuner.fit()"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "df42b8b3",
"metadata": {},
"source": [
"Now there are two hyperparameter sets for the two objectives."
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "183fef1a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best hyperparameters for loss found were: {'steps': 100, 'width': 10.925599395387751, 'height': -47.52361963011387, 'activation': 'tanh'}\n",
"Best hyperparameters for gain found were: {'steps': 100, 'width': 16.90386360893735, 'height': 54.91329340230965, 'activation': 'relu'}\n"
]
}
],
"source": [
"print(\"Best hyperparameters for loss found were: \", results.get_best_result(\"loss\", \"min\").config)\n",
"print(\"Best hyperparameters for gain found were: \", results.get_best_result(\"gain\", \"max\").config)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "cdf4d49a",
"metadata": {},
"source": [
"We can mix-and-match the use of initial hyperparameter evaluations, conditional search spaces via define-by-run functions, and multi-objective tasks. This is also true of scheduler usage, with the exception of multi-objective optimization-- schedulers typically rely on a single scalar score, rather than the two scores we use here: loss, gain."
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "a058fdb3",
"metadata": {
"tags": [
"remove-cell"
]
},
"outputs": [],
"source": [
"ray.shutdown()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
},
"orphan": true
},
"nbformat": 4,
"nbformat_minor": 5
}