Running Tune experiments with Skopt
Contents
Running Tune experiments with Skopt#
In this tutorial we introduce Skopt, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with Skopt and, as a result, allow you to seamlessly scale up a Skopt optimization process - without sacrificing performance.
Scikit-Optimize, or skopt, is a simple and efficient library to optimize expensive and noisy black-box functions, e.g. large-scale ML experiments. It implements several methods for sequential model-based optimization. Noteably, skopt does not perform gradient-based optimization, and instead uses computationally cheap surrogate models to
approximate the expensive function. In this example we minimize a simple objective to briefly demonstrate the usage of Skopt with Ray Tune via SkOptSearch. It’s useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume scikit-opitmize==0.8.1 library is installed. To learn more, please refer to the Scikit-Optimize website.
Click below to see all the imports we need for this example. You can also launch directly into a Binder instance to run this notebook yourself. Just click on the rocket symbol at the top of the navigation.
import time
from typing import Dict, Optional, Any
import ray
import skopt
from ray import tune
from ray.air import session
from ray.tune.search import ConcurrencyLimiter
from ray.tune.search.skopt import SkOptSearch
Let’s start by defining a simple evaluation function. Again, an explicit math formula is queried here for demonstration, yet in practice this is typically a black-box function– e.g. the performance results after training an ML model. We artificially sleep for a bit (0.1 seconds) to simulate a long-running ML experiment. This setup assumes that we’re running multiple steps of an experiment while tuning three hyperparameters, namely width, height, and activation.
def evaluate(step, width, height, activation):
time.sleep(0.1)
activation_boost = 10 if activation=="relu" else 0
return (0.1 + width * step / 100) ** (-1) + height * 0.1 + activation_boost
Next, our objective function to be optimized takes a Tune config, evaluates the score of your experiment in a training loop,
and uses session.report to report the score back to Tune.
def objective(config):
for step in range(config["steps"]):
score = evaluate(step, config["width"], config["height"], config["activation"])
session.report({"iterations": step, "mean_loss": score})
Next we define a search space. The critical assumption is that the optimal hyperparamters live within this space. Yet, if the space is very large, then those hyperparameters may be difficult to find in a short amount of time.
search_space = {
"steps": 100,
"width": tune.uniform(0, 20),
"height": tune.uniform(-100, 100),
"activation": tune.choice(["relu", "tanh"]),
}
The search algorithm is instantiated from the SkOptSearch class. We also constrain the number of concurrent trials to 4 with a ConcurrencyLimiter.
algo = SkOptSearch()
algo = ConcurrencyLimiter(algo, max_concurrent=4)
The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to 1000 samples.
(you can decrease this if it takes too long on your machine).
num_samples = 1000
Finally, we run the experiment to "min"imize the “mean_loss” of the objective by searching search_config via algo, num_samples times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute tuner.fit().
tuner = tune.Tuner(
objective,
tune_config=tune.TuneConfig(
metric="mean_loss",
mode="min",
search_alg=algo,
num_samples=num_samples,
),
param_space=search_space,
)
results = tuner.fit()
Function checkpointing is disabled. This may result in unexpected behavior when using checkpointing features or certain schedulers. To enable, set the train function arguments to be `func(config, checkpoint_dir=None)`.
== Status ==
Current time: 2022-07-22 15:47:07 (running for 00:00:44.11)
Memory usage on this node: 9.9/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.42 GiB heap, 0.0/2.0 GiB objects
Current best trial: 20e93604 with mean_loss=-8.869507755259557 and parameters={'steps': 100, 'width': 8.246480084328711, 'height': -89.90514307848873, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-46-22
Number of trials: 10/10 (10 TERMINATED)
Current time: 2022-07-22 15:47:07 (running for 00:00:44.11)
Memory usage on this node: 9.9/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.42 GiB heap, 0.0/2.0 GiB objects
Current best trial: 20e93604 with mean_loss=-8.869507755259557 and parameters={'steps': 100, 'width': 8.246480084328711, 'height': -89.90514307848873, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-46-22
Number of trials: 10/10 (10 TERMINATED)
| Trial name | status | loc | activation | height | width | loss | iter | total time (s) | iterations | neg_mean_loss |
|---|---|---|---|---|---|---|---|---|---|---|
| objective_0ec1ec00 | TERMINATED | 127.0.0.1:48435 | tanh | 10.9976 | 8.60092 | 1.21584 | 100 | 10.941 | 99 | -1.21584 |
| objective_107c17fa | TERMINATED | 127.0.0.1:48446 | relu | 49.5455 | 11.8085 | 15.0394 | 100 | 11.5877 | 99 | -15.0394 |
| objective_107d75f0 | TERMINATED | 127.0.0.1:48447 | tanh | -7.6131 | 3.10991 | -0.446727 | 100 | 11.7109 | 99 | 0.446727 |
| objective_107ef722 | TERMINATED | 127.0.0.1:48448 | relu | -75.8366 | 5.30975 | 2.60302 | 100 | 11.6501 | 99 | -2.60302 |
| objective_17033b12 | TERMINATED | 127.0.0.1:48470 | tanh | -89.1891 | 12.1463 | -8.83643 | 100 | 10.7255 | 99 | 8.83643 |
| objective_18fbb20a | TERMINATED | 127.0.0.1:48475 | tanh | 33.5568 | 9.75437 | 3.45818 | 100 | 10.7563 | 99 | -3.45818 |
| objective_190ab4d0 | TERMINATED | 127.0.0.1:48478 | relu | -65.148 | 4.19546 | 3.7203 | 100 | 10.7123 | 99 | -3.7203 |
| objective_191ca3de | TERMINATED | 127.0.0.1:48481 | tanh | -13.4892 | 6.74078 | -1.20128 | 100 | 13.219 | 99 | 1.20128 |
| objective_1eec012e | TERMINATED | 127.0.0.1:48494 | tanh | -50.2819 | 14.8287 | -4.96053 | 100 | 13.2232 | 99 | 4.96053 |
| objective_20e93604 | TERMINATED | 127.0.0.1:48502 | tanh | -89.9051 | 8.24648 | -8.86951 | 100 | 10.7925 | 99 | 8.86951 |
Result for objective_0ec1ec00:
date: 2022-07-22_15-46-26
done: false
experiment_id: c3e1e5ac1d6c4d67878bf793d020f99e
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 11.099760178357887
neg_mean_loss: -11.099760178357887
node_ip: 127.0.0.1
pid: 48435
time_since_restore: 0.10384106636047363
time_this_iter_s: 0.10384106636047363
time_total_s: 0.10384106636047363
timestamp: 1658501186
timesteps_since_restore: 0
training_iteration: 1
trial_id: 0ec1ec00
warmup_time: 0.003099679946899414
Result for objective_107c17fa:
date: 2022-07-22_15-46-29
done: false
experiment_id: f2a0e8ad4ea64c489424406b75b5c051
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 24.95455186350225
neg_mean_loss: -24.95455186350225
node_ip: 127.0.0.1
pid: 48446
time_since_restore: 0.10309791564941406
time_this_iter_s: 0.10309791564941406
time_total_s: 0.10309791564941406
timestamp: 1658501189
timesteps_since_restore: 0
training_iteration: 1
trial_id: 107c17fa
warmup_time: 0.0029959678649902344
Result for objective_107ef722:
date: 2022-07-22_15-46-29
done: false
experiment_id: 286beb82d0f0493dacbafb0ceb9bfe2d
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 12.416336029599446
neg_mean_loss: -12.416336029599446
node_ip: 127.0.0.1
pid: 48448
time_since_restore: 0.10424971580505371
time_this_iter_s: 0.10424971580505371
time_total_s: 0.10424971580505371
timestamp: 1658501189
timesteps_since_restore: 0
training_iteration: 1
trial_id: 107ef722
warmup_time: 0.0032057762145996094
Result for objective_107d75f0:
date: 2022-07-22_15-46-29
done: false
experiment_id: a44e7e5deb2e4bc68b31ca0466db50f5
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 9.238689666819806
neg_mean_loss: -9.238689666819806
node_ip: 127.0.0.1
pid: 48447
time_since_restore: 0.10475015640258789
time_this_iter_s: 0.10475015640258789
time_total_s: 0.10475015640258789
timestamp: 1658501189
timesteps_since_restore: 0
training_iteration: 1
trial_id: 107d75f0
warmup_time: 0.0029137134552001953
Result for objective_0ec1ec00:
date: 2022-07-22_15-46-31
done: false
experiment_id: c3e1e5ac1d6c4d67878bf793d020f99e
hostname: Kais-MacBook-Pro.local
iterations: 45
iterations_since_restore: 46
mean_loss: 1.3516231012646176
neg_mean_loss: -1.3516231012646176
node_ip: 127.0.0.1
pid: 48435
time_since_restore: 5.127159833908081
time_this_iter_s: 0.10582566261291504
time_total_s: 5.127159833908081
timestamp: 1658501191
timesteps_since_restore: 0
training_iteration: 46
trial_id: 0ec1ec00
warmup_time: 0.003099679946899414
Result for objective_107c17fa:
date: 2022-07-22_15-46-34
done: false
experiment_id: f2a0e8ad4ea64c489424406b75b5c051
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 15.131543197018463
neg_mean_loss: -15.131543197018463
node_ip: 127.0.0.1
pid: 48446
time_since_restore: 5.161302089691162
time_this_iter_s: 0.10862898826599121
time_total_s: 5.161302089691162
timestamp: 1658501194
timesteps_since_restore: 0
training_iteration: 48
trial_id: 107c17fa
warmup_time: 0.0029959678649902344
Result for objective_107ef722:
date: 2022-07-22_15-46-34
done: false
experiment_id: 286beb82d0f0493dacbafb0ceb9bfe2d
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 2.8016060611446587
neg_mean_loss: -2.8016060611446587
node_ip: 127.0.0.1
pid: 48448
time_since_restore: 5.157445907592773
time_this_iter_s: 0.10624003410339355
time_total_s: 5.157445907592773
timestamp: 1658501194
timesteps_since_restore: 0
training_iteration: 48
trial_id: 107ef722
warmup_time: 0.0032057762145996094
Result for objective_107d75f0:
date: 2022-07-22_15-46-34
done: false
experiment_id: a44e7e5deb2e4bc68b31ca0466db50f5
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -0.12096483365591593
neg_mean_loss: 0.12096483365591593
node_ip: 127.0.0.1
pid: 48447
time_since_restore: 5.159098863601685
time_this_iter_s: 0.10865378379821777
time_total_s: 5.159098863601685
timestamp: 1658501194
timesteps_since_restore: 0
training_iteration: 48
trial_id: 107d75f0
warmup_time: 0.0029137134552001953
Result for objective_0ec1ec00:
date: 2022-07-22_15-46-36
done: false
experiment_id: c3e1e5ac1d6c4d67878bf793d020f99e
hostname: Kais-MacBook-Pro.local
iterations: 92
iterations_since_restore: 93
mean_loss: 1.2245597846713034
neg_mean_loss: -1.2245597846713034
node_ip: 127.0.0.1
pid: 48435
time_since_restore: 10.187216997146606
time_this_iter_s: 0.10774493217468262
time_total_s: 10.187216997146606
timestamp: 1658501196
timesteps_since_restore: 0
training_iteration: 93
trial_id: 0ec1ec00
warmup_time: 0.003099679946899414
Result for objective_0ec1ec00:
date: 2022-07-22_15-46-37
done: true
experiment_id: c3e1e5ac1d6c4d67878bf793d020f99e
experiment_tag: 1_activation=tanh,height=10.9976,steps=100,width=8.6009
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 1.2158379996438775
neg_mean_loss: -1.2158379996438775
node_ip: 127.0.0.1
pid: 48435
time_since_restore: 10.9409658908844
time_this_iter_s: 0.10762286186218262
time_total_s: 10.9409658908844
timestamp: 1658501197
timesteps_since_restore: 0
training_iteration: 100
trial_id: 0ec1ec00
warmup_time: 0.003099679946899414
Result for objective_107ef722:
date: 2022-07-22_15-46-39
done: false
experiment_id: 286beb82d0f0493dacbafb0ceb9bfe2d
hostname: Kais-MacBook-Pro.local
iterations: 91
iterations_since_restore: 92
mean_loss: 2.619098804697928
neg_mean_loss: -2.619098804697928
node_ip: 127.0.0.1
pid: 48448
time_since_restore: 9.851147890090942
time_this_iter_s: 0.1064901351928711
time_total_s: 9.851147890090942
timestamp: 1658501199
timesteps_since_restore: 0
training_iteration: 92
trial_id: 107ef722
warmup_time: 0.0032057762145996094
Result for objective_107c17fa:
date: 2022-07-22_15-46-39
done: false
experiment_id: f2a0e8ad4ea64c489424406b75b5c051
hostname: Kais-MacBook-Pro.local
iterations: 92
iterations_since_restore: 93
mean_loss: 15.0457609631331
neg_mean_loss: -15.0457609631331
node_ip: 127.0.0.1
pid: 48446
time_since_restore: 9.976460695266724
time_this_iter_s: 0.10694289207458496
time_total_s: 9.976460695266724
timestamp: 1658501199
timesteps_since_restore: 0
training_iteration: 93
trial_id: 107c17fa
warmup_time: 0.0029959678649902344
Result for objective_107d75f0:
date: 2022-07-22_15-46-39
done: false
experiment_id: a44e7e5deb2e4bc68b31ca0466db50f5
hostname: Kais-MacBook-Pro.local
iterations: 90
iterations_since_restore: 91
mean_loss: -0.41635399078466045
neg_mean_loss: 0.41635399078466045
node_ip: 127.0.0.1
pid: 48447
time_since_restore: 9.776082992553711
time_this_iter_s: 0.10715007781982422
time_total_s: 9.776082992553711
timestamp: 1658501199
timesteps_since_restore: 0
training_iteration: 91
trial_id: 107d75f0
warmup_time: 0.0029137134552001953
Result for objective_17033b12:
date: 2022-07-22_15-46-40
done: false
experiment_id: b5f76922e6424055b6aa255677e2f4b8
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 1.0810921171053653
neg_mean_loss: -1.0810921171053653
node_ip: 127.0.0.1
pid: 48470
time_since_restore: 0.10409784317016602
time_this_iter_s: 0.10409784317016602
time_total_s: 0.10409784317016602
timestamp: 1658501200
timesteps_since_restore: 0
training_iteration: 1
trial_id: 17033b12
warmup_time: 0.0029649734497070312
Result for objective_107c17fa:
date: 2022-07-22_15-46-41
done: true
experiment_id: f2a0e8ad4ea64c489424406b75b5c051
experiment_tag: 2_activation=relu,height=49.5455,steps=100,width=11.8085
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 15.039366532944502
neg_mean_loss: -15.039366532944502
node_ip: 127.0.0.1
pid: 48446
time_since_restore: 11.587661743164062
time_this_iter_s: 0.10614180564880371
time_total_s: 11.587661743164062
timestamp: 1658501201
timesteps_since_restore: 0
training_iteration: 100
trial_id: 107c17fa
warmup_time: 0.0029959678649902344
Result for objective_107ef722:
date: 2022-07-22_15-46-41
done: true
experiment_id: 286beb82d0f0493dacbafb0ceb9bfe2d
experiment_tag: 4_activation=relu,height=-75.8366,steps=100,width=5.3097
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 2.603019812948376
neg_mean_loss: -2.603019812948376
node_ip: 127.0.0.1
pid: 48448
time_since_restore: 11.65014386177063
time_this_iter_s: 0.10755181312561035
time_total_s: 11.65014386177063
timestamp: 1658501201
timesteps_since_restore: 0
training_iteration: 100
trial_id: 107ef722
warmup_time: 0.0032057762145996094
Result for objective_107d75f0:
date: 2022-07-22_15-46-41
done: true
experiment_id: a44e7e5deb2e4bc68b31ca0466db50f5
experiment_tag: 3_activation=tanh,height=-7.6131,steps=100,width=3.1099
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -0.44672713582845847
neg_mean_loss: 0.44672713582845847
node_ip: 127.0.0.1
pid: 48447
time_since_restore: 11.710935115814209
time_this_iter_s: 0.1116342544555664
time_total_s: 11.710935115814209
timestamp: 1658501201
timesteps_since_restore: 0
training_iteration: 100
trial_id: 107d75f0
warmup_time: 0.0029137134552001953
Result for objective_18fbb20a:
date: 2022-07-22_15-46-43
done: false
experiment_id: 5e399bc125a548739dd433f4153019fe
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 13.355684066332913
neg_mean_loss: -13.355684066332913
node_ip: 127.0.0.1
pid: 48475
time_since_restore: 0.10402607917785645
time_this_iter_s: 0.10402607917785645
time_total_s: 0.10402607917785645
timestamp: 1658501203
timesteps_since_restore: 0
training_iteration: 1
trial_id: 18fbb20a
warmup_time: 0.003000974655151367
Result for objective_190ab4d0:
date: 2022-07-22_15-46-43
done: false
experiment_id: 1b007a40cf6d425085379a4314739912
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 13.48520025532623
neg_mean_loss: -13.48520025532623
node_ip: 127.0.0.1
pid: 48478
time_since_restore: 0.10344505310058594
time_this_iter_s: 0.10344505310058594
time_total_s: 0.10344505310058594
timestamp: 1658501203
timesteps_since_restore: 0
training_iteration: 1
trial_id: 190ab4d0
warmup_time: 0.0028760433197021484
Result for objective_191ca3de:
date: 2022-07-22_15-46-43
done: false
experiment_id: e3e72bbeb8ae46f0a4f1624db62bc730
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 8.651082223836086
neg_mean_loss: -8.651082223836086
node_ip: 127.0.0.1
pid: 48481
time_since_restore: 0.1036369800567627
time_this_iter_s: 0.1036369800567627
time_total_s: 0.1036369800567627
timestamp: 1658501203
timesteps_since_restore: 0
training_iteration: 1
trial_id: 191ca3de
warmup_time: 0.0029621124267578125
Result for objective_17033b12:
date: 2022-07-22_15-46-45
done: false
experiment_id: b5f76922e6424055b6aa255677e2f4b8
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -8.746754140047871
neg_mean_loss: 8.746754140047871
node_ip: 127.0.0.1
pid: 48470
time_since_restore: 5.144078969955444
time_this_iter_s: 0.10620617866516113
time_total_s: 5.144078969955444
timestamp: 1658501205
timesteps_since_restore: 0
training_iteration: 48
trial_id: 17033b12
warmup_time: 0.0029649734497070312
Result for objective_18fbb20a:
date: 2022-07-22_15-46-48
done: false
experiment_id: 5e399bc125a548739dd433f4153019fe
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 3.5691516291476226
neg_mean_loss: -3.5691516291476226
node_ip: 127.0.0.1
pid: 48475
time_since_restore: 5.178721189498901
time_this_iter_s: 0.10647201538085938
time_total_s: 5.178721189498901
timestamp: 1658501208
timesteps_since_restore: 0
training_iteration: 48
trial_id: 18fbb20a
warmup_time: 0.003000974655151367
Result for objective_190ab4d0:
date: 2022-07-22_15-46-48
done: false
experiment_id: 1b007a40cf6d425085379a4314739912
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 3.9678564240341583
neg_mean_loss: -3.9678564240341583
node_ip: 127.0.0.1
pid: 48478
time_since_restore: 5.156033754348755
time_this_iter_s: 0.10652899742126465
time_total_s: 5.156033754348755
timestamp: 1658501208
timesteps_since_restore: 0
training_iteration: 48
trial_id: 190ab4d0
warmup_time: 0.0028760433197021484
Result for objective_191ca3de:
date: 2022-07-22_15-46-48
done: false
experiment_id: e3e72bbeb8ae46f0a4f1624db62bc730
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -1.04293599865772
neg_mean_loss: 1.04293599865772
node_ip: 127.0.0.1
pid: 48481
time_since_restore: 5.1839048862457275
time_this_iter_s: 0.10760998725891113
time_total_s: 5.1839048862457275
timestamp: 1658501208
timesteps_since_restore: 0
training_iteration: 48
trial_id: 191ca3de
warmup_time: 0.0029621124267578125
Result for objective_17033b12:
date: 2022-07-22_15-46-50
done: false
experiment_id: b5f76922e6424055b6aa255677e2f4b8
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: -8.832083655687075
neg_mean_loss: 8.832083655687075
node_ip: 127.0.0.1
pid: 48470
time_since_restore: 10.193100929260254
time_this_iter_s: 0.10694503784179688
time_total_s: 10.193100929260254
timestamp: 1658501210
timesteps_since_restore: 0
training_iteration: 95
trial_id: 17033b12
warmup_time: 0.0029649734497070312
Result for objective_17033b12:
date: 2022-07-22_15-46-50
done: true
experiment_id: b5f76922e6424055b6aa255677e2f4b8
experiment_tag: 5_activation=tanh,height=-89.1891,steps=100,width=12.1463
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -8.836432551723352
neg_mean_loss: 8.836432551723352
node_ip: 127.0.0.1
pid: 48470
time_since_restore: 10.725535869598389
time_this_iter_s: 0.10705089569091797
time_total_s: 10.725535869598389
timestamp: 1658501210
timesteps_since_restore: 0
training_iteration: 100
trial_id: 17033b12
warmup_time: 0.0029649734497070312
Result for objective_1eec012e:
date: 2022-07-22_15-46-53
done: false
experiment_id: a3cab04eb23543ddbf9b1daa9e915851
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 4.9718091604083385
neg_mean_loss: -4.9718091604083385
node_ip: 127.0.0.1
pid: 48494
time_since_restore: 0.10102462768554688
time_this_iter_s: 0.10102462768554688
time_total_s: 0.10102462768554688
timestamp: 1658501213
timesteps_since_restore: 0
training_iteration: 1
trial_id: 1eec012e
warmup_time: 0.002748250961303711
Result for objective_18fbb20a:
date: 2022-07-22_15-46-53
done: false
experiment_id: 5e399bc125a548739dd433f4153019fe
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 3.4635693481489094
neg_mean_loss: -3.4635693481489094
node_ip: 127.0.0.1
pid: 48475
time_since_restore: 10.22235107421875
time_this_iter_s: 0.10835385322570801
time_total_s: 10.22235107421875
timestamp: 1658501213
timesteps_since_restore: 0
training_iteration: 95
trial_id: 18fbb20a
warmup_time: 0.003000974655151367
Result for objective_190ab4d0:
date: 2022-07-22_15-46-53
done: false
experiment_id: 1b007a40cf6d425085379a4314739912
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 3.732496287450277
neg_mean_loss: -3.732496287450277
node_ip: 127.0.0.1
pid: 48478
time_since_restore: 10.17775583267212
time_this_iter_s: 0.10712075233459473
time_total_s: 10.17775583267212
timestamp: 1658501213
timesteps_since_restore: 0
training_iteration: 95
trial_id: 190ab4d0
warmup_time: 0.0028760433197021484
Result for objective_191ca3de:
date: 2022-07-22_15-46-54
done: false
experiment_id: e3e72bbeb8ae46f0a4f1624db62bc730
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: -1.193549900472242
neg_mean_loss: 1.193549900472242
node_ip: 127.0.0.1
pid: 48481
time_since_restore: 10.230173826217651
time_this_iter_s: 0.10641312599182129
time_total_s: 10.230173826217651
timestamp: 1658501214
timesteps_since_restore: 0
training_iteration: 95
trial_id: 191ca3de
warmup_time: 0.0029621124267578125
Result for objective_18fbb20a:
date: 2022-07-22_15-46-54
done: true
experiment_id: 5e399bc125a548739dd433f4153019fe
experiment_tag: 6_activation=tanh,height=33.5568,steps=100,width=9.7544
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 3.4581764420917516
neg_mean_loss: -3.4581764420917516
node_ip: 127.0.0.1
pid: 48475
time_since_restore: 10.756349325180054
time_this_iter_s: 0.10465312004089355
time_total_s: 10.756349325180054
timestamp: 1658501214
timesteps_since_restore: 0
training_iteration: 100
trial_id: 18fbb20a
warmup_time: 0.003000974655151367
Result for objective_190ab4d0:
date: 2022-07-22_15-46-54
done: true
experiment_id: 1b007a40cf6d425085379a4314739912
experiment_tag: 7_activation=relu,height=-65.1480,steps=100,width=4.1955
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 3.7203002216299694
neg_mean_loss: -3.7203002216299694
node_ip: 127.0.0.1
pid: 48478
time_since_restore: 10.712332725524902
time_this_iter_s: 0.10630083084106445
time_total_s: 10.712332725524902
timestamp: 1658501214
timesteps_since_restore: 0
training_iteration: 100
trial_id: 190ab4d0
warmup_time: 0.0028760433197021484
Result for objective_20e93604:
date: 2022-07-22_15-46-56
done: false
experiment_id: e2c12d5518de42acb5f8413171242ef8
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 1.0094856921511255
neg_mean_loss: -1.0094856921511255
node_ip: 127.0.0.1
pid: 48502
time_since_restore: 0.10361671447753906
time_this_iter_s: 0.10361671447753906
time_total_s: 0.10361671447753906
timestamp: 1658501216
timesteps_since_restore: 0
training_iteration: 1
trial_id: 20e93604
warmup_time: 0.0035440921783447266
Result for objective_191ca3de:
date: 2022-07-22_15-46-57
done: true
experiment_id: e3e72bbeb8ae46f0a4f1624db62bc730
experiment_tag: 8_activation=tanh,height=-13.4892,steps=100,width=6.7408
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -1.2012809142667369
neg_mean_loss: 1.2012809142667369
node_ip: 127.0.0.1
pid: 48481
time_since_restore: 13.218971729278564
time_this_iter_s: 2.560670852661133
time_total_s: 13.218971729278564
timestamp: 1658501217
timesteps_since_restore: 0
training_iteration: 100
trial_id: 191ca3de
warmup_time: 0.0029621124267578125
Result for objective_1eec012e:
date: 2022-07-22_15-46-58
done: false
experiment_id: a3cab04eb23543ddbf9b1daa9e915851
hostname: Kais-MacBook-Pro.local
iterations: 24
iterations_since_restore: 25
mean_loss: -4.754884423931921
neg_mean_loss: 4.754884423931921
node_ip: 127.0.0.1
pid: 48494
time_since_restore: 5.151007890701294
time_this_iter_s: 0.1075141429901123
time_total_s: 5.151007890701294
timestamp: 1658501218
timesteps_since_restore: 0
training_iteration: 25
trial_id: 1eec012e
warmup_time: 0.002748250961303711
Result for objective_20e93604:
date: 2022-07-22_15-47-02
done: false
experiment_id: e2c12d5518de42acb5f8413171242ef8
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -8.738995488800374
neg_mean_loss: 8.738995488800374
node_ip: 127.0.0.1
pid: 48502
time_since_restore: 5.183563709259033
time_this_iter_s: 0.10645675659179688
time_total_s: 5.183563709259033
timestamp: 1658501222
timesteps_since_restore: 0
training_iteration: 48
trial_id: 20e93604
warmup_time: 0.0035440921783447266
Result for objective_1eec012e:
date: 2022-07-22_15-47-03
done: false
experiment_id: a3cab04eb23543ddbf9b1daa9e915851
hostname: Kais-MacBook-Pro.local
iterations: 71
iterations_since_restore: 72
mean_loss: -4.9341033303033965
neg_mean_loss: 4.9341033303033965
node_ip: 127.0.0.1
pid: 48494
time_since_restore: 10.200856685638428
time_this_iter_s: 0.10717487335205078
time_total_s: 10.200856685638428
timestamp: 1658501223
timesteps_since_restore: 0
training_iteration: 72
trial_id: 1eec012e
warmup_time: 0.002748250961303711
Result for objective_1eec012e:
date: 2022-07-22_15-47-06
done: true
experiment_id: a3cab04eb23543ddbf9b1daa9e915851
experiment_tag: 9_activation=tanh,height=-50.2819,steps=100,width=14.8287
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -4.960533899755889
neg_mean_loss: 4.960533899755889
node_ip: 127.0.0.1
pid: 48494
time_since_restore: 13.22315764427185
time_this_iter_s: 0.10771775245666504
time_total_s: 13.22315764427185
timestamp: 1658501226
timesteps_since_restore: 0
training_iteration: 100
trial_id: 1eec012e
warmup_time: 0.002748250961303711
Result for objective_20e93604:
date: 2022-07-22_15-47-07
done: false
experiment_id: e2c12d5518de42acb5f8413171242ef8
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: -8.863153212712554
neg_mean_loss: 8.863153212712554
node_ip: 127.0.0.1
pid: 48502
time_since_restore: 10.251462936401367
time_this_iter_s: 0.1052391529083252
time_total_s: 10.251462936401367
timestamp: 1658501227
timesteps_since_restore: 0
training_iteration: 95
trial_id: 20e93604
warmup_time: 0.0035440921783447266
Result for objective_20e93604:
date: 2022-07-22_15-47-07
done: true
experiment_id: e2c12d5518de42acb5f8413171242ef8
experiment_tag: 10_activation=tanh,height=-89.9051,steps=100,width=8.2465
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -8.869507755259557
neg_mean_loss: 8.869507755259557
node_ip: 127.0.0.1
pid: 48502
time_since_restore: 10.792518854141235
time_this_iter_s: 0.10802984237670898
time_total_s: 10.792518854141235
timestamp: 1658501227
timesteps_since_restore: 0
training_iteration: 100
trial_id: 20e93604
warmup_time: 0.0035440921783447266
We now have hyperparameters found to minimize the mean loss.
print("Best hyperparameters found were: ", results.get_best_result().config)
Best hyperparameters found were: {'steps': 100, 'width': 8.246480084328711, 'height': -89.90514307848873, 'activation': 'tanh'}
Providing an initial set of hyperparameters#
While defining the search algorithm, we may choose to provide an initial set of hyperparameters that we believe are especially promising or informative, and
pass this information as a helpful starting point for the SkOptSearch object. We also can pass the known rewards for these initial params to save on unnecessary computation.
initial_params = [
{"width": 10, "height": 0, "activation": "relu"},
{"width": 15, "height": -20, "activation": "tanh"}
]
known_rewards = [-189, -1144]
Now the search_alg built using SkOptSearch takes points_to_evaluate.
algo = SkOptSearch(points_to_evaluate=initial_params)
algo = ConcurrencyLimiter(algo, max_concurrent=4)
And again run the experiment, this time with initial hyperparameter evaluations:
tuner = tune.Tuner(
objective,
tune_config=tune.TuneConfig(
metric="mean_loss",
mode="min",
search_alg=algo,
num_samples=num_samples,
),
param_space=search_space,
)
results = tuner.fit()
== Status ==
Current time: 2022-07-22 15:49:08 (running for 00:00:43.84)
Memory usage on this node: 10.5/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.42 GiB heap, 0.0/2.0 GiB objects
Current best trial: 60e6657e with mean_loss=-6.564839315448637 and parameters={'steps': 100, 'width': 11.45060059898019, 'height': -66.52281754112352, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-48-24
Number of trials: 10/10 (10 TERMINATED)
Current time: 2022-07-22 15:49:08 (running for 00:00:43.84)
Memory usage on this node: 10.5/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.42 GiB heap, 0.0/2.0 GiB objects
Current best trial: 60e6657e with mean_loss=-6.564839315448637 and parameters={'steps': 100, 'width': 11.45060059898019, 'height': -66.52281754112352, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-48-24
Number of trials: 10/10 (10 TERMINATED)
| Trial name | status | loc | activation | height | width | loss | iter | total time (s) | iterations | neg_mean_loss |
|---|---|---|---|---|---|---|---|---|---|---|
| objective_56a79fce | TERMINATED | 127.0.0.1:48590 | relu | 0 | 10 | 10.1 | 100 | 10.9393 | 99 | -10.1 |
| objective_583ee2ca | TERMINATED | 127.0.0.1:48605 | tanh | -20 | 15 | -1.93311 | 100 | 11.6366 | 99 | 1.93311 |
| objective_58405952 | TERMINATED | 127.0.0.1:48606 | relu | -87.8025 | 19.608 | 1.271 | 100 | 11.752 | 99 | -1.271 |
| objective_5841ebe6 | TERMINATED | 127.0.0.1:48607 | relu | 49.51 | 15.0894 | 15.0175 | 100 | 11.6761 | 99 | -15.0175 |
| objective_5ec4da6e | TERMINATED | 127.0.0.1:48629 | relu | 76.0837 | 13.7995 | 17.681 | 100 | 10.7667 | 99 | -17.681 |
| objective_60d7fb56 | TERMINATED | 127.0.0.1:48634 | relu | -50.4534 | 16.3133 | 5.0162 | 100 | 10.75 | 99 | -5.0162 |
| objective_60e6657e | TERMINATED | 127.0.0.1:48637 | tanh | -66.5228 | 11.4506 | -6.56484 | 100 | 10.759 | 99 | 6.56484 |
| objective_60f84e42 | TERMINATED | 127.0.0.1:48640 | tanh | 99.5965 | 9.04425 | 10.0701 | 100 | 10.71 | 99 | -10.0701 |
| objective_66bdb9e8 | TERMINATED | 127.0.0.1:48653 | tanh | -61.9547 | 12.3263 | -6.11419 | 100 | 10.724 | 99 | 6.11419 |
| objective_68c346b8 | TERMINATED | 127.0.0.1:48660 | tanh | 23.7916 | 2.108 | 2.83643 | 100 | 10.7375 | 99 | -2.83643 |
Result for objective_56a79fce:
date: 2022-07-22_15-48-27
done: false
experiment_id: 7046e554c1db465bbed80331b034b40f
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 20.0
neg_mean_loss: -20.0
node_ip: 127.0.0.1
pid: 48590
time_since_restore: 0.10438704490661621
time_this_iter_s: 0.10438704490661621
time_total_s: 0.10438704490661621
timestamp: 1658501307
timesteps_since_restore: 0
training_iteration: 1
trial_id: 56a79fce
warmup_time: 0.002905130386352539
Result for objective_583ee2ca:
date: 2022-07-22_15-48-30
done: false
experiment_id: 59c76129c7074e41ab39b330d898ba03
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 8.0
neg_mean_loss: -8.0
node_ip: 127.0.0.1
pid: 48605
time_since_restore: 0.10483098030090332
time_this_iter_s: 0.10483098030090332
time_total_s: 0.10483098030090332
timestamp: 1658501310
timesteps_since_restore: 0
training_iteration: 1
trial_id: 583ee2ca
warmup_time: 0.0031731128692626953
Result for objective_5841ebe6:
date: 2022-07-22_15-48-30
done: false
experiment_id: fde2877ded13416195ab1af070eda91a
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 24.951000429599397
neg_mean_loss: -24.951000429599397
node_ip: 127.0.0.1
pid: 48607
time_since_restore: 0.10440397262573242
time_this_iter_s: 0.10440397262573242
time_total_s: 0.10440397262573242
timestamp: 1658501310
timesteps_since_restore: 0
training_iteration: 1
trial_id: 5841ebe6
warmup_time: 0.0033330917358398438
Result for objective_58405952:
date: 2022-07-22_15-48-30
done: false
experiment_id: 5902a2d532b3437e8b94d02919e1604c
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 11.21975075417543
neg_mean_loss: -11.21975075417543
node_ip: 127.0.0.1
pid: 48606
time_since_restore: 0.10400986671447754
time_this_iter_s: 0.10400986671447754
time_total_s: 0.10400986671447754
timestamp: 1658501310
timesteps_since_restore: 0
training_iteration: 1
trial_id: '58405952'
warmup_time: 0.0027670860290527344
Result for objective_56a79fce:
date: 2022-07-22_15-48-32
done: false
experiment_id: 7046e554c1db465bbed80331b034b40f
hostname: Kais-MacBook-Pro.local
iterations: 45
iterations_since_restore: 46
mean_loss: 10.217391304347826
neg_mean_loss: -10.217391304347826
node_ip: 127.0.0.1
pid: 48590
time_since_restore: 5.135921239852905
time_this_iter_s: 0.10727715492248535
time_total_s: 5.135921239852905
timestamp: 1658501312
timesteps_since_restore: 0
training_iteration: 46
trial_id: 56a79fce
warmup_time: 0.002905130386352539
Result for objective_583ee2ca:
date: 2022-07-22_15-48-35
done: false
experiment_id: 59c76129c7074e41ab39b330d898ba03
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -1.8601398601398602
neg_mean_loss: 1.8601398601398602
node_ip: 127.0.0.1
pid: 48605
time_since_restore: 5.154622793197632
time_this_iter_s: 0.10759091377258301
time_total_s: 5.154622793197632
timestamp: 1658501315
timesteps_since_restore: 0
training_iteration: 48
trial_id: 583ee2ca
warmup_time: 0.0031731128692626953
Result for objective_5841ebe6:
date: 2022-07-22_15-48-35
done: false
experiment_id: fde2877ded13416195ab1af070eda91a
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 15.090043780907216
neg_mean_loss: -15.090043780907216
node_ip: 127.0.0.1
pid: 48607
time_since_restore: 5.153692960739136
time_this_iter_s: 0.10715007781982422
time_total_s: 5.153692960739136
timestamp: 1658501315
timesteps_since_restore: 0
training_iteration: 48
trial_id: 5841ebe6
warmup_time: 0.0033330917358398438
Result for objective_58405952:
date: 2022-07-22_15-48-35
done: false
experiment_id: 5902a2d532b3437e8b94d02919e1604c
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 1.3270955262548618
neg_mean_loss: -1.3270955262548618
node_ip: 127.0.0.1
pid: 48606
time_since_restore: 5.145922660827637
time_this_iter_s: 0.10549187660217285
time_total_s: 5.145922660827637
timestamp: 1658501315
timesteps_since_restore: 0
training_iteration: 48
trial_id: '58405952'
warmup_time: 0.0027670860290527344
Result for objective_56a79fce:
date: 2022-07-22_15-48-37
done: false
experiment_id: 7046e554c1db465bbed80331b034b40f
hostname: Kais-MacBook-Pro.local
iterations: 92
iterations_since_restore: 93
mean_loss: 10.10752688172043
neg_mean_loss: -10.10752688172043
node_ip: 127.0.0.1
pid: 48590
time_since_restore: 10.195838212966919
time_this_iter_s: 0.10654616355895996
time_total_s: 10.195838212966919
timestamp: 1658501317
timesteps_since_restore: 0
training_iteration: 93
trial_id: 56a79fce
warmup_time: 0.002905130386352539
Result for objective_56a79fce:
date: 2022-07-22_15-48-38
done: true
experiment_id: 7046e554c1db465bbed80331b034b40f
experiment_tag: 1_activation=relu,height=0,steps=100,width=10
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 10.1
neg_mean_loss: -10.1
node_ip: 127.0.0.1
pid: 48590
time_since_restore: 10.939304113388062
time_this_iter_s: 0.10537099838256836
time_total_s: 10.939304113388062
timestamp: 1658501318
timesteps_since_restore: 0
training_iteration: 100
trial_id: 56a79fce
warmup_time: 0.002905130386352539
Result for objective_5841ebe6:
date: 2022-07-22_15-48-39
done: false
experiment_id: fde2877ded13416195ab1af070eda91a
hostname: Kais-MacBook-Pro.local
iterations: 90
iterations_since_restore: 91
mean_loss: 15.024097555037425
neg_mean_loss: -15.024097555037425
node_ip: 127.0.0.1
pid: 48607
time_since_restore: 9.817665100097656
time_this_iter_s: 0.10596418380737305
time_total_s: 9.817665100097656
timestamp: 1658501319
timesteps_since_restore: 0
training_iteration: 91
trial_id: 5841ebe6
warmup_time: 0.0033330917358398438
Result for objective_58405952:
date: 2022-07-22_15-48-39
done: false
experiment_id: 5902a2d532b3437e8b94d02919e1604c
hostname: Kais-MacBook-Pro.local
iterations: 89
iterations_since_restore: 90
mean_loss: 1.2767270681910414
neg_mean_loss: -1.2767270681910414
node_ip: 127.0.0.1
pid: 48606
time_since_restore: 9.72554087638855
time_this_iter_s: 0.10558199882507324
time_total_s: 9.72554087638855
timestamp: 1658501319
timesteps_since_restore: 0
training_iteration: 90
trial_id: '58405952'
warmup_time: 0.0027670860290527344
Result for objective_583ee2ca:
date: 2022-07-22_15-48-39
done: false
experiment_id: 59c76129c7074e41ab39b330d898ba03
hostname: Kais-MacBook-Pro.local
iterations: 91
iterations_since_restore: 92
mean_loss: -1.9272727272727272
neg_mean_loss: 1.9272727272727272
node_ip: 127.0.0.1
pid: 48605
time_since_restore: 9.925616025924683
time_this_iter_s: 0.1071469783782959
time_total_s: 9.925616025924683
timestamp: 1658501319
timesteps_since_restore: 0
training_iteration: 92
trial_id: 583ee2ca
warmup_time: 0.0031731128692626953
Result for objective_5ec4da6e:
date: 2022-07-22_15-48-40
done: false
experiment_id: 24cf29dbb0e44fa7a2dbffe353a56c35
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 27.60836685236658
neg_mean_loss: -27.60836685236658
node_ip: 127.0.0.1
pid: 48629
time_since_restore: 0.10513186454772949
time_this_iter_s: 0.10513186454772949
time_total_s: 0.10513186454772949
timestamp: 1658501320
timesteps_since_restore: 0
training_iteration: 1
trial_id: 5ec4da6e
warmup_time: 0.002888917922973633
Result for objective_583ee2ca:
date: 2022-07-22_15-48-41
done: true
experiment_id: 59c76129c7074e41ab39b330d898ba03
experiment_tag: 2_activation=tanh,height=-20,steps=100,width=15
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -1.9331103678929766
neg_mean_loss: 1.9331103678929766
node_ip: 127.0.0.1
pid: 48605
time_since_restore: 11.636637926101685
time_this_iter_s: 0.10811305046081543
time_total_s: 11.636637926101685
timestamp: 1658501321
timesteps_since_restore: 0
training_iteration: 100
trial_id: 583ee2ca
warmup_time: 0.0031731128692626953
Result for objective_5841ebe6:
date: 2022-07-22_15-48-41
done: true
experiment_id: fde2877ded13416195ab1af070eda91a
experiment_tag: 4_activation=relu,height=49.5100,steps=100,width=15.0894
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 15.017496549772488
neg_mean_loss: -15.017496549772488
node_ip: 127.0.0.1
pid: 48607
time_since_restore: 11.676102876663208
time_this_iter_s: 0.1074519157409668
time_total_s: 11.676102876663208
timestamp: 1658501321
timesteps_since_restore: 0
training_iteration: 100
trial_id: 5841ebe6
warmup_time: 0.0033330917358398438
Result for objective_58405952:
date: 2022-07-22_15-48-41
done: true
experiment_id: 5902a2d532b3437e8b94d02919e1604c
experiment_tag: 3_activation=relu,height=-87.8025,steps=100,width=19.6080
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 1.271001380631752
neg_mean_loss: -1.271001380631752
node_ip: 127.0.0.1
pid: 48606
time_since_restore: 11.751991033554077
time_this_iter_s: 0.1101231575012207
time_total_s: 11.751991033554077
timestamp: 1658501321
timesteps_since_restore: 0
training_iteration: 100
trial_id: '58405952'
warmup_time: 0.0027670860290527344
Result for objective_60d7fb56:
date: 2022-07-22_15-48-44
done: false
experiment_id: 26d42c9dcc6e4354b3fa7355fc806522
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 14.954664969255782
neg_mean_loss: -14.954664969255782
node_ip: 127.0.0.1
pid: 48634
time_since_restore: 0.10394525527954102
time_this_iter_s: 0.10394525527954102
time_total_s: 0.10394525527954102
timestamp: 1658501324
timesteps_since_restore: 0
training_iteration: 1
trial_id: 60d7fb56
warmup_time: 0.003077268600463867
Result for objective_60e6657e:
date: 2022-07-22_15-48-44
done: false
experiment_id: 3f2014eb8b41451bb277cabed9951d90
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 3.3477182458876484
neg_mean_loss: -3.3477182458876484
node_ip: 127.0.0.1
pid: 48637
time_since_restore: 0.10410618782043457
time_this_iter_s: 0.10410618782043457
time_total_s: 0.10410618782043457
timestamp: 1658501324
timesteps_since_restore: 0
training_iteration: 1
trial_id: 60e6657e
warmup_time: 0.0029239654541015625
Result for objective_60f84e42:
date: 2022-07-22_15-48-44
done: false
experiment_id: 481882ef37194e6f995c4294a3453fd2
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 19.959649161689548
neg_mean_loss: -19.959649161689548
node_ip: 127.0.0.1
pid: 48640
time_since_restore: 0.10413289070129395
time_this_iter_s: 0.10413289070129395
time_total_s: 0.10413289070129395
timestamp: 1658501324
timesteps_since_restore: 0
training_iteration: 1
trial_id: 60f84e42
warmup_time: 0.0028841495513916016
Result for objective_5ec4da6e:
date: 2022-07-22_15-48-45
done: false
experiment_id: 24cf29dbb0e44fa7a2dbffe353a56c35
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 17.760209862707853
neg_mean_loss: -17.760209862707853
node_ip: 127.0.0.1
pid: 48629
time_since_restore: 5.151746988296509
time_this_iter_s: 0.10759496688842773
time_total_s: 5.151746988296509
timestamp: 1658501325
timesteps_since_restore: 0
training_iteration: 48
trial_id: 5ec4da6e
warmup_time: 0.002888917922973633
Result for objective_60d7fb56:
date: 2022-07-22_15-48-49
done: false
experiment_id: 26d42c9dcc6e4354b3fa7355fc806522
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 5.08341058857216
neg_mean_loss: -5.08341058857216
node_ip: 127.0.0.1
pid: 48634
time_since_restore: 5.146520137786865
time_this_iter_s: 0.10820508003234863
time_total_s: 5.146520137786865
timestamp: 1658501329
timesteps_since_restore: 0
training_iteration: 48
trial_id: 60d7fb56
warmup_time: 0.003077268600463867
Result for objective_60e6657e:
date: 2022-07-22_15-48-49
done: false
experiment_id: 3f2014eb8b41451bb277cabed9951d90
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -6.469859332234883
neg_mean_loss: 6.469859332234883
node_ip: 127.0.0.1
pid: 48637
time_since_restore: 5.153999090194702
time_this_iter_s: 0.10596871376037598
time_total_s: 5.153999090194702
timestamp: 1658501329
timesteps_since_restore: 0
training_iteration: 48
trial_id: 60e6657e
warmup_time: 0.0029239654541015625
Result for objective_60f84e42:
date: 2022-07-22_15-48-49
done: false
experiment_id: 481882ef37194e6f995c4294a3453fd2
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 10.189492101039557
neg_mean_loss: -10.189492101039557
node_ip: 127.0.0.1
pid: 48640
time_since_restore: 5.137938976287842
time_this_iter_s: 0.10651087760925293
time_total_s: 5.137938976287842
timestamp: 1658501329
timesteps_since_restore: 0
training_iteration: 48
trial_id: 60f84e42
warmup_time: 0.0028841495513916016
Result for objective_5ec4da6e:
date: 2022-07-22_15-48-50
done: false
experiment_id: 24cf29dbb0e44fa7a2dbffe353a56c35
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 17.684869174683147
neg_mean_loss: -17.684869174683147
node_ip: 127.0.0.1
pid: 48629
time_since_restore: 10.229070901870728
time_this_iter_s: 0.11342406272888184
time_total_s: 10.229070901870728
timestamp: 1658501330
timesteps_since_restore: 0
training_iteration: 95
trial_id: 5ec4da6e
warmup_time: 0.002888917922973633
Result for objective_5ec4da6e:
date: 2022-07-22_15-48-51
done: true
experiment_id: 24cf29dbb0e44fa7a2dbffe353a56c35
experiment_tag: 5_activation=relu,height=76.0837,steps=100,width=13.7995
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 17.6810334976324
neg_mean_loss: -17.6810334976324
node_ip: 127.0.0.1
pid: 48629
time_since_restore: 10.766729831695557
time_this_iter_s: 0.10697484016418457
time_total_s: 10.766729831695557
timestamp: 1658501331
timesteps_since_restore: 0
training_iteration: 100
trial_id: 5ec4da6e
warmup_time: 0.002888917922973633
Result for objective_66bdb9e8:
date: 2022-07-22_15-48-54
done: false
experiment_id: fae00fc97db044e9965698562b20039b
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 3.804528837217097
neg_mean_loss: -3.804528837217097
node_ip: 127.0.0.1
pid: 48653
time_since_restore: 0.10318303108215332
time_this_iter_s: 0.10318303108215332
time_total_s: 0.10318303108215332
timestamp: 1658501334
timesteps_since_restore: 0
training_iteration: 1
trial_id: 66bdb9e8
warmup_time: 0.004212856292724609
Result for objective_60d7fb56:
date: 2022-07-22_15-48-54
done: false
experiment_id: 26d42c9dcc6e4354b3fa7355fc806522
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 5.019454849577342
neg_mean_loss: -5.019454849577342
node_ip: 127.0.0.1
pid: 48634
time_since_restore: 10.212354183197021
time_this_iter_s: 0.10782504081726074
time_total_s: 10.212354183197021
timestamp: 1658501334
timesteps_since_restore: 0
training_iteration: 95
trial_id: 60d7fb56
warmup_time: 0.003077268600463867
Result for objective_60e6657e:
date: 2022-07-22_15-48-54
done: false
experiment_id: 3f2014eb8b41451bb277cabed9951d90
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: -6.560230936519563
neg_mean_loss: 6.560230936519563
node_ip: 127.0.0.1
pid: 48637
time_since_restore: 10.219763278961182
time_this_iter_s: 0.10767722129821777
time_total_s: 10.219763278961182
timestamp: 1658501334
timesteps_since_restore: 0
training_iteration: 95
trial_id: 60e6657e
warmup_time: 0.0029239654541015625
Result for objective_60f84e42:
date: 2022-07-22_15-48-54
done: false
experiment_id: 481882ef37194e6f995c4294a3453fd2
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 10.075906679849322
neg_mean_loss: -10.075906679849322
node_ip: 127.0.0.1
pid: 48640
time_since_restore: 10.16911792755127
time_this_iter_s: 0.10426807403564453
time_total_s: 10.16911792755127
timestamp: 1658501334
timesteps_since_restore: 0
training_iteration: 95
trial_id: 60f84e42
warmup_time: 0.0028841495513916016
Result for objective_60d7fb56:
date: 2022-07-22_15-48-54
done: true
experiment_id: 26d42c9dcc6e4354b3fa7355fc806522
experiment_tag: 6_activation=relu,height=-50.4534,steps=100,width=16.3133
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 5.016202769898167
neg_mean_loss: -5.016202769898167
node_ip: 127.0.0.1
pid: 48634
time_since_restore: 10.75002908706665
time_this_iter_s: 0.10717272758483887
time_total_s: 10.75002908706665
timestamp: 1658501334
timesteps_since_restore: 0
training_iteration: 100
trial_id: 60d7fb56
warmup_time: 0.003077268600463867
Result for objective_60e6657e:
date: 2022-07-22_15-48-54
done: true
experiment_id: 3f2014eb8b41451bb277cabed9951d90
experiment_tag: 7_activation=tanh,height=-66.5228,steps=100,width=11.4506
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -6.564839315448637
neg_mean_loss: 6.564839315448637
node_ip: 127.0.0.1
pid: 48637
time_since_restore: 10.759015321731567
time_this_iter_s: 0.10623526573181152
time_total_s: 10.759015321731567
timestamp: 1658501334
timesteps_since_restore: 0
training_iteration: 100
trial_id: 60e6657e
warmup_time: 0.0029239654541015625
Result for objective_60f84e42:
date: 2022-07-22_15-48-55
done: true
experiment_id: 481882ef37194e6f995c4294a3453fd2
experiment_tag: 8_activation=tanh,height=99.5965,steps=100,width=9.0442
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 10.070099940211907
neg_mean_loss: -10.070099940211907
node_ip: 127.0.0.1
pid: 48640
time_since_restore: 10.710038900375366
time_this_iter_s: 0.1064748764038086
time_total_s: 10.710038900375366
timestamp: 1658501335
timesteps_since_restore: 0
training_iteration: 100
trial_id: 60f84e42
warmup_time: 0.0028841495513916016
Result for objective_68c346b8:
date: 2022-07-22_15-48-57
done: false
experiment_id: fe241f21de6a4466bcdf713da6d4cd92
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 12.37916148081603
neg_mean_loss: -12.37916148081603
node_ip: 127.0.0.1
pid: 48660
time_since_restore: 0.10358977317810059
time_this_iter_s: 0.10358977317810059
time_total_s: 0.10358977317810059
timestamp: 1658501337
timesteps_since_restore: 0
training_iteration: 1
trial_id: 68c346b8
warmup_time: 0.0028688907623291016
Result for objective_66bdb9e8:
date: 2022-07-22_15-48-59
done: false
experiment_id: fae00fc97db044e9965698562b20039b
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -6.025789125550826
neg_mean_loss: 6.025789125550826
node_ip: 127.0.0.1
pid: 48653
time_since_restore: 5.1440629959106445
time_this_iter_s: 0.10767483711242676
time_total_s: 5.1440629959106445
timestamp: 1658501339
timesteps_since_restore: 0
training_iteration: 48
trial_id: 66bdb9e8
warmup_time: 0.004212856292724609
Result for objective_68c346b8:
date: 2022-07-22_15-49-02
done: false
experiment_id: fe241f21de6a4466bcdf713da6d4cd92
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 3.295952835154158
neg_mean_loss: -3.295952835154158
node_ip: 127.0.0.1
pid: 48660
time_since_restore: 5.15015983581543
time_this_iter_s: 0.10423922538757324
time_total_s: 5.15015983581543
timestamp: 1658501342
timesteps_since_restore: 0
training_iteration: 48
trial_id: 68c346b8
warmup_time: 0.0028688907623291016
Result for objective_66bdb9e8:
date: 2022-07-22_15-49-04
done: false
experiment_id: fae00fc97db044e9965698562b20039b
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: -6.109904185213174
neg_mean_loss: 6.109904185213174
node_ip: 127.0.0.1
pid: 48653
time_since_restore: 10.1879301071167
time_this_iter_s: 0.10626506805419922
time_total_s: 10.1879301071167
timestamp: 1658501344
timesteps_since_restore: 0
training_iteration: 95
trial_id: 66bdb9e8
warmup_time: 0.004212856292724609
Result for objective_66bdb9e8:
date: 2022-07-22_15-49-04
done: true
experiment_id: fae00fc97db044e9965698562b20039b
experiment_tag: 9_activation=tanh,height=-61.9547,steps=100,width=12.3263
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -6.114190623827535
neg_mean_loss: 6.114190623827535
node_ip: 127.0.0.1
pid: 48653
time_since_restore: 10.724038124084473
time_this_iter_s: 0.1088099479675293
time_total_s: 10.724038124084473
timestamp: 1658501344
timesteps_since_restore: 0
training_iteration: 100
trial_id: 66bdb9e8
warmup_time: 0.004212856292724609
Result for objective_68c346b8:
date: 2022-07-22_15-49-07
done: false
experiment_id: fe241f21de6a4466bcdf713da6d4cd92
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 2.859579303290837
neg_mean_loss: -2.859579303290837
node_ip: 127.0.0.1
pid: 48660
time_since_restore: 10.199071884155273
time_this_iter_s: 0.10729789733886719
time_total_s: 10.199071884155273
timestamp: 1658501347
timesteps_since_restore: 0
training_iteration: 95
trial_id: 68c346b8
warmup_time: 0.0028688907623291016
Result for objective_68c346b8:
date: 2022-07-22_15-49-08
done: true
experiment_id: fe241f21de6a4466bcdf713da6d4cd92
experiment_tag: 10_activation=tanh,height=23.7916,steps=100,width=2.1080
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 2.836425260080649
neg_mean_loss: -2.836425260080649
node_ip: 127.0.0.1
pid: 48660
time_since_restore: 10.7375328540802
time_this_iter_s: 0.1081850528717041
time_total_s: 10.7375328540802
timestamp: 1658501348
timesteps_since_restore: 0
training_iteration: 100
trial_id: 68c346b8
warmup_time: 0.0028688907623291016
And we again show the ideal hyperparameters.
print("Best hyperparameters found were: ", results.get_best_result().config)
Best hyperparameters found were: {'steps': 100, 'width': 11.45060059898019, 'height': -66.52281754112352, 'activation': 'tanh'}