Running Tune experiments with ZOOpt
Contents
Running Tune experiments with ZOOpt#
In this tutorial we introduce ZOOpt, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with ZOOpt and, as a result, allow you to seamlessly scale up a ZOOpt optimization process - without sacrificing performance.
Zeroth-order optimization (ZOOpt) does not rely on the gradient of the objective function, but instead, learns from samples of the search space. It is suitable for optimizing functions that are nondifferentiable, with many local minima, or even unknown but only testable. Therefore, zeroth-order optimization is commonly referred to as “derivative-free optimization” and “black-box optimization”. In this example we minimize a simple objective to briefly demonstrate the usage of ZOOpt with Ray Tune via ZOOptSearch. It’s useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume zoopt==0.4.1 library is installed. To learn more, please refer to the ZOOpt website.
Click below to see all the imports we need for this example. You can also launch directly into a Binder instance to run this notebook yourself. Just click on the rocket symbol at the top of the navigation.
import time
import ray
from ray import tune
from ray.air import session
from ray.tune.search.zoopt import ZOOptSearch
from zoopt import ValueType
Let’s start by defining a simple evaluation function.
We artificially sleep for a bit (0.1 seconds) to simulate a long-running ML experiment.
This setup assumes that we’re running multiple steps of an experiment and try to tune two hyperparameters,
namely width and height, and activation.
def evaluate(step, width, height):
time.sleep(0.1)
return (0.1 + width * step / 100) ** (-1) + height * 0.1
Next, our objective function takes a Tune config, evaluates the score of your experiment in a training loop,
and uses session.report to report the score back to Tune.
def objective(config):
for step in range(config["steps"]):
score = evaluate(step, config["width"], config["height"])
session.report({"iterations": step, "mean_loss": score})
Next we define a search space. The critical assumption is that the optimal hyperparameters live within this space. Yet, if the space is very large, then those hyperparameters may be difficult to find in a short amount of time.
search_config = {
"steps": 100,
"width": tune.randint(0, 10),
"height": tune.quniform(-10, 10, 1e-2),
"activation": tune.choice(["relu, tanh"])
}
The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to 1000 samples.
(you can decrease this if it takes too long on your machine).
num_samples = 1000
Next we define the search algorithm built from ZOOptSearch, constrained to a maximum of 8 concurrent trials via ZOOpt’s internal "parallel_num".
zoopt_config = {
"parallel_num": 8
}
algo = ZOOptSearch(
algo="Asracos", # only supports ASRacos currently
budget=num_samples,
**zoopt_config,
)
Finally, we run the experiment to "min"imize the “mean_loss” of the objective by searching search_config via algo, num_samples times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute tuner.fit().
tuner = tune.Tuner(
objective,
tune_config=tune.TuneConfig(
metric="mean_loss",
mode="min",
search_alg=algo,
num_samples=num_samples,
),
param_space=search_config,
)
results = tuner.fit()
Function checkpointing is disabled. This may result in unexpected behavior when using checkpointing features or certain schedulers. To enable, set the train function arguments to be `func(config, checkpoint_dir=None)`.
== Status ==
Current time: 2022-07-22 15:36:05 (running for 00:00:29.94)
Memory usage on this node: 11.1/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.39 GiB heap, 0.0/2.0 GiB objects
Current best trial: 963faf2a with mean_loss=-0.39843708609271516 and parameters={'steps': 100, 'width': 6, 'height': -5.64, 'activation': 'relu, tanh'}
Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-35-34
Number of trials: 7/10 (7 TERMINATED)
Current time: 2022-07-22 15:36:05 (running for 00:00:29.94)
Memory usage on this node: 11.1/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.39 GiB heap, 0.0/2.0 GiB objects
Current best trial: 963faf2a with mean_loss=-0.39843708609271516 and parameters={'steps': 100, 'width': 6, 'height': -5.64, 'activation': 'relu, tanh'}
Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-35-34
Number of trials: 7/10 (7 TERMINATED)
| Trial name | status | loc | activation | height | width | loss | iter | total time (s) | iterations | neg_mean_loss |
|---|---|---|---|---|---|---|---|---|---|---|
| objective_8c72f588 | TERMINATED | 127.0.0.1:47662 | relu, tanh | -3.94 | 0 | 9.606 | 100 | 10.9102 | 99 | -9.606 |
| objective_8e2f11ae | TERMINATED | 127.0.0.1:47667 | relu, tanh | -0.68 | 6 | 0.0975629 | 100 | 10.7479 | 99 | -0.0975629 |
| objective_8e30a596 | TERMINATED | 127.0.0.1:47668 | relu, tanh | -5.84 | 0 | 9.416 | 100 | 10.7724 | 99 | -9.416 |
| objective_8e32324e | TERMINATED | 127.0.0.1:47669 | relu, tanh | -2.15 | 3 | 0.110733 | 100 | 10.7684 | 99 | -0.110733 |
| objective_963faf2a | TERMINATED | 127.0.0.1:47689 | relu, tanh | -5.64 | 6 | -0.398437 | 100 | 10.8267 | 99 | 0.398437 |
| objective_96417df0 | TERMINATED | 127.0.0.1:47690 | relu, tanh | 2.84 | 6 | 0.449563 | 100 | 10.821 | 99 | -0.449563 |
| objective_96435da0 | TERMINATED | 127.0.0.1:47691 | relu, tanh | 2.6 | 6 | 0.425563 | 100 | 10.7694 | 99 | -0.425563 |
Result for objective_8c72f588:
date: 2022-07-22_15-35-38
done: false
experiment_id: 3eb9bcef55e341b0970abd6c1f97eda7
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 9.606
neg_mean_loss: -9.606
node_ip: 127.0.0.1
pid: 47662
time_since_restore: 0.10410094261169434
time_this_iter_s: 0.10410094261169434
time_total_s: 0.10410094261169434
timestamp: 1658500538
timesteps_since_restore: 0
training_iteration: 1
trial_id: 8c72f588
warmup_time: 0.003092050552368164
Result for objective_8e30a596:
date: 2022-07-22_15-35-41
done: false
experiment_id: d58453075b71453ab615e10ae9713072
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 9.416
neg_mean_loss: -9.416
node_ip: 127.0.0.1
pid: 47668
time_since_restore: 0.1051950454711914
time_this_iter_s: 0.1051950454711914
time_total_s: 0.1051950454711914
timestamp: 1658500541
timesteps_since_restore: 0
training_iteration: 1
trial_id: 8e30a596
warmup_time: 0.004169940948486328
Result for objective_8e32324e:
date: 2022-07-22_15-35-41
done: false
experiment_id: 22c7ba8baa2644479b661e6d91e5fae8
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 9.785
neg_mean_loss: -9.785
node_ip: 127.0.0.1
pid: 47669
time_since_restore: 0.10500001907348633
time_this_iter_s: 0.10500001907348633
time_total_s: 0.10500001907348633
timestamp: 1658500541
timesteps_since_restore: 0
training_iteration: 1
trial_id: 8e32324e
warmup_time: 0.004729032516479492
Result for objective_8e2f11ae:
date: 2022-07-22_15-35-41
done: false
experiment_id: c4d325574058491c8af3a0c869f0ebe5
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 9.932
neg_mean_loss: -9.932
node_ip: 127.0.0.1
pid: 47667
time_since_restore: 0.10310196876525879
time_this_iter_s: 0.10310196876525879
time_total_s: 0.10310196876525879
timestamp: 1658500541
timesteps_since_restore: 0
training_iteration: 1
trial_id: 8e2f11ae
warmup_time: 0.0029730796813964844
Result for objective_8c72f588:
date: 2022-07-22_15-35-43
done: false
experiment_id: 3eb9bcef55e341b0970abd6c1f97eda7
hostname: Kais-MacBook-Pro.local
iterations: 45
iterations_since_restore: 46
mean_loss: 9.606
neg_mean_loss: -9.606
node_ip: 127.0.0.1
pid: 47662
time_since_restore: 5.112913131713867
time_this_iter_s: 0.10695910453796387
time_total_s: 5.112913131713867
timestamp: 1658500543
timesteps_since_restore: 0
training_iteration: 46
trial_id: 8c72f588
warmup_time: 0.003092050552368164
Result for objective_8e30a596:
date: 2022-07-22_15-35-46
done: false
experiment_id: d58453075b71453ab615e10ae9713072
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 9.416
neg_mean_loss: -9.416
node_ip: 127.0.0.1
pid: 47668
time_since_restore: 5.1615166664123535
time_this_iter_s: 0.10595178604125977
time_total_s: 5.1615166664123535
timestamp: 1658500546
timesteps_since_restore: 0
training_iteration: 48
trial_id: 8e30a596
warmup_time: 0.004169940948486328
Result for objective_8e2f11ae:
date: 2022-07-22_15-35-46
done: false
experiment_id: c4d325574058491c8af3a0c869f0ebe5
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 0.2744657534246575
neg_mean_loss: -0.2744657534246575
node_ip: 127.0.0.1
pid: 47667
time_since_restore: 5.1498119831085205
time_this_iter_s: 0.10741090774536133
time_total_s: 5.1498119831085205
timestamp: 1658500546
timesteps_since_restore: 0
training_iteration: 48
trial_id: 8e2f11ae
warmup_time: 0.0029730796813964844
Result for objective_8e32324e:
date: 2022-07-22_15-35-46
done: false
experiment_id: 22c7ba8baa2644479b661e6d91e5fae8
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 0.44725165562913916
neg_mean_loss: -0.44725165562913916
node_ip: 127.0.0.1
pid: 47669
time_since_restore: 5.166383981704712
time_this_iter_s: 0.1064291000366211
time_total_s: 5.166383981704712
timestamp: 1658500546
timesteps_since_restore: 0
training_iteration: 48
trial_id: 8e32324e
warmup_time: 0.004729032516479492
Result for objective_8c72f588:
date: 2022-07-22_15-35-48
done: false
experiment_id: 3eb9bcef55e341b0970abd6c1f97eda7
hostname: Kais-MacBook-Pro.local
iterations: 92
iterations_since_restore: 93
mean_loss: 9.606
neg_mean_loss: -9.606
node_ip: 127.0.0.1
pid: 47662
time_since_restore: 10.156940937042236
time_this_iter_s: 0.10845208168029785
time_total_s: 10.156940937042236
timestamp: 1658500548
timesteps_since_restore: 0
training_iteration: 93
trial_id: 8c72f588
warmup_time: 0.003092050552368164
Result for objective_8c72f588:
date: 2022-07-22_15-35-49
done: true
experiment_id: 3eb9bcef55e341b0970abd6c1f97eda7
experiment_tag: 1_activation=relu_tanh,height=-3.9400,steps=100,width=0
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 9.606
neg_mean_loss: -9.606
node_ip: 127.0.0.1
pid: 47662
time_since_restore: 10.910246133804321
time_this_iter_s: 0.1059122085571289
time_total_s: 10.910246133804321
timestamp: 1658500549
timesteps_since_restore: 0
training_iteration: 100
trial_id: 8c72f588
warmup_time: 0.003092050552368164
Result for objective_8e2f11ae:
date: 2022-07-22_15-35-51
done: false
experiment_id: c4d325574058491c8af3a0c869f0ebe5
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 0.10621602787456447
neg_mean_loss: -0.10621602787456447
node_ip: 127.0.0.1
pid: 47667
time_since_restore: 10.211436986923218
time_this_iter_s: 0.10804891586303711
time_total_s: 10.211436986923218
timestamp: 1658500551
timesteps_since_restore: 0
training_iteration: 95
trial_id: 8e2f11ae
warmup_time: 0.0029730796813964844
Result for objective_8e32324e:
date: 2022-07-22_15-35-51
done: false
experiment_id: 22c7ba8baa2644479b661e6d91e5fae8
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 0.12746575342465752
neg_mean_loss: -0.12746575342465752
node_ip: 127.0.0.1
pid: 47669
time_since_restore: 10.228847980499268
time_this_iter_s: 0.10761308670043945
time_total_s: 10.228847980499268
timestamp: 1658500551
timesteps_since_restore: 0
training_iteration: 95
trial_id: 8e32324e
warmup_time: 0.004729032516479492
Result for objective_8e30a596:
date: 2022-07-22_15-35-51
done: false
experiment_id: d58453075b71453ab615e10ae9713072
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 9.416
neg_mean_loss: -9.416
node_ip: 127.0.0.1
pid: 47668
time_since_restore: 10.231056928634644
time_this_iter_s: 0.10677409172058105
time_total_s: 10.231056928634644
timestamp: 1658500551
timesteps_since_restore: 0
training_iteration: 95
trial_id: 8e30a596
warmup_time: 0.004169940948486328
Result for objective_8e2f11ae:
date: 2022-07-22_15-35-52
done: true
experiment_id: c4d325574058491c8af3a0c869f0ebe5
experiment_tag: 2_activation=relu_tanh,height=-0.6800,steps=100,width=6
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 0.09756291390728478
neg_mean_loss: -0.09756291390728478
node_ip: 127.0.0.1
pid: 47667
time_since_restore: 10.747868061065674
time_this_iter_s: 0.10819792747497559
time_total_s: 10.747868061065674
timestamp: 1658500552
timesteps_since_restore: 0
training_iteration: 100
trial_id: 8e2f11ae
warmup_time: 0.0029730796813964844
Result for objective_8e32324e:
date: 2022-07-22_15-35-52
done: true
experiment_id: 22c7ba8baa2644479b661e6d91e5fae8
experiment_tag: 4_activation=relu_tanh,height=-2.1500,steps=100,width=3
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 0.11073289902280128
neg_mean_loss: -0.11073289902280128
node_ip: 127.0.0.1
pid: 47669
time_since_restore: 10.768368005752563
time_this_iter_s: 0.10648989677429199
time_total_s: 10.768368005752563
timestamp: 1658500552
timesteps_since_restore: 0
training_iteration: 100
trial_id: 8e32324e
warmup_time: 0.004729032516479492
Result for objective_8e30a596:
date: 2022-07-22_15-35-52
done: true
experiment_id: d58453075b71453ab615e10ae9713072
experiment_tag: 3_activation=relu_tanh,height=-5.8400,steps=100,width=0
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 9.416
neg_mean_loss: -9.416
node_ip: 127.0.0.1
pid: 47668
time_since_restore: 10.7723867893219
time_this_iter_s: 0.10686278343200684
time_total_s: 10.7723867893219
timestamp: 1658500552
timesteps_since_restore: 0
training_iteration: 100
trial_id: 8e30a596
warmup_time: 0.004169940948486328
Result for objective_96417df0:
date: 2022-07-22_15-35-54
done: false
experiment_id: e98b245717b4423d8917589cf5d42088
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 10.284
neg_mean_loss: -10.284
node_ip: 127.0.0.1
pid: 47690
time_since_restore: 0.10359907150268555
time_this_iter_s: 0.10359907150268555
time_total_s: 0.10359907150268555
timestamp: 1658500554
timesteps_since_restore: 0
training_iteration: 1
trial_id: 96417df0
warmup_time: 0.003325939178466797
Result for objective_96435da0:
date: 2022-07-22_15-35-54
done: false
experiment_id: 8680a80a099c452dadd3be44d3457ca5
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 10.26
neg_mean_loss: -10.26
node_ip: 127.0.0.1
pid: 47691
time_since_restore: 0.10206389427185059
time_this_iter_s: 0.10206389427185059
time_total_s: 0.10206389427185059
timestamp: 1658500554
timesteps_since_restore: 0
training_iteration: 1
trial_id: 96435da0
warmup_time: 0.002891063690185547
Result for objective_963faf2a:
date: 2022-07-22_15-35-54
done: false
experiment_id: f28968c2634348c2ae4b0118cc844687
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 9.436
neg_mean_loss: -9.436
node_ip: 127.0.0.1
pid: 47689
time_since_restore: 0.10424089431762695
time_this_iter_s: 0.10424089431762695
time_total_s: 0.10424089431762695
timestamp: 1658500554
timesteps_since_restore: 0
training_iteration: 1
trial_id: 963faf2a
warmup_time: 0.002835988998413086
Result for objective_96417df0:
date: 2022-07-22_15-35-59
done: false
experiment_id: e98b245717b4423d8917589cf5d42088
hostname: Kais-MacBook-Pro.local
iterations: 46
iterations_since_restore: 47
mean_loss: 0.6336503496503496
neg_mean_loss: -0.6336503496503496
node_ip: 127.0.0.1
pid: 47690
time_since_restore: 5.134061098098755
time_this_iter_s: 0.1091001033782959
time_total_s: 5.134061098098755
timestamp: 1658500559
timesteps_since_restore: 0
training_iteration: 47
trial_id: 96417df0
warmup_time: 0.003325939178466797
Result for objective_96435da0:
date: 2022-07-22_15-35-59
done: false
experiment_id: 8680a80a099c452dadd3be44d3457ca5
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 0.6024657534246576
neg_mean_loss: -0.6024657534246576
node_ip: 127.0.0.1
pid: 47691
time_since_restore: 5.1902501583099365
time_this_iter_s: 0.10941314697265625
time_total_s: 5.1902501583099365
timestamp: 1658500559
timesteps_since_restore: 0
training_iteration: 48
trial_id: 96435da0
warmup_time: 0.002891063690185547
Result for objective_963faf2a:
date: 2022-07-22_15-35-59
done: false
experiment_id: f28968c2634348c2ae4b0118cc844687
hostname: Kais-MacBook-Pro.local
iterations: 46
iterations_since_restore: 47
mean_loss: -0.21434965034965026
neg_mean_loss: 0.21434965034965026
node_ip: 127.0.0.1
pid: 47689
time_since_restore: 5.137102127075195
time_this_iter_s: 0.10854196548461914
time_total_s: 5.137102127075195
timestamp: 1658500559
timesteps_since_restore: 0
training_iteration: 47
trial_id: 963faf2a
warmup_time: 0.002835988998413086
Result for objective_96417df0:
date: 2022-07-22_15-36-04
done: false
experiment_id: e98b245717b4423d8917589cf5d42088
hostname: Kais-MacBook-Pro.local
iterations: 93
iterations_since_restore: 94
mean_loss: 0.46005633802816903
neg_mean_loss: -0.46005633802816903
node_ip: 127.0.0.1
pid: 47690
time_since_restore: 10.175445079803467
time_this_iter_s: 0.10581207275390625
time_total_s: 10.175445079803467
timestamp: 1658500564
timesteps_since_restore: 0
training_iteration: 94
trial_id: 96417df0
warmup_time: 0.003325939178466797
Result for objective_96435da0:
date: 2022-07-22_15-36-05
done: false
experiment_id: 8680a80a099c452dadd3be44d3457ca5
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 0.4342160278745645
neg_mean_loss: -0.4342160278745645
node_ip: 127.0.0.1
pid: 47691
time_since_restore: 10.23218584060669
time_this_iter_s: 0.10732793807983398
time_total_s: 10.23218584060669
timestamp: 1658500565
timesteps_since_restore: 0
training_iteration: 95
trial_id: 96435da0
warmup_time: 0.002891063690185547
Result for objective_963faf2a:
date: 2022-07-22_15-36-05
done: false
experiment_id: f28968c2634348c2ae4b0118cc844687
hostname: Kais-MacBook-Pro.local
iterations: 93
iterations_since_restore: 94
mean_loss: -0.38794366197183094
neg_mean_loss: 0.38794366197183094
node_ip: 127.0.0.1
pid: 47689
time_since_restore: 10.181179761886597
time_this_iter_s: 0.10726284980773926
time_total_s: 10.181179761886597
timestamp: 1658500565
timesteps_since_restore: 0
training_iteration: 94
trial_id: 963faf2a
warmup_time: 0.002835988998413086
Result for objective_96417df0:
date: 2022-07-22_15-36-05
done: true
experiment_id: e98b245717b4423d8917589cf5d42088
experiment_tag: 6_activation=relu_tanh,height=2.8400,steps=100,width=6
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 0.44956291390728476
neg_mean_loss: -0.44956291390728476
node_ip: 127.0.0.1
pid: 47690
time_since_restore: 10.820996046066284
time_this_iter_s: 0.10588788986206055
time_total_s: 10.820996046066284
timestamp: 1658500565
timesteps_since_restore: 0
training_iteration: 100
trial_id: 96417df0
warmup_time: 0.003325939178466797
Result for objective_96435da0:
date: 2022-07-22_15-36-05
done: true
experiment_id: 8680a80a099c452dadd3be44d3457ca5
experiment_tag: 7_activation=relu_tanh,height=2.6000,steps=100,width=6
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 0.4255629139072848
neg_mean_loss: -0.4255629139072848
node_ip: 127.0.0.1
pid: 47691
time_since_restore: 10.769440174102783
time_this_iter_s: 0.10849618911743164
time_total_s: 10.769440174102783
timestamp: 1658500565
timesteps_since_restore: 0
training_iteration: 100
trial_id: 96435da0
warmup_time: 0.002891063690185547
Result for objective_963faf2a:
date: 2022-07-22_15-36-05
done: true
experiment_id: f28968c2634348c2ae4b0118cc844687
experiment_tag: 5_activation=relu_tanh,height=-5.6400,steps=100,width=6
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -0.39843708609271516
neg_mean_loss: 0.39843708609271516
node_ip: 127.0.0.1
pid: 47689
time_since_restore: 10.826673984527588
time_this_iter_s: 0.10800504684448242
time_total_s: 10.826673984527588
timestamp: 1658500565
timesteps_since_restore: 0
training_iteration: 100
trial_id: 963faf2a
warmup_time: 0.002835988998413086
Here are the hyperparamters found to minimize the mean loss of the defined objective.
print("Best hyperparameters found were: ", results.get_best_result().config)
Best hyperparameters found were: {'steps': 100, 'width': 6, 'height': -5.64, 'activation': 'relu, tanh'}
Optional: passing the parameter space into the search algorithm#
We can also pass the parameter space ourselves in the following formats:
continuous dimensions: (continuous, search_range, precision)
discrete dimensions: (discrete, search_range, has_order)
grid dimensions: (grid, grid_list)
space = {
"height": (ValueType.CONTINUOUS, [-10, 10], 1e-2),
"width": (ValueType.DISCRETE, [0, 10], True),
"layers": (ValueType.GRID, [4, 8, 16])
}
ZOOpt again handles constraining the amount of concurrent trials with "parallel_num".
zoopt_search_config = {
"parallel_num": 8,
"metric": "mean_loss",
"mode": "min"
}
algo = ZOOptSearch(
algo="Asracos",
budget=num_samples,
dim_dict=space,
**zoopt_search_config
)
This time we pass only "steps" and "activation" to the Tune config because "height" and "width" have been passed into ZOOptSearch to create the search_algo.
Again, we run the experiment to "min"imize the “mean_loss” of the objective by searching search_config via algo, num_samples times.
tuner = tune.Tuner(
objective,
tune_config=tune.TuneConfig(
metric="mean_loss",
mode="min",
search_alg=algo,
num_samples=num_samples,
),
param_space={"steps": 100},
)
results = tuner.fit()
== Status ==
Current time: 2022-07-22 15:36:35 (running for 00:00:29.76)
Memory usage on this node: 8.2/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.39 GiB heap, 0.0/2.0 GiB objects
Current best trial: a7fcf02e with mean_loss=-0.7457524893314368 and parameters={'steps': 100, 'height': -8.88, 'width': 7, 'layers': 16}
Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-36-05
Number of trials: 10/10 (10 TERMINATED)
Current time: 2022-07-22 15:36:35 (running for 00:00:29.76)
Memory usage on this node: 8.2/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.39 GiB heap, 0.0/2.0 GiB objects
Current best trial: a7fcf02e with mean_loss=-0.7457524893314368 and parameters={'steps': 100, 'height': -8.88, 'width': 7, 'layers': 16}
Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-36-05
Number of trials: 10/10 (10 TERMINATED)
| Trial name | status | loc | height | layers | width | loss | iter | total time (s) | iterations | neg_mean_loss |
|---|---|---|---|---|---|---|---|---|---|---|
| objective_9e64c92e | TERMINATED | 127.0.0.1:47713 | -8.08 | 16 | 0 | 9.192 | 100 | 10.7118 | 99 | -9.192 |
| objective_9ff31930 | TERMINATED | 127.0.0.1:47718 | 0.38 | 16 | 7 | 0.180248 | 100 | 10.7315 | 99 | -0.180248 |
| objective_9ff47d0c | TERMINATED | 127.0.0.1:47719 | 5.09 | 4 | 10 | 0.609 | 100 | 10.7924 | 99 | -0.609 |
| objective_9ff5c2b6 | TERMINATED | 127.0.0.1:47720 | 5.26 | 16 | 1 | 1.44343 | 100 | 10.7868 | 99 | -1.44343 |
| objective_a7f414d6 | TERMINATED | 127.0.0.1:47737 | 0.38 | 4 | 7 | 0.180248 | 100 | 10.7232 | 99 | -0.180248 |
| objective_a7f5c682 | TERMINATED | 127.0.0.1:47738 | -2.38 | 16 | 7 | -0.0957525 | 100 | 10.7337 | 99 | 0.0957525 |
| objective_a7f7c162 | TERMINATED | 127.0.0.1:47739 | 0.38 | 8 | 7 | 0.180248 | 100 | 10.7452 | 99 | -0.180248 |
| objective_a7f96fda | TERMINATED | 127.0.0.1:47740 | 0.38 | 16 | 4 | 0.284305 | 100 | 10.7079 | 99 | -0.284305 |
| objective_a7fb1844 | TERMINATED | 127.0.0.1:47741 | 0.38 | 8 | 7 | 0.180248 | 100 | 10.7157 | 99 | -0.180248 |
| objective_a7fcf02e | TERMINATED | 127.0.0.1:47742 | -8.88 | 16 | 7 | -0.745752 | 100 | 10.7305 | 99 | 0.745752 |
Result for objective_9e64c92e:
date: 2022-07-22_15-36-08
done: false
experiment_id: fafb7d88360d408286e616de3dcd4407
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 9.192
neg_mean_loss: -9.192
node_ip: 127.0.0.1
pid: 47713
time_since_restore: 0.1042020320892334
time_this_iter_s: 0.1042020320892334
time_total_s: 0.1042020320892334
timestamp: 1658500568
timesteps_since_restore: 0
training_iteration: 1
trial_id: 9e64c92e
warmup_time: 0.0030999183654785156
Result for objective_9ff47d0c:
date: 2022-07-22_15-36-11
done: false
experiment_id: de33a45d0c344d3aa344d8cff20b6c37
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 10.509
neg_mean_loss: -10.509
node_ip: 127.0.0.1
pid: 47719
time_since_restore: 0.10373878479003906
time_this_iter_s: 0.10373878479003906
time_total_s: 0.10373878479003906
timestamp: 1658500571
timesteps_since_restore: 0
training_iteration: 1
trial_id: 9ff47d0c
warmup_time: 0.003734111785888672
Result for objective_9ff31930:
date: 2022-07-22_15-36-11
done: false
experiment_id: 404b066d5e154070b5a3f6f6ef3acb33
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 10.038
neg_mean_loss: -10.038
node_ip: 127.0.0.1
pid: 47718
time_since_restore: 0.10415983200073242
time_this_iter_s: 0.10415983200073242
time_total_s: 0.10415983200073242
timestamp: 1658500571
timesteps_since_restore: 0
training_iteration: 1
trial_id: 9ff31930
warmup_time: 0.0033788681030273438
Result for objective_9ff5c2b6:
date: 2022-07-22_15-36-11
done: false
experiment_id: aeb7cb0b0c7c4f6692d47dcd1fe36462
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 10.526
neg_mean_loss: -10.526
node_ip: 127.0.0.1
pid: 47720
time_since_restore: 0.10468196868896484
time_this_iter_s: 0.10468196868896484
time_total_s: 0.10468196868896484
timestamp: 1658500571
timesteps_since_restore: 0
training_iteration: 1
trial_id: 9ff5c2b6
warmup_time: 0.0027132034301757812
Result for objective_9e64c92e:
date: 2022-07-22_15-36-13
done: false
experiment_id: fafb7d88360d408286e616de3dcd4407
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 9.192
neg_mean_loss: -9.192
node_ip: 127.0.0.1
pid: 47713
time_since_restore: 5.111851215362549
time_this_iter_s: 0.10771799087524414
time_total_s: 5.111851215362549
timestamp: 1658500573
timesteps_since_restore: 0
training_iteration: 48
trial_id: 9e64c92e
warmup_time: 0.0030999183654785156
Result for objective_9ff47d0c:
date: 2022-07-22_15-36-16
done: false
experiment_id: de33a45d0c344d3aa344d8cff20b6c37
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 0.7173333333333334
neg_mean_loss: -0.7173333333333334
node_ip: 127.0.0.1
pid: 47719
time_since_restore: 5.179923057556152
time_this_iter_s: 0.12629103660583496
time_total_s: 5.179923057556152
timestamp: 1658500576
timesteps_since_restore: 0
training_iteration: 48
trial_id: 9ff47d0c
warmup_time: 0.003734111785888672
Result for objective_9ff31930:
date: 2022-07-22_15-36-16
done: false
experiment_id: 404b066d5e154070b5a3f6f6ef3acb33
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 0.3329852507374631
neg_mean_loss: -0.3329852507374631
node_ip: 127.0.0.1
pid: 47718
time_since_restore: 5.155292749404907
time_this_iter_s: 0.10897374153137207
time_total_s: 5.155292749404907
timestamp: 1658500576
timesteps_since_restore: 0
training_iteration: 48
trial_id: 9ff31930
warmup_time: 0.0033788681030273438
Result for objective_9ff5c2b6:
date: 2022-07-22_15-36-16
done: false
experiment_id: aeb7cb0b0c7c4f6692d47dcd1fe36462
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 2.2803859649122806
neg_mean_loss: -2.2803859649122806
node_ip: 127.0.0.1
pid: 47720
time_since_restore: 5.177261829376221
time_this_iter_s: 0.10725784301757812
time_total_s: 5.177261829376221
timestamp: 1658500576
timesteps_since_restore: 0
training_iteration: 48
trial_id: 9ff5c2b6
warmup_time: 0.0027132034301757812
Result for objective_9e64c92e:
date: 2022-07-22_15-36-18
done: false
experiment_id: fafb7d88360d408286e616de3dcd4407
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 9.192
neg_mean_loss: -9.192
node_ip: 127.0.0.1
pid: 47713
time_since_restore: 10.170606136322021
time_this_iter_s: 0.10434508323669434
time_total_s: 10.170606136322021
timestamp: 1658500578
timesteps_since_restore: 0
training_iteration: 95
trial_id: 9e64c92e
warmup_time: 0.0030999183654785156
Result for objective_9e64c92e:
date: 2022-07-22_15-36-19
done: true
experiment_id: fafb7d88360d408286e616de3dcd4407
experiment_tag: 1_height=-8.0800,layers=16,steps=100,width=0
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 9.192
neg_mean_loss: -9.192
node_ip: 127.0.0.1
pid: 47713
time_since_restore: 10.711803197860718
time_this_iter_s: 0.10728001594543457
time_total_s: 10.711803197860718
timestamp: 1658500579
timesteps_since_restore: 0
training_iteration: 100
trial_id: 9e64c92e
warmup_time: 0.0030999183654785156
Result for objective_9ff31930:
date: 2022-07-22_15-36-21
done: false
experiment_id: 404b066d5e154070b5a3f6f6ef3acb33
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 0.18770059880239523
neg_mean_loss: -0.18770059880239523
node_ip: 127.0.0.1
pid: 47718
time_since_restore: 10.197859048843384
time_this_iter_s: 0.10710310935974121
time_total_s: 10.197859048843384
timestamp: 1658500581
timesteps_since_restore: 0
training_iteration: 95
trial_id: 9ff31930
warmup_time: 0.0033788681030273438
Result for objective_9ff47d0c:
date: 2022-07-22_15-36-21
done: false
experiment_id: de33a45d0c344d3aa344d8cff20b6c37
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 0.6142631578947368
neg_mean_loss: -0.6142631578947368
node_ip: 127.0.0.1
pid: 47719
time_since_restore: 10.256150722503662
time_this_iter_s: 0.11135077476501465
time_total_s: 10.256150722503662
timestamp: 1658500581
timesteps_since_restore: 0
training_iteration: 95
trial_id: 9ff47d0c
warmup_time: 0.003734111785888672
Result for objective_9ff5c2b6:
date: 2022-07-22_15-36-21
done: false
experiment_id: aeb7cb0b0c7c4f6692d47dcd1fe36462
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 1.4875384615384615
neg_mean_loss: -1.4875384615384615
node_ip: 127.0.0.1
pid: 47720
time_since_restore: 10.24931287765503
time_this_iter_s: 0.10830807685852051
time_total_s: 10.24931287765503
timestamp: 1658500581
timesteps_since_restore: 0
training_iteration: 95
trial_id: 9ff5c2b6
warmup_time: 0.0027132034301757812
Result for objective_9ff31930:
date: 2022-07-22_15-36-21
done: true
experiment_id: 404b066d5e154070b5a3f6f6ef3acb33
experiment_tag: 2_height=0.3800,layers=16,steps=100,width=7
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 0.18024751066856332
neg_mean_loss: -0.18024751066856332
node_ip: 127.0.0.1
pid: 47718
time_since_restore: 10.731510877609253
time_this_iter_s: 0.1073448657989502
time_total_s: 10.731510877609253
timestamp: 1658500581
timesteps_since_restore: 0
training_iteration: 100
trial_id: 9ff31930
warmup_time: 0.0033788681030273438
Result for objective_9ff47d0c:
date: 2022-07-22_15-36-21
done: true
experiment_id: de33a45d0c344d3aa344d8cff20b6c37
experiment_tag: 3_height=5.0900,layers=4,steps=100,width=10
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 0.609
neg_mean_loss: -0.609
node_ip: 127.0.0.1
pid: 47719
time_since_restore: 10.792427778244019
time_this_iter_s: 0.10576391220092773
time_total_s: 10.792427778244019
timestamp: 1658500581
timesteps_since_restore: 0
training_iteration: 100
trial_id: 9ff47d0c
warmup_time: 0.003734111785888672
Result for objective_9ff5c2b6:
date: 2022-07-22_15-36-21
done: true
experiment_id: aeb7cb0b0c7c4f6692d47dcd1fe36462
experiment_tag: 4_height=5.2600,layers=16,steps=100,width=1
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 1.4434311926605505
neg_mean_loss: -1.4434311926605505
node_ip: 127.0.0.1
pid: 47720
time_since_restore: 10.786811828613281
time_this_iter_s: 0.10388994216918945
time_total_s: 10.786811828613281
timestamp: 1658500581
timesteps_since_restore: 0
training_iteration: 100
trial_id: 9ff5c2b6
warmup_time: 0.0027132034301757812
Result for objective_a7fb1844:
date: 2022-07-22_15-36-24
done: false
experiment_id: 668080a52a3f4c4fad2258177b1fb755
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 10.038
neg_mean_loss: -10.038
node_ip: 127.0.0.1
pid: 47741
time_since_restore: 0.10408973693847656
time_this_iter_s: 0.10408973693847656
time_total_s: 0.10408973693847656
timestamp: 1658500584
timesteps_since_restore: 0
training_iteration: 1
trial_id: a7fb1844
warmup_time: 0.004377126693725586
Result for objective_a7f5c682:
date: 2022-07-22_15-36-24
done: false
experiment_id: 4449a3ef690c4b76a0cfbf7c4de7df43
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 9.762
neg_mean_loss: -9.762
node_ip: 127.0.0.1
pid: 47738
time_since_restore: 0.10507631301879883
time_this_iter_s: 0.10507631301879883
time_total_s: 0.10507631301879883
timestamp: 1658500584
timesteps_since_restore: 0
training_iteration: 1
trial_id: a7f5c682
warmup_time: 0.003899097442626953
Result for objective_a7f414d6:
date: 2022-07-22_15-36-24
done: false
experiment_id: 20575f602c234dd7a167a93ff353299b
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 10.038
neg_mean_loss: -10.038
node_ip: 127.0.0.1
pid: 47737
time_since_restore: 0.10182499885559082
time_this_iter_s: 0.10182499885559082
time_total_s: 0.10182499885559082
timestamp: 1658500584
timesteps_since_restore: 0
training_iteration: 1
trial_id: a7f414d6
warmup_time: 0.0046689510345458984
Result for objective_a7f7c162:
date: 2022-07-22_15-36-24
done: false
experiment_id: f4b185b58be045f594bf127addecbace
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 10.038
neg_mean_loss: -10.038
node_ip: 127.0.0.1
pid: 47739
time_since_restore: 0.10427188873291016
time_this_iter_s: 0.10427188873291016
time_total_s: 0.10427188873291016
timestamp: 1658500584
timesteps_since_restore: 0
training_iteration: 1
trial_id: a7f7c162
warmup_time: 0.004554033279418945
Result for objective_a7fcf02e:
date: 2022-07-22_15-36-24
done: false
experiment_id: 5646116e3b69493b920c031d27eeb11b
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 9.112
neg_mean_loss: -9.112
node_ip: 127.0.0.1
pid: 47742
time_since_restore: 0.10358119010925293
time_this_iter_s: 0.10358119010925293
time_total_s: 0.10358119010925293
timestamp: 1658500584
timesteps_since_restore: 0
training_iteration: 1
trial_id: a7fcf02e
warmup_time: 0.003072977066040039
Result for objective_a7f96fda:
date: 2022-07-22_15-36-25
done: false
experiment_id: 1983d487678f485a8a8ae7da4b7495a0
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 10.038
neg_mean_loss: -10.038
node_ip: 127.0.0.1
pid: 47740
time_since_restore: 0.10406088829040527
time_this_iter_s: 0.10406088829040527
time_total_s: 0.10406088829040527
timestamp: 1658500585
timesteps_since_restore: 0
training_iteration: 1
trial_id: a7f96fda
warmup_time: 0.0031909942626953125
Result for objective_a7fb1844:
date: 2022-07-22_15-36-29
done: false
experiment_id: 668080a52a3f4c4fad2258177b1fb755
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 0.3329852507374631
neg_mean_loss: -0.3329852507374631
node_ip: 127.0.0.1
pid: 47741
time_since_restore: 5.134153842926025
time_this_iter_s: 0.10844898223876953
time_total_s: 5.134153842926025
timestamp: 1658500589
timesteps_since_restore: 0
training_iteration: 48
trial_id: a7fb1844
warmup_time: 0.004377126693725586
Result for objective_a7f5c682:
date: 2022-07-22_15-36-29
done: false
experiment_id: 4449a3ef690c4b76a0cfbf7c4de7df43
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 0.05698525073746313
neg_mean_loss: -0.05698525073746313
node_ip: 127.0.0.1
pid: 47738
time_since_restore: 5.151050090789795
time_this_iter_s: 0.10810613632202148
time_total_s: 5.151050090789795
timestamp: 1658500589
timesteps_since_restore: 0
training_iteration: 48
trial_id: a7f5c682
warmup_time: 0.003899097442626953
Result for objective_a7fcf02e:
date: 2022-07-22_15-36-29
done: false
experiment_id: 5646116e3b69493b920c031d27eeb11b
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -0.593014749262537
neg_mean_loss: 0.593014749262537
node_ip: 127.0.0.1
pid: 47742
time_since_restore: 5.1300742626190186
time_this_iter_s: 0.10815715789794922
time_total_s: 5.1300742626190186
timestamp: 1658500589
timesteps_since_restore: 0
training_iteration: 48
trial_id: a7fcf02e
warmup_time: 0.003072977066040039
Result for objective_a7f414d6:
date: 2022-07-22_15-36-29
done: false
experiment_id: 20575f602c234dd7a167a93ff353299b
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 0.3329852507374631
neg_mean_loss: -0.3329852507374631
node_ip: 127.0.0.1
pid: 47737
time_since_restore: 5.151458024978638
time_this_iter_s: 0.10657620429992676
time_total_s: 5.151458024978638
timestamp: 1658500589
timesteps_since_restore: 0
training_iteration: 48
trial_id: a7f414d6
warmup_time: 0.0046689510345458984
Result for objective_a7f7c162:
date: 2022-07-22_15-36-29
done: false
experiment_id: f4b185b58be045f594bf127addecbace
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 0.3329852507374631
neg_mean_loss: -0.3329852507374631
node_ip: 127.0.0.1
pid: 47739
time_since_restore: 5.154465675354004
time_this_iter_s: 0.1063377857208252
time_total_s: 5.154465675354004
timestamp: 1658500589
timesteps_since_restore: 0
training_iteration: 48
trial_id: a7f7c162
warmup_time: 0.004554033279418945
Result for objective_a7f96fda:
date: 2022-07-22_15-36-30
done: false
experiment_id: 1983d487678f485a8a8ae7da4b7495a0
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 0.5430505050505051
neg_mean_loss: -0.5430505050505051
node_ip: 127.0.0.1
pid: 47740
time_since_restore: 5.135071039199829
time_this_iter_s: 0.10689830780029297
time_total_s: 5.135071039199829
timestamp: 1658500590
timesteps_since_restore: 0
training_iteration: 48
trial_id: a7f96fda
warmup_time: 0.0031909942626953125
Result for objective_a7fb1844:
date: 2022-07-22_15-36-35
done: false
experiment_id: 668080a52a3f4c4fad2258177b1fb755
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 0.18770059880239523
neg_mean_loss: -0.18770059880239523
node_ip: 127.0.0.1
pid: 47741
time_since_restore: 10.180493831634521
time_this_iter_s: 0.10809183120727539
time_total_s: 10.180493831634521
timestamp: 1658500595
timesteps_since_restore: 0
training_iteration: 95
trial_id: a7fb1844
warmup_time: 0.004377126693725586
Result for objective_a7f414d6:
date: 2022-07-22_15-36-35
done: false
experiment_id: 20575f602c234dd7a167a93ff353299b
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 0.18770059880239523
neg_mean_loss: -0.18770059880239523
node_ip: 127.0.0.1
pid: 47737
time_since_restore: 10.188506126403809
time_this_iter_s: 0.10684013366699219
time_total_s: 10.188506126403809
timestamp: 1658500595
timesteps_since_restore: 0
training_iteration: 95
trial_id: a7f414d6
warmup_time: 0.0046689510345458984
Result for objective_a7f5c682:
date: 2022-07-22_15-36-35
done: false
experiment_id: 4449a3ef690c4b76a0cfbf7c4de7df43
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: -0.08829940119760477
neg_mean_loss: 0.08829940119760477
node_ip: 127.0.0.1
pid: 47738
time_since_restore: 10.20038390159607
time_this_iter_s: 0.10745882987976074
time_total_s: 10.20038390159607
timestamp: 1658500595
timesteps_since_restore: 0
training_iteration: 95
trial_id: a7f5c682
warmup_time: 0.003899097442626953
Result for objective_a7f7c162:
date: 2022-07-22_15-36-35
done: false
experiment_id: f4b185b58be045f594bf127addecbace
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 0.18770059880239523
neg_mean_loss: -0.18770059880239523
node_ip: 127.0.0.1
pid: 47739
time_since_restore: 10.203772783279419
time_this_iter_s: 0.1077718734741211
time_total_s: 10.203772783279419
timestamp: 1658500595
timesteps_since_restore: 0
training_iteration: 95
trial_id: a7f7c162
warmup_time: 0.004554033279418945
Result for objective_a7fcf02e:
date: 2022-07-22_15-36-35
done: false
experiment_id: 5646116e3b69493b920c031d27eeb11b
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: -0.738299401197605
neg_mean_loss: 0.738299401197605
node_ip: 127.0.0.1
pid: 47742
time_since_restore: 10.187027215957642
time_this_iter_s: 0.10881304740905762
time_total_s: 10.187027215957642
timestamp: 1658500595
timesteps_since_restore: 0
training_iteration: 95
trial_id: a7fcf02e
warmup_time: 0.003072977066040039
Result for objective_a7f96fda:
date: 2022-07-22_15-36-35
done: false
experiment_id: 1983d487678f485a8a8ae7da4b7495a0
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 0.29706735751295343
neg_mean_loss: -0.29706735751295343
node_ip: 127.0.0.1
pid: 47740
time_since_restore: 10.170713901519775
time_this_iter_s: 0.10522603988647461
time_total_s: 10.170713901519775
timestamp: 1658500595
timesteps_since_restore: 0
training_iteration: 95
trial_id: a7f96fda
warmup_time: 0.0031909942626953125
Result for objective_a7fb1844:
date: 2022-07-22_15-36-35
done: true
experiment_id: 668080a52a3f4c4fad2258177b1fb755
experiment_tag: 9_height=0.3800,layers=8,steps=100,width=7
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 0.18024751066856332
neg_mean_loss: -0.18024751066856332
node_ip: 127.0.0.1
pid: 47741
time_since_restore: 10.715673923492432
time_this_iter_s: 0.10643696784973145
time_total_s: 10.715673923492432
timestamp: 1658500595
timesteps_since_restore: 0
training_iteration: 100
trial_id: a7fb1844
warmup_time: 0.004377126693725586
Result for objective_a7f414d6:
date: 2022-07-22_15-36-35
done: true
experiment_id: 20575f602c234dd7a167a93ff353299b
experiment_tag: 5_height=0.3800,layers=4,steps=100,width=7
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 0.18024751066856332
neg_mean_loss: -0.18024751066856332
node_ip: 127.0.0.1
pid: 47737
time_since_restore: 10.723177909851074
time_this_iter_s: 0.1052548885345459
time_total_s: 10.723177909851074
timestamp: 1658500595
timesteps_since_restore: 0
training_iteration: 100
trial_id: a7f414d6
warmup_time: 0.0046689510345458984
Result for objective_a7f5c682:
date: 2022-07-22_15-36-35
done: true
experiment_id: 4449a3ef690c4b76a0cfbf7c4de7df43
experiment_tag: 6_height=-2.3800,layers=16,steps=100,width=7
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -0.09575248933143668
neg_mean_loss: 0.09575248933143668
node_ip: 127.0.0.1
pid: 47738
time_since_restore: 10.733658075332642
time_this_iter_s: 0.10680294036865234
time_total_s: 10.733658075332642
timestamp: 1658500595
timesteps_since_restore: 0
training_iteration: 100
trial_id: a7f5c682
warmup_time: 0.003899097442626953
Result for objective_a7f7c162:
date: 2022-07-22_15-36-35
done: true
experiment_id: f4b185b58be045f594bf127addecbace
experiment_tag: 7_height=0.3800,layers=8,steps=100,width=7
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 0.18024751066856332
neg_mean_loss: -0.18024751066856332
node_ip: 127.0.0.1
pid: 47739
time_since_restore: 10.745207786560059
time_this_iter_s: 0.10880804061889648
time_total_s: 10.745207786560059
timestamp: 1658500595
timesteps_since_restore: 0
training_iteration: 100
trial_id: a7f7c162
warmup_time: 0.004554033279418945
Result for objective_a7fcf02e:
date: 2022-07-22_15-36-35
done: true
experiment_id: 5646116e3b69493b920c031d27eeb11b
experiment_tag: 10_height=-8.8800,layers=16,steps=100,width=7
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -0.7457524893314368
neg_mean_loss: 0.7457524893314368
node_ip: 127.0.0.1
pid: 47742
time_since_restore: 10.730549097061157
time_this_iter_s: 0.11018085479736328
time_total_s: 10.730549097061157
timestamp: 1658500595
timesteps_since_restore: 0
training_iteration: 100
trial_id: a7fcf02e
warmup_time: 0.003072977066040039
Result for objective_a7f96fda:
date: 2022-07-22_15-36-35
done: true
experiment_id: 1983d487678f485a8a8ae7da4b7495a0
experiment_tag: 8_height=0.3800,layers=16,steps=100,width=4
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 0.2843054187192119
neg_mean_loss: -0.2843054187192119
node_ip: 127.0.0.1
pid: 47740
time_since_restore: 10.707929849624634
time_this_iter_s: 0.10735177993774414
time_total_s: 10.707929849624634
timestamp: 1658500595
timesteps_since_restore: 0
training_iteration: 100
trial_id: a7f96fda
warmup_time: 0.0031909942626953125
Here are the hyperparamters found to minimize the mean loss of the defined objective.
print("Best hyperparameters found were: ", results.get_best_result().config)
Best hyperparameters found were: {'steps': 100, 'height': -8.88, 'width': 7, 'layers': 16}