Running Tune experiments with HEBOSearch
Running Tune experiments with HEBOSearch#
In this tutorial we introduce HEBO, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with ZOOpt and, as a result, allow you to seamlessly scale up a HEBO optimization process - without sacrificing performance.
Heteroscadastic Evolutionary Bayesian Optimization (HEBO) does not rely on the gradient of the objective function, but instead, learns from samples of the search space. It is suitable for optimizing functions that are nondifferentiable, with many local minima, or even unknown but only testable. This necessarily makes the algorithm belong to the domain of “derivative-free optimization” and “black-box optimization”.
In this example we minimize a simple objective to briefly demonstrate the usage of HEBO with Ray Tune via HEBOSearch. It’s useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume zoopt==0.4.1 library is installed. To learn more, please refer to the HEBO website.
Click below to see all the imports we need for this example. You can also launch directly into a Binder instance to run this notebook yourself. Just click on the rocket symbol at the top of the navigation.
import time
import ray
from ray import tune
from ray.air import session
from ray.tune.search.hebo import HEBOSearch
Let’s start by defining a simple evaluation function.
We artificially sleep for a bit (0.1 seconds) to simulate a long-running ML experiment.
This setup assumes that we’re running multiple steps of an experiment and try to tune two hyperparameters,
namely width and height, and activation.
def evaluate(step, width, height, activation):
time.sleep(0.1)
activation_boost = 10 if activation=="relu" else 1
return (0.1 + width * step / 100) ** (-1) + height * 0.1 + activation_boost
Next, our objective function takes a Tune config, evaluates the score of your experiment in a training loop,
and uses session.report to report the score back to Tune.
def objective(config):
for step in range(config["steps"]):
score = evaluate(step, config["width"], config["height"], config["activation"])
session.report({"iterations": step, "mean_loss": score})
While defining the search algorithm, we may choose to provide an initial set of hyperparameters that we believe are especially promising or informative, and
pass this information as a helpful starting point for the HyperOptSearch object.
We also set the maximum concurrent trials to 8.
previously_run_params = [
{"width": 10, "height": 0, "activation": "relu"},
{"width": 15, "height": -20, "activation": "tanh"},
]
known_rewards = [-189, -1144]
max_concurrent = 8
algo = HEBOSearch(
metric="mean_loss",
mode="min",
points_to_evaluate=previously_run_params,
evaluated_rewards=known_rewards,
random_state_seed=123,
max_concurrent=max_concurrent,
)
The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to 1000 samples.
(you can decrease this if it takes too long on your machine).
num_samples = 1000
Next we define a search space. The critical assumption is that the optimal hyperparamters live within this space. Yet, if the space is very large, then those hyperparameters may be difficult to find in a short amount of time.
search_config = {
"steps": 100,
"width": tune.uniform(0, 20),
"height": tune.uniform(-100, 100),
"activation": tune.choice(["relu", "tanh"])
}
Finally, we run the experiment to "min"imize the “mean_loss” of the objective by searching search_config via algo, num_samples times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute tuner.fit().
tuner = tune.Tuner(
objective,
tune_config=tune.TuneConfig(
metric="mean_loss",
mode="min",
search_alg=algo,
num_samples=num_samples,
),
param_space=search_config,
)
results = tuner.fit()
== Status ==
Current time: 2022-07-22 15:35:11 (running for 00:00:36.78)
Memory usage on this node: 10.2/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.3 GiB heap, 0.0/2.0 GiB objects
Current best trial: 72267d26 with mean_loss=-8.280721582416527 and parameters={'steps': 100, 'width': 16.267813332265522, 'height': -93.42430416543701, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-34-34
Number of trials: 10/10 (10 TERMINATED)
Current time: 2022-07-22 15:35:11 (running for 00:00:36.78)
Memory usage on this node: 10.2/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.3 GiB heap, 0.0/2.0 GiB objects
Current best trial: 72267d26 with mean_loss=-8.280721582416527 and parameters={'steps': 100, 'width': 16.267813332265522, 'height': -93.42430416543701, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-34-34
Number of trials: 10/10 (10 TERMINATED)
| Trial name | status | loc | activation | height | width | loss | iter | total time (s) | iterations | neg_mean_loss |
|---|---|---|---|---|---|---|---|---|---|---|
| objective_67ec1a0a | TERMINATED | 127.0.0.1:47498 | relu | -100 | 0 | 10 | 100 | 11.4386 | 99 | -10 |
| objective_69ac3226 | TERMINATED | 127.0.0.1:47512 | relu | 0 | 10 | 10.1 | 100 | 10.9018 | 99 | -10.1 |
| objective_69ada8d6 | TERMINATED | 127.0.0.1:47513 | relu | -50 | 15 | 5.06689 | 100 | 10.7768 | 99 | -5.06689 |
| objective_69af2530 | TERMINATED | 127.0.0.1:47514 | tanh | 50 | 5 | 6.19802 | 100 | 10.9312 | 99 | -6.19802 |
| objective_69b0a8a6 | TERMINATED | 127.0.0.1:47515 | tanh | -25 | 7.5 | -1.36711 | 100 | 10.7948 | 99 | 1.36711 |
| objective_69b2375c | TERMINATED | 127.0.0.1:47516 | relu | 75 | 17.5 | 17.5574 | 100 | 10.8966 | 99 | -17.5574 |
| objective_69b3bb9a | TERMINATED | 127.0.0.1:47517 | tanh | -75 | 12.5 | -6.41984 | 100 | 10.9022 | 99 | 6.41984 |
| objective_69b58f60 | TERMINATED | 127.0.0.1:47519 | relu | 25 | 2.5 | 12.8883 | 100 | 10.8995 | 99 | -12.8883 |
| objective_72267d26 | TERMINATED | 127.0.0.1:47563 | tanh | -93.4243 | 16.2678 | -8.28072 | 100 | 10.7101 | 99 | 8.28072 |
| objective_75ed3e0e | TERMINATED | 127.0.0.1:47568 | tanh | 28.8058 | 15.0428 | 3.94728 | 100 | 10.7472 | 99 | -3.94728 |
Result for objective_67ec1a0a:
date: 2022-07-22_15-34-37
done: false
experiment_id: b2cc3485f1024cbbbb5947a9acd341e9
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 10.0
neg_mean_loss: -10.0
node_ip: 127.0.0.1
pid: 47498
time_since_restore: 0.10423088073730469
time_this_iter_s: 0.10423088073730469
time_total_s: 0.10423088073730469
timestamp: 1658500477
timesteps_since_restore: 0
training_iteration: 1
trial_id: 67ec1a0a
warmup_time: 0.0028820037841796875
Result for objective_69ada8d6:
date: 2022-07-22_15-34-40
done: false
experiment_id: 9ca732d0f466455cbaa1da6f553a17ab
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 15.0
neg_mean_loss: -15.0
node_ip: 127.0.0.1
pid: 47513
time_since_restore: 0.10410189628601074
time_this_iter_s: 0.10410189628601074
time_total_s: 0.10410189628601074
timestamp: 1658500480
timesteps_since_restore: 0
training_iteration: 1
trial_id: 69ada8d6
warmup_time: 0.00498199462890625
Result for objective_69ac3226:
date: 2022-07-22_15-34-40
done: false
experiment_id: 5fa0de7eaf624b22bf76f0407a5dc3cd
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 20.0
neg_mean_loss: -20.0
node_ip: 127.0.0.1
pid: 47512
time_since_restore: 0.10359311103820801
time_this_iter_s: 0.10359311103820801
time_total_s: 0.10359311103820801
timestamp: 1658500480
timesteps_since_restore: 0
training_iteration: 1
trial_id: 69ac3226
warmup_time: 0.007561922073364258
Result for objective_69af2530:
date: 2022-07-22_15-34-40
done: false
experiment_id: 4c5fc14d64b04ec2b071fb751a9c6bde
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 16.0
neg_mean_loss: -16.0
node_ip: 127.0.0.1
pid: 47514
time_since_restore: 0.1039130687713623
time_this_iter_s: 0.1039130687713623
time_total_s: 0.1039130687713623
timestamp: 1658500480
timesteps_since_restore: 0
training_iteration: 1
trial_id: 69af2530
warmup_time: 0.002995014190673828
Result for objective_69b0a8a6:
date: 2022-07-22_15-34-40
done: false
experiment_id: 013499ab54ed4a4f92666a27945d673e
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 8.5
neg_mean_loss: -8.5
node_ip: 127.0.0.1
pid: 47515
time_since_restore: 0.10396409034729004
time_this_iter_s: 0.10396409034729004
time_total_s: 0.10396409034729004
timestamp: 1658500480
timesteps_since_restore: 0
training_iteration: 1
trial_id: 69b0a8a6
warmup_time: 0.00310516357421875
Result for objective_69b2375c:
date: 2022-07-22_15-34-40
done: false
experiment_id: 7e5aeb8fba3a42e6ae6471eedfc75fd2
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 27.5
neg_mean_loss: -27.5
node_ip: 127.0.0.1
pid: 47516
time_since_restore: 0.10422396659851074
time_this_iter_s: 0.10422396659851074
time_total_s: 0.10422396659851074
timestamp: 1658500480
timesteps_since_restore: 0
training_iteration: 1
trial_id: 69b2375c
warmup_time: 0.0030279159545898438
Result for objective_69b58f60:
date: 2022-07-22_15-34-40
done: false
experiment_id: 02a11d6943f04adaaf8a7c50cbcee0dd
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 22.5
neg_mean_loss: -22.5
node_ip: 127.0.0.1
pid: 47519
time_since_restore: 0.1043708324432373
time_this_iter_s: 0.1043708324432373
time_total_s: 0.1043708324432373
timestamp: 1658500480
timesteps_since_restore: 0
training_iteration: 1
trial_id: 69b58f60
warmup_time: 0.0027899742126464844
Result for objective_69b3bb9a:
date: 2022-07-22_15-34-40
done: false
experiment_id: a2d184f9ca934a768b78fbb438dbf28f
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 3.5
neg_mean_loss: -3.5
node_ip: 127.0.0.1
pid: 47517
time_since_restore: 0.10400700569152832
time_this_iter_s: 0.10400700569152832
time_total_s: 0.10400700569152832
timestamp: 1658500480
timesteps_since_restore: 0
training_iteration: 1
trial_id: 69b3bb9a
warmup_time: 0.002424955368041992
Result for objective_67ec1a0a:
date: 2022-07-22_15-34-42
done: false
experiment_id: b2cc3485f1024cbbbb5947a9acd341e9
hostname: Kais-MacBook-Pro.local
iterations: 41
iterations_since_restore: 42
mean_loss: 10.0
neg_mean_loss: -10.0
node_ip: 127.0.0.1
pid: 47498
time_since_restore: 5.111451864242554
time_this_iter_s: 0.10680818557739258
time_total_s: 5.111451864242554
timestamp: 1658500482
timesteps_since_restore: 0
training_iteration: 42
trial_id: 67ec1a0a
warmup_time: 0.0028820037841796875
Result for objective_69ac3226:
date: 2022-07-22_15-34-45
done: false
experiment_id: 5fa0de7eaf624b22bf76f0407a5dc3cd
hostname: Kais-MacBook-Pro.local
iterations: 46
iterations_since_restore: 47
mean_loss: 10.212765957446809
neg_mean_loss: -10.212765957446809
node_ip: 127.0.0.1
pid: 47512
time_since_restore: 5.153754234313965
time_this_iter_s: 0.10711407661437988
time_total_s: 5.153754234313965
timestamp: 1658500485
timesteps_since_restore: 0
training_iteration: 47
trial_id: 69ac3226
warmup_time: 0.007561922073364258
Result for objective_69af2530:
date: 2022-07-22_15-34-45
done: false
experiment_id: 4c5fc14d64b04ec2b071fb751a9c6bde
hostname: Kais-MacBook-Pro.local
iterations: 46
iterations_since_restore: 47
mean_loss: 6.416666666666667
neg_mean_loss: -6.416666666666667
node_ip: 127.0.0.1
pid: 47514
time_since_restore: 5.158767938613892
time_this_iter_s: 0.10651683807373047
time_total_s: 5.158767938613892
timestamp: 1658500485
timesteps_since_restore: 0
training_iteration: 47
trial_id: 69af2530
warmup_time: 0.002995014190673828
Result for objective_69b3bb9a:
date: 2022-07-22_15-34-45
done: false
experiment_id: a2d184f9ca934a768b78fbb438dbf28f
hostname: Kais-MacBook-Pro.local
iterations: 46
iterations_since_restore: 47
mean_loss: -6.329059829059829
neg_mean_loss: 6.329059829059829
node_ip: 127.0.0.1
pid: 47517
time_since_restore: 5.1304240226745605
time_this_iter_s: 0.10833311080932617
time_total_s: 5.1304240226745605
timestamp: 1658500485
timesteps_since_restore: 0
training_iteration: 47
trial_id: 69b3bb9a
warmup_time: 0.002424955368041992
Result for objective_69b58f60:
date: 2022-07-22_15-34-45
done: false
experiment_id: 02a11d6943f04adaaf8a7c50cbcee0dd
hostname: Kais-MacBook-Pro.local
iterations: 46
iterations_since_restore: 47
mean_loss: 13.3
neg_mean_loss: -13.3
node_ip: 127.0.0.1
pid: 47519
time_since_restore: 5.138491868972778
time_this_iter_s: 0.10869002342224121
time_total_s: 5.138491868972778
timestamp: 1658500485
timesteps_since_restore: 0
training_iteration: 47
trial_id: 69b58f60
warmup_time: 0.0027899742126464844
Result for objective_69b2375c:
date: 2022-07-22_15-34-45
done: false
experiment_id: 7e5aeb8fba3a42e6ae6471eedfc75fd2
hostname: Kais-MacBook-Pro.local
iterations: 46
iterations_since_restore: 47
mean_loss: 17.62269938650307
neg_mean_loss: -17.62269938650307
node_ip: 127.0.0.1
pid: 47516
time_since_restore: 5.13613486289978
time_this_iter_s: 0.10693097114562988
time_total_s: 5.13613486289978
timestamp: 1658500485
timesteps_since_restore: 0
training_iteration: 47
trial_id: 69b2375c
warmup_time: 0.0030279159545898438
Result for objective_69ada8d6:
date: 2022-07-22_15-34-45
done: false
experiment_id: 9ca732d0f466455cbaa1da6f553a17ab
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 5.13986013986014
neg_mean_loss: -5.13986013986014
node_ip: 127.0.0.1
pid: 47513
time_since_restore: 5.1575539112091064
time_this_iter_s: 0.10637593269348145
time_total_s: 5.1575539112091064
timestamp: 1658500485
timesteps_since_restore: 0
training_iteration: 48
trial_id: 69ada8d6
warmup_time: 0.00498199462890625
Result for objective_69b0a8a6:
date: 2022-07-22_15-34-45
done: false
experiment_id: 013499ab54ed4a4f92666a27945d673e
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -1.2241379310344827
neg_mean_loss: 1.2241379310344827
node_ip: 127.0.0.1
pid: 47515
time_since_restore: 5.211113929748535
time_this_iter_s: 0.10501360893249512
time_total_s: 5.211113929748535
timestamp: 1658500485
timesteps_since_restore: 0
training_iteration: 48
trial_id: 69b0a8a6
warmup_time: 0.00310516357421875
Result for objective_67ec1a0a:
date: 2022-07-22_15-34-47
done: false
experiment_id: b2cc3485f1024cbbbb5947a9acd341e9
hostname: Kais-MacBook-Pro.local
iterations: 87
iterations_since_restore: 88
mean_loss: 10.0
neg_mean_loss: -10.0
node_ip: 127.0.0.1
pid: 47498
time_since_restore: 10.140707731246948
time_this_iter_s: 0.10805296897888184
time_total_s: 10.140707731246948
timestamp: 1658500487
timesteps_since_restore: 0
training_iteration: 88
trial_id: 67ec1a0a
warmup_time: 0.0028820037841796875
Result for objective_67ec1a0a:
date: 2022-07-22_15-34-48
done: true
experiment_id: b2cc3485f1024cbbbb5947a9acd341e9
experiment_tag: 1_activation=relu,height=-100.0000,steps=100,width=0.0000
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 10.0
neg_mean_loss: -10.0
node_ip: 127.0.0.1
pid: 47498
time_since_restore: 11.438636064529419
time_this_iter_s: 0.1079721450805664
time_total_s: 11.438636064529419
timestamp: 1658500488
timesteps_since_restore: 0
training_iteration: 100
trial_id: 67ec1a0a
warmup_time: 0.0028820037841796875
Result for objective_69b3bb9a:
date: 2022-07-22_15-34-50
done: false
experiment_id: a2d184f9ca934a768b78fbb438dbf28f
hostname: Kais-MacBook-Pro.local
iterations: 92
iterations_since_restore: 93
mean_loss: -6.413793103448276
neg_mean_loss: 6.413793103448276
node_ip: 127.0.0.1
pid: 47517
time_since_restore: 10.136809825897217
time_this_iter_s: 0.10945367813110352
time_total_s: 10.136809825897217
timestamp: 1658500490
timesteps_since_restore: 0
training_iteration: 93
trial_id: 69b3bb9a
warmup_time: 0.002424955368041992
Result for objective_69ac3226:
date: 2022-07-22_15-34-50
done: false
experiment_id: 5fa0de7eaf624b22bf76f0407a5dc3cd
hostname: Kais-MacBook-Pro.local
iterations: 93
iterations_since_restore: 94
mean_loss: 10.106382978723405
neg_mean_loss: -10.106382978723405
node_ip: 127.0.0.1
pid: 47512
time_since_restore: 10.222928285598755
time_this_iter_s: 0.10617327690124512
time_total_s: 10.222928285598755
timestamp: 1658500490
timesteps_since_restore: 0
training_iteration: 94
trial_id: 69ac3226
warmup_time: 0.007561922073364258
Result for objective_69ada8d6:
date: 2022-07-22_15-34-50
done: false
experiment_id: 9ca732d0f466455cbaa1da6f553a17ab
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 5.070422535211268
neg_mean_loss: -5.070422535211268
node_ip: 127.0.0.1
pid: 47513
time_since_restore: 10.23993706703186
time_this_iter_s: 0.10660290718078613
time_total_s: 10.23993706703186
timestamp: 1658500490
timesteps_since_restore: 0
training_iteration: 95
trial_id: 69ada8d6
warmup_time: 0.00498199462890625
Result for objective_69b2375c:
date: 2022-07-22_15-34-50
done: false
experiment_id: 7e5aeb8fba3a42e6ae6471eedfc75fd2
hostname: Kais-MacBook-Pro.local
iterations: 93
iterations_since_restore: 94
mean_loss: 17.561068702290076
neg_mean_loss: -17.561068702290076
node_ip: 127.0.0.1
pid: 47516
time_since_restore: 10.21057415008545
time_this_iter_s: 0.10399723052978516
time_total_s: 10.21057415008545
timestamp: 1658500490
timesteps_since_restore: 0
training_iteration: 94
trial_id: 69b2375c
warmup_time: 0.0030279159545898438
Result for objective_69b58f60:
date: 2022-07-22_15-34-50
done: false
experiment_id: 02a11d6943f04adaaf8a7c50cbcee0dd
hostname: Kais-MacBook-Pro.local
iterations: 93
iterations_since_restore: 94
mean_loss: 12.912371134020619
neg_mean_loss: -12.912371134020619
node_ip: 127.0.0.1
pid: 47519
time_since_restore: 10.214950799942017
time_this_iter_s: 0.10687804222106934
time_total_s: 10.214950799942017
timestamp: 1658500490
timesteps_since_restore: 0
training_iteration: 94
trial_id: 69b58f60
warmup_time: 0.0027899742126464844
Result for objective_69af2530:
date: 2022-07-22_15-34-50
done: false
experiment_id: 4c5fc14d64b04ec2b071fb751a9c6bde
hostname: Kais-MacBook-Pro.local
iterations: 93
iterations_since_restore: 94
mean_loss: 6.2105263157894735
neg_mean_loss: -6.2105263157894735
node_ip: 127.0.0.1
pid: 47514
time_since_restore: 10.267423152923584
time_this_iter_s: 0.10761213302612305
time_total_s: 10.267423152923584
timestamp: 1658500490
timesteps_since_restore: 0
training_iteration: 94
trial_id: 69af2530
warmup_time: 0.002995014190673828
Result for objective_69b0a8a6:
date: 2022-07-22_15-34-50
done: false
experiment_id: 013499ab54ed4a4f92666a27945d673e
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: -1.36013986013986
neg_mean_loss: 1.36013986013986
node_ip: 127.0.0.1
pid: 47515
time_since_restore: 10.256262063980103
time_this_iter_s: 0.10606575012207031
time_total_s: 10.256262063980103
timestamp: 1658500490
timesteps_since_restore: 0
training_iteration: 95
trial_id: 69b0a8a6
warmup_time: 0.00310516357421875
Result for objective_69ada8d6:
date: 2022-07-22_15-34-51
done: true
experiment_id: 9ca732d0f466455cbaa1da6f553a17ab
experiment_tag: 3_activation=relu,height=-50.0000,steps=100,width=15.0000
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 5.066889632107023
neg_mean_loss: -5.066889632107023
node_ip: 127.0.0.1
pid: 47513
time_since_restore: 10.77684497833252
time_this_iter_s: 0.10641121864318848
time_total_s: 10.77684497833252
timestamp: 1658500491
timesteps_since_restore: 0
training_iteration: 100
trial_id: 69ada8d6
warmup_time: 0.00498199462890625
Result for objective_69b0a8a6:
date: 2022-07-22_15-34-51
done: true
experiment_id: 013499ab54ed4a4f92666a27945d673e
experiment_tag: 5_activation=tanh,height=-25.0000,steps=100,width=7.5000
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -1.367109634551495
neg_mean_loss: 1.367109634551495
node_ip: 127.0.0.1
pid: 47515
time_since_restore: 10.794761180877686
time_this_iter_s: 0.10670304298400879
time_total_s: 10.794761180877686
timestamp: 1658500491
timesteps_since_restore: 0
training_iteration: 100
trial_id: 69b0a8a6
warmup_time: 0.00310516357421875
Result for objective_69ac3226:
date: 2022-07-22_15-34-51
done: true
experiment_id: 5fa0de7eaf624b22bf76f0407a5dc3cd
experiment_tag: 2_activation=relu,height=0.0000,steps=100,width=10.0000
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 10.1
neg_mean_loss: -10.1
node_ip: 127.0.0.1
pid: 47512
time_since_restore: 10.901827096939087
time_this_iter_s: 0.13848495483398438
time_total_s: 10.901827096939087
timestamp: 1658500491
timesteps_since_restore: 0
training_iteration: 100
trial_id: 69ac3226
warmup_time: 0.007561922073364258
Result for objective_69b2375c:
date: 2022-07-22_15-34-51
done: true
experiment_id: 7e5aeb8fba3a42e6ae6471eedfc75fd2
experiment_tag: 6_activation=relu,height=75.0000,steps=100,width=17.5000
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 17.55738880918221
neg_mean_loss: -17.55738880918221
node_ip: 127.0.0.1
pid: 47516
time_since_restore: 10.896636962890625
time_this_iter_s: 0.14625000953674316
time_total_s: 10.896636962890625
timestamp: 1658500491
timesteps_since_restore: 0
training_iteration: 100
trial_id: 69b2375c
warmup_time: 0.0030279159545898438
Result for objective_69b3bb9a:
date: 2022-07-22_15-34-51
done: true
experiment_id: a2d184f9ca934a768b78fbb438dbf28f
experiment_tag: 7_activation=tanh,height=-75.0000,steps=100,width=12.5000
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -6.419839679358717
neg_mean_loss: 6.419839679358717
node_ip: 127.0.0.1
pid: 47517
time_since_restore: 10.902234077453613
time_this_iter_s: 0.12049722671508789
time_total_s: 10.902234077453613
timestamp: 1658500491
timesteps_since_restore: 0
training_iteration: 100
trial_id: 69b3bb9a
warmup_time: 0.002424955368041992
Result for objective_69b58f60:
date: 2022-07-22_15-34-51
done: true
experiment_id: 02a11d6943f04adaaf8a7c50cbcee0dd
experiment_tag: 8_activation=relu,height=25.0000,steps=100,width=2.5000
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 12.888349514563107
neg_mean_loss: -12.888349514563107
node_ip: 127.0.0.1
pid: 47519
time_since_restore: 10.899547815322876
time_this_iter_s: 0.1467878818511963
time_total_s: 10.899547815322876
timestamp: 1658500491
timesteps_since_restore: 0
training_iteration: 100
trial_id: 69b58f60
warmup_time: 0.0027899742126464844
Result for objective_69af2530:
date: 2022-07-22_15-34-51
done: true
experiment_id: 4c5fc14d64b04ec2b071fb751a9c6bde
experiment_tag: 4_activation=tanh,height=50.0000,steps=100,width=5.0000
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 6.198019801980198
neg_mean_loss: -6.198019801980198
node_ip: 127.0.0.1
pid: 47514
time_since_restore: 10.931232929229736
time_this_iter_s: 0.12574982643127441
time_total_s: 10.931232929229736
timestamp: 1658500491
timesteps_since_restore: 0
training_iteration: 100
trial_id: 69af2530
warmup_time: 0.002995014190673828
Result for objective_72267d26:
date: 2022-07-22_15-34-58
done: false
experiment_id: 05d0fd74bba34c209c3fb167e5aabb6e
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 1.657569583456299
neg_mean_loss: -1.657569583456299
node_ip: 127.0.0.1
pid: 47563
time_since_restore: 0.10434603691101074
time_this_iter_s: 0.10434603691101074
time_total_s: 0.10434603691101074
timestamp: 1658500498
timesteps_since_restore: 0
training_iteration: 1
trial_id: 72267d26
warmup_time: 0.0029430389404296875
Result for objective_75ed3e0e:
date: 2022-07-22_15-35-00
done: false
experiment_id: 57d23b5e98454d9eb68f3dee5b5f2642
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 13.88058437447561
neg_mean_loss: -13.88058437447561
node_ip: 127.0.0.1
pid: 47568
time_since_restore: 0.10134601593017578
time_this_iter_s: 0.10134601593017578
time_total_s: 0.10134601593017578
timestamp: 1658500500
timesteps_since_restore: 0
training_iteration: 1
trial_id: 75ed3e0e
warmup_time: 0.002665996551513672
Result for objective_72267d26:
date: 2022-07-22_15-35-03
done: false
experiment_id: 05d0fd74bba34c209c3fb167e5aabb6e
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -8.213329397880102
neg_mean_loss: 8.213329397880102
node_ip: 127.0.0.1
pid: 47563
time_since_restore: 5.1267828941345215
time_this_iter_s: 0.10927891731262207
time_total_s: 5.1267828941345215
timestamp: 1658500503
timesteps_since_restore: 0
training_iteration: 48
trial_id: 72267d26
warmup_time: 0.0029430389404296875
Result for objective_75ed3e0e:
date: 2022-07-22_15-35-05
done: false
experiment_id: 57d23b5e98454d9eb68f3dee5b5f2642
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 4.020052046405574
neg_mean_loss: -4.020052046405574
node_ip: 127.0.0.1
pid: 47568
time_since_restore: 5.158367156982422
time_this_iter_s: 0.10702204704284668
time_total_s: 5.158367156982422
timestamp: 1658500505
timesteps_since_restore: 0
training_iteration: 48
trial_id: 75ed3e0e
warmup_time: 0.002665996551513672
Result for objective_72267d26:
date: 2022-07-22_15-35-08
done: false
experiment_id: 05d0fd74bba34c209c3fb167e5aabb6e
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: -8.277460523241512
neg_mean_loss: 8.277460523241512
node_ip: 127.0.0.1
pid: 47563
time_since_restore: 10.168545961380005
time_this_iter_s: 0.10625672340393066
time_total_s: 10.168545961380005
timestamp: 1658500508
timesteps_since_restore: 0
training_iteration: 95
trial_id: 72267d26
warmup_time: 0.0029430389404296875
Result for objective_72267d26:
date: 2022-07-22_15-35-08
done: true
experiment_id: 05d0fd74bba34c209c3fb167e5aabb6e
experiment_tag: 9_activation=tanh,height=-93.4243,steps=100,width=16.2678
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -8.280721582416527
neg_mean_loss: 8.280721582416527
node_ip: 127.0.0.1
pid: 47563
time_since_restore: 10.71009612083435
time_this_iter_s: 0.10849308967590332
time_total_s: 10.71009612083435
timestamp: 1658500508
timesteps_since_restore: 0
training_iteration: 100
trial_id: 72267d26
warmup_time: 0.0029430389404296875
Result for objective_75ed3e0e:
date: 2022-07-22_15-35-10
done: false
experiment_id: 57d23b5e98454d9eb68f3dee5b5f2642
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 3.950807906063858
neg_mean_loss: -3.950807906063858
node_ip: 127.0.0.1
pid: 47568
time_since_restore: 10.20597505569458
time_this_iter_s: 0.10656380653381348
time_total_s: 10.20597505569458
timestamp: 1658500510
timesteps_since_restore: 0
training_iteration: 95
trial_id: 75ed3e0e
warmup_time: 0.002665996551513672
Result for objective_75ed3e0e:
date: 2022-07-22_15-35-11
done: true
experiment_id: 57d23b5e98454d9eb68f3dee5b5f2642
experiment_tag: 10_activation=tanh,height=28.8058,steps=100,width=15.0428
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 3.947284919356474
neg_mean_loss: -3.947284919356474
node_ip: 127.0.0.1
pid: 47568
time_since_restore: 10.74724817276001
time_this_iter_s: 0.10794186592102051
time_total_s: 10.74724817276001
timestamp: 1658500511
timesteps_since_restore: 0
training_iteration: 100
trial_id: 75ed3e0e
warmup_time: 0.002665996551513672
Here are the hyperparamters found to minimize the mean loss of the defined objective.
print("Best hyperparameters found were: ", results.get_best_result().config)
Best hyperparameters found were: {'steps': 100, 'width': 16.267813332265522, 'height': -93.42430416543701, 'activation': 'tanh'}