Running Tune experiments with HyperOpt#

In this tutorial we introduce HyperOpt, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with HyperOpt and, as a result, allow you to seamlessly scale up a Hyperopt optimization process - without sacrificing performance.

HyperOpt provides gradient/derivative-free optimization able to handle noise over the objective landscape, including evolutionary, bandit, and Bayesian optimization algorithms. HyperOpt internally supports search spaces which are continuous, discrete or a mixture of thereof. It also provides a library of functions on which to test the optimization algorithms and compare with other benchmarks.

In this example we minimize a simple objective to briefly demonstrate the usage of HyperOpt with Ray Tune via HyperOptSearch. It’s useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume hyperopt==0.2.5 library is installed. To learn more, please refer to HyperOpt website.

We include a important example on conditional search spaces (stringing together relationships among hyperparameters).

Background information:

HyperOpt website

Necessary requirements:

pip install ray[tune]
pip install hyperopt==0.2.5

Click below to see all the imports we need for this example. You can also launch directly into a Binder instance to run this notebook yourself. Just click on the rocket symbol at the top of the navigation.

Let’s start by defining a simple evaluation function. We artificially sleep for a bit (0.1 seconds) to simulate a long-running ML experiment. This setup assumes that we’re running multiple steps of an experiment and try to tune two hyperparameters, namely width and height.

def evaluate(step, width, height):
    time.sleep(0.1)
    return (0.1 + width * step / 100) ** (-1) + height * 0.1

Next, our objective function takes a Tune config, evaluates the score of your experiment in a training loop, and uses train.report to report the score back to Tune.

def objective(config):
    for step in range(config["steps"]):
        score = evaluate(step, config["width"], config["height"])
        train.report({"iterations": step, "mean_loss": score})

While defining the search algorithm, we may choose to provide an initial set of hyperparameters that we believe are especially promising or informative, and pass this information as a helpful starting point for the HyperOptSearch object.

We also set the maximum concurrent trials to 4 with a ConcurrencyLimiter.

initial_params = [
    {"width": 1, "height": 2, "activation": "relu"},
    {"width": 4, "height": 2, "activation": "tanh"},
]
algo = HyperOptSearch(points_to_evaluate=initial_params)
algo = ConcurrencyLimiter(algo, max_concurrent=4)

The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to 1000 samples. (you can decrease this if it takes too long on your machine).

num_samples = 1000

Next we define a search space. The critical assumption is that the optimal hyperparamters live within this space. Yet, if the space is very large, then those hyperparameters may be difficult to find in a short amount of time.

search_config = {
    "steps": 100,
    "width": tune.uniform(0, 20),
    "height": tune.uniform(-100, 100),
    "activation": tune.choice(["relu", "tanh"])
}

Finally, we run the experiment to "min"imize the “mean_loss” of the objective by searching search_config via algo, num_samples times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute tuner.fit().

tuner = tune.Tuner(
    objective,
    tune_config=tune.TuneConfig(
        metric="mean_loss",
        mode="min",
        search_alg=algo,
        num_samples=num_samples,
    ),
    param_space=search_config,
)
results = tuner.fit()

== Status ==
Current time: 2022-07-22 15:31:49 (running for 00:00:43.71)
Memory usage on this node: 10.4/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.29 GiB heap, 0.0/2.0 GiB objects
Current best trial: f59fe9d6 with mean_loss=-2.5719085451008423 and parameters={'steps': 100, 'width': 5.584304853357766, 'height': -27.49576980043919, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-31-04
Number of trials: 10/10 (10 TERMINATED)

Trial name	status	loc	activation	height	steps	width	loss	iter	total time (s)	iterations	neg_mean_loss
objective_eb74f122	TERMINATED	127.0.0.1:47156	relu	2	100	1	1.11743	100	10.9008	99	-1.11743
objective_ed2010ec	TERMINATED	127.0.0.1:47161	tanh	2	100	4	0.446305	100	11.5098	99	-0.446305
objective_ed217cf2	TERMINATED	127.0.0.1:47162	tanh	-20.6075	100	13.061	-1.98401	100	11.5623	99	1.98401
objective_ed2322be	TERMINATED	127.0.0.1:47163	tanh	19.9564	100	10.6836	2.0893	100	11.6053	99	-2.0893
objective_f3a0bef8	TERMINATED	127.0.0.1:47180	tanh	-7.43915	100	2.23969	-0.312378	100	10.7378	99	0.312378
objective_f59fe9d6	TERMINATED	127.0.0.1:47185	tanh	-27.4958	100	5.5843	-2.57191	100	10.7145	99	2.57191
objective_f5aedf9a	TERMINATED	127.0.0.1:47188	tanh	48.706	100	19.7352	4.92153	100	10.7341	99	-4.92153
objective_f5b1ec08	TERMINATED	127.0.0.1:47189	tanh	4.14098	100	16.2739	0.475784	100	10.7034	99	-0.475784
objective_fb926d32	TERMINATED	127.0.0.1:47205	tanh	44.778	100	10.0724	4.57708	100	13.1109	99	-4.57708
objective_fd91e28e	TERMINATED	127.0.0.1:47214	relu	-2.9623	100	11.8215	-0.211508	100	10.7934	99	0.211508

Result for objective_eb74f122:
  date: 2022-07-22_15-31-08
  done: false
  experiment_id: 0ed1b6ba6d99477dba0632102d1bc531
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 10.2
  neg_mean_loss: -10.2
  node_ip: 127.0.0.1
  pid: 47156
  time_since_restore: 0.10458922386169434
  time_this_iter_s: 0.10458922386169434
  time_total_s: 0.10458922386169434
  timestamp: 1658500268
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: eb74f122
  warmup_time: 0.002730131149291992
  
Result for objective_ed2010ec:
  date: 2022-07-22_15-31-11
  done: false
  experiment_id: 3acf6a8ccf8442a7adcf98cc92362216
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 10.2
  neg_mean_loss: -10.2
  node_ip: 127.0.0.1
  pid: 47161
  time_since_restore: 0.1025240421295166
  time_this_iter_s: 0.1025240421295166
  time_total_s: 0.1025240421295166
  timestamp: 1658500271
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: ed2010ec
  warmup_time: 0.0034351348876953125
  
Result for objective_ed2322be:
  date: 2022-07-22_15-31-11
  done: false
  experiment_id: e95895ab00b54841933d324c3de8f58e
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 11.99564236159307
  neg_mean_loss: -11.99564236159307
  node_ip: 127.0.0.1
  pid: 47163
  time_since_restore: 0.10490107536315918
  time_this_iter_s: 0.10490107536315918
  time_total_s: 0.10490107536315918
  timestamp: 1658500271
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: ed2322be
  warmup_time: 0.0032410621643066406
  
Result for objective_ed217cf2:
  date: 2022-07-22_15-31-11
  done: false
  experiment_id: 52186ede891e429aac44318036e7a7bb
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 7.939249801790311
  neg_mean_loss: -7.939249801790311
  node_ip: 127.0.0.1
  pid: 47162
  time_since_restore: 0.10419487953186035
  time_this_iter_s: 0.10419487953186035
  time_total_s: 0.10419487953186035
  timestamp: 1658500271
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: ed217cf2
  warmup_time: 0.0031850337982177734
  
Result for objective_eb74f122:
  date: 2022-07-22_15-31-13
  done: false
  experiment_id: 0ed1b6ba6d99477dba0632102d1bc531
  hostname: Kais-MacBook-Pro.local
  iterations: 46
  iterations_since_restore: 47
  mean_loss: 1.9857142857142855
  neg_mean_loss: -1.9857142857142855
  node_ip: 127.0.0.1
  pid: 47156
  time_since_restore: 5.155240058898926
  time_this_iter_s: 0.11003398895263672
  time_total_s: 5.155240058898926
  timestamp: 1658500273
  timesteps_since_restore: 0
  training_iteration: 47
  trial_id: eb74f122
  warmup_time: 0.002730131149291992
  
Result for objective_ed2010ec:
  date: 2022-07-22_15-31-16
  done: false
  experiment_id: 3acf6a8ccf8442a7adcf98cc92362216
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 0.7050505050505051
  neg_mean_loss: -0.7050505050505051
  node_ip: 127.0.0.1
  pid: 47161
  time_since_restore: 5.1636271476745605
  time_this_iter_s: 0.1071469783782959
  time_total_s: 5.1636271476745605
  timestamp: 1658500276
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: ed2010ec
  warmup_time: 0.0034351348876953125
  
Result for objective_ed2322be:
  date: 2022-07-22_15-31-16
  done: false
  experiment_id: e95895ab00b54841933d324c3de8f58e
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 2.1909053484385796
  neg_mean_loss: -2.1909053484385796
  node_ip: 127.0.0.1
  pid: 47163
  time_since_restore: 5.168597936630249
  time_this_iter_s: 0.10674691200256348
  time_total_s: 5.168597936630249
  timestamp: 1658500276
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: ed2322be
  warmup_time: 0.0032410621643066406
  
Result for objective_ed217cf2:
  date: 2022-07-22_15-31-16
  done: false
  experiment_id: 52186ede891e429aac44318036e7a7bb
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: -1.9004596703789314
  neg_mean_loss: 1.9004596703789314
  node_ip: 127.0.0.1
  pid: 47162
  time_since_restore: 5.14047384262085
  time_this_iter_s: 0.10724306106567383
  time_total_s: 5.14047384262085
  timestamp: 1658500276
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: ed217cf2
  warmup_time: 0.0031850337982177734
  
Result for objective_eb74f122:
  date: 2022-07-22_15-31-18
  done: false
  experiment_id: 0ed1b6ba6d99477dba0632102d1bc531
  hostname: Kais-MacBook-Pro.local
  iterations: 93
  iterations_since_restore: 94
  mean_loss: 1.170873786407767
  neg_mean_loss: -1.170873786407767
  node_ip: 127.0.0.1
  pid: 47156
  time_since_restore: 10.253305196762085
  time_this_iter_s: 0.10791301727294922
  time_total_s: 10.253305196762085
  timestamp: 1658500278
  timesteps_since_restore: 0
  training_iteration: 94
  trial_id: eb74f122
  warmup_time: 0.002730131149291992
  
Result for objective_eb74f122:
  date: 2022-07-22_15-31-19
  done: true
  experiment_id: 0ed1b6ba6d99477dba0632102d1bc531
  experiment_tag: 1_activation=relu,height=2.0000,steps=100,width=1.0000
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 1.1174311926605505
  neg_mean_loss: -1.1174311926605505
  node_ip: 127.0.0.1
  pid: 47156
  time_since_restore: 10.900829076766968
  time_this_iter_s: 0.10532379150390625
  time_total_s: 10.900829076766968
  timestamp: 1658500279
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: eb74f122
  warmup_time: 0.002730131149291992
  
Result for objective_ed217cf2:
  date: 2022-07-22_15-31-21
  done: false
  experiment_id: 52186ede891e429aac44318036e7a7bb
  hostname: Kais-MacBook-Pro.local
  iterations: 91
  iterations_since_restore: 92
  mean_loss: -1.9773161428398818
  neg_mean_loss: 1.9773161428398818
  node_ip: 127.0.0.1
  pid: 47162
  time_since_restore: 9.86834192276001
  time_this_iter_s: 0.10606908798217773
  time_total_s: 9.86834192276001
  timestamp: 1658500281
  timesteps_since_restore: 0
  training_iteration: 92
  trial_id: ed217cf2
  warmup_time: 0.0031850337982177734
  
Result for objective_ed2322be:
  date: 2022-07-22_15-31-21
  done: false
  experiment_id: e95895ab00b54841933d324c3de8f58e
  hostname: Kais-MacBook-Pro.local
  iterations: 91
  iterations_since_restore: 92
  mean_loss: 2.0974537058269864
  neg_mean_loss: -2.0974537058269864
  node_ip: 127.0.0.1
  pid: 47163
  time_since_restore: 9.877221822738647
  time_this_iter_s: 0.10554194450378418
  time_total_s: 9.877221822738647
  timestamp: 1658500281
  timesteps_since_restore: 0
  training_iteration: 92
  trial_id: ed2322be
  warmup_time: 0.0032410621643066406
  
Result for objective_ed2010ec:
  date: 2022-07-22_15-31-21
  done: false
  experiment_id: 3acf6a8ccf8442a7adcf98cc92362216
  hostname: Kais-MacBook-Pro.local
  iterations: 92
  iterations_since_restore: 93
  mean_loss: 0.46455026455026455
  neg_mean_loss: -0.46455026455026455
  node_ip: 127.0.0.1
  pid: 47161
  time_since_restore: 10.010878086090088
  time_this_iter_s: 0.12952017784118652
  time_total_s: 10.010878086090088
  timestamp: 1658500281
  timesteps_since_restore: 0
  training_iteration: 93
  trial_id: ed2010ec
  warmup_time: 0.0034351348876953125
  
Result for objective_f3a0bef8:
  date: 2022-07-22_15-31-22
  done: false
  experiment_id: 7bbba7b85dbc4029bc8b22b7866d091f
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 9.256084922802092
  neg_mean_loss: -9.256084922802092
  node_ip: 127.0.0.1
  pid: 47180
  time_since_restore: 0.10452008247375488
  time_this_iter_s: 0.10452008247375488
  time_total_s: 0.10452008247375488
  timestamp: 1658500282
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: f3a0bef8
  warmup_time: 0.003498077392578125
  
Result for objective_ed2010ec:
  date: 2022-07-22_15-31-22
  done: true
  experiment_id: 3acf6a8ccf8442a7adcf98cc92362216
  experiment_tag: 2_activation=tanh,height=2.0000,steps=100,width=4.0000
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 0.44630541871921187
  neg_mean_loss: -0.44630541871921187
  node_ip: 127.0.0.1
  pid: 47161
  time_since_restore: 11.509758949279785
  time_this_iter_s: 0.10628509521484375
  time_total_s: 11.509758949279785
  timestamp: 1658500282
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: ed2010ec
  warmup_time: 0.0034351348876953125
  
Result for objective_ed217cf2:
  date: 2022-07-22_15-31-22
  done: true
  experiment_id: 52186ede891e429aac44318036e7a7bb
  experiment_tag: 3_activation=tanh,height=-20.6075,steps=100,width=13.0610
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -1.9840065470391304
  neg_mean_loss: 1.9840065470391304
  node_ip: 127.0.0.1
  pid: 47162
  time_since_restore: 11.562283754348755
  time_this_iter_s: 0.10557794570922852
  time_total_s: 11.562283754348755
  timestamp: 1658500282
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: ed217cf2
  warmup_time: 0.0031850337982177734
  
Result for objective_ed2322be:
  date: 2022-07-22_15-31-22
  done: true
  experiment_id: e95895ab00b54841933d324c3de8f58e
  experiment_tag: 4_activation=tanh,height=19.9564,steps=100,width=10.6836
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 2.0893035832616653
  neg_mean_loss: -2.0893035832616653
  node_ip: 127.0.0.1
  pid: 47163
  time_since_restore: 11.605261087417603
  time_this_iter_s: 0.10673737525939941
  time_total_s: 11.605261087417603
  timestamp: 1658500282
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: ed2322be
  warmup_time: 0.0032410621643066406
  

Result for objective_f59fe9d6:
  date: 2022-07-22_15-31-25
  done: false
  experiment_id: efbda212c15c4bd38cfc5f39dfae8b73
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 7.250423019956081
  neg_mean_loss: -7.250423019956081
  node_ip: 127.0.0.1
  pid: 47185
  time_since_restore: 0.10512709617614746
  time_this_iter_s: 0.10512709617614746
  time_total_s: 0.10512709617614746
  timestamp: 1658500285
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: f59fe9d6
  warmup_time: 0.003314971923828125
  
Result for objective_f5b1ec08:
  date: 2022-07-22_15-31-25
  done: false
  experiment_id: dadb542868ba4b10adf1ef161c75ea17
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 10.41409792482367
  neg_mean_loss: -10.41409792482367
  node_ip: 127.0.0.1
  pid: 47189
  time_since_restore: 0.10109281539916992
  time_this_iter_s: 0.10109281539916992
  time_total_s: 0.10109281539916992
  timestamp: 1658500285
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: f5b1ec08
  warmup_time: 0.0031630992889404297
  
Result for objective_f5aedf9a:
  date: 2022-07-22_15-31-25
  done: false
  experiment_id: 9f7847b7440648cb86e73bf1be0ccc05
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 14.870604096990785
  neg_mean_loss: -14.870604096990785
  node_ip: 127.0.0.1
  pid: 47188
  time_since_restore: 0.10486412048339844
  time_this_iter_s: 0.10486412048339844
  time_total_s: 0.10486412048339844
  timestamp: 1658500285
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: f5aedf9a
  warmup_time: 0.002830028533935547
  
Result for objective_f3a0bef8:
  date: 2022-07-22_15-31-27
  done: false
  experiment_id: 7bbba7b85dbc4029bc8b22b7866d091f
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 0.12364690126601663
  neg_mean_loss: -0.12364690126601663
  node_ip: 127.0.0.1
  pid: 47180
  time_since_restore: 5.149861097335815
  time_this_iter_s: 0.1075279712677002
  time_total_s: 5.149861097335815
  timestamp: 1658500287
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: f3a0bef8
  warmup_time: 0.003498077392578125
  
Result for objective_f59fe9d6:
  date: 2022-07-22_15-31-30
  done: false
  experiment_id: efbda212c15c4bd38cfc5f39dfae8b73
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: -2.3825537636804834
  neg_mean_loss: 2.3825537636804834
  node_ip: 127.0.0.1
  pid: 47185
  time_since_restore: 5.170727014541626
  time_this_iter_s: 0.10740494728088379
  time_total_s: 5.170727014541626
  timestamp: 1658500290
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: f59fe9d6
  warmup_time: 0.003314971923828125
  
Result for objective_f5b1ec08:
  date: 2022-07-22_15-31-30
  done: false
  experiment_id: dadb542868ba4b10adf1ef161c75ea17
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 0.5431509247331814
  neg_mean_loss: -0.5431509247331814
  node_ip: 127.0.0.1
  pid: 47189
  time_since_restore: 5.153015613555908
  time_this_iter_s: 0.10737276077270508
  time_total_s: 5.153015613555908
  timestamp: 1658500290
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: f5b1ec08
  warmup_time: 0.0031630992889404297
  
Result for objective_f5aedf9a:
  date: 2022-07-22_15-31-30
  done: false
  experiment_id: 9f7847b7440648cb86e73bf1be0ccc05
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 4.977264708542528
  neg_mean_loss: -4.977264708542528
  node_ip: 127.0.0.1
  pid: 47188
  time_since_restore: 5.183044195175171
  time_this_iter_s: 0.10844588279724121
  time_total_s: 5.183044195175171
  timestamp: 1658500290
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: f5aedf9a
  warmup_time: 0.002830028533935547
  
Result for objective_f3a0bef8:
  date: 2022-07-22_15-31-32
  done: false
  experiment_id: 7bbba7b85dbc4029bc8b22b7866d091f
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: -0.29046425326874115
  neg_mean_loss: 0.29046425326874115
  node_ip: 127.0.0.1
  pid: 47180
  time_since_restore: 10.200733184814453
  time_this_iter_s: 0.10860228538513184
  time_total_s: 10.200733184814453
  timestamp: 1658500292
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: f3a0bef8
  warmup_time: 0.003498077392578125
  
Result for objective_f3a0bef8:
  date: 2022-07-22_15-31-32
  done: true
  experiment_id: 7bbba7b85dbc4029bc8b22b7866d091f
  experiment_tag: 5_activation=tanh,height=-7.4392,steps=100,width=2.2397
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -0.3123775218508783
  neg_mean_loss: 0.3123775218508783
  node_ip: 127.0.0.1
  pid: 47180
  time_since_restore: 10.737780094146729
  time_this_iter_s: 0.10722112655639648
  time_total_s: 10.737780094146729
  timestamp: 1658500292
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: f3a0bef8
  warmup_time: 0.003498077392578125
  
Result for objective_fb926d32:
  date: 2022-07-22_15-31-35
  done: false
  experiment_id: c6af4463441e4a3a8ca1298a043e6151
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 14.477795797697073
  neg_mean_loss: -14.477795797697073
  node_ip: 127.0.0.1
  pid: 47205
  time_since_restore: 0.10373091697692871
  time_this_iter_s: 0.10373091697692871
  time_total_s: 0.10373091697692871
  timestamp: 1658500295
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: fb926d32
  warmup_time: 0.0029900074005126953
  
Result for objective_f59fe9d6:
  date: 2022-07-22_15-31-35
  done: false
  experiment_id: efbda212c15c4bd38cfc5f39dfae8b73
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: -2.5626347652141552
  neg_mean_loss: 2.5626347652141552
  node_ip: 127.0.0.1
  pid: 47185
  time_since_restore: 10.179347038269043
  time_this_iter_s: 0.1074531078338623
  time_total_s: 10.179347038269043
  timestamp: 1658500295
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: f59fe9d6
  warmup_time: 0.003314971923828125
  
Result for objective_f5b1ec08:
  date: 2022-07-22_15-31-35
  done: false
  experiment_id: dadb542868ba4b10adf1ef161c75ea17
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: 0.4790434958168009
  neg_mean_loss: -0.4790434958168009
  node_ip: 127.0.0.1
  pid: 47189
  time_since_restore: 10.170033931732178
  time_this_iter_s: 0.10643315315246582
  time_total_s: 10.170033931732178
  timestamp: 1658500295
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: f5b1ec08
  warmup_time: 0.0031630992889404297
  
Result for objective_f5aedf9a:
  date: 2022-07-22_15-31-35
  done: false
  experiment_id: 9f7847b7440648cb86e73bf1be0ccc05
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: 4.924220339829169
  neg_mean_loss: -4.924220339829169
  node_ip: 127.0.0.1
  pid: 47188
  time_since_restore: 10.189186096191406
  time_this_iter_s: 0.1078641414642334
  time_total_s: 10.189186096191406
  timestamp: 1658500295
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: f5aedf9a
  warmup_time: 0.002830028533935547
  
Result for objective_f59fe9d6:
  date: 2022-07-22_15-31-36
  done: true
  experiment_id: efbda212c15c4bd38cfc5f39dfae8b73
  experiment_tag: 6_activation=tanh,height=-27.4958,steps=100,width=5.5843
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -2.5719085451008423
  neg_mean_loss: 2.5719085451008423
  node_ip: 127.0.0.1
  pid: 47185
  time_since_restore: 10.714513063430786
  time_this_iter_s: 0.1071622371673584
  time_total_s: 10.714513063430786
  timestamp: 1658500296
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: f59fe9d6
  warmup_time: 0.003314971923828125
  
Result for objective_f5b1ec08:
  date: 2022-07-22_15-31-36
  done: true
  experiment_id: dadb542868ba4b10adf1ef161c75ea17
  experiment_tag: 8_activation=tanh,height=4.1410,steps=100,width=16.2739
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 0.4757836498809659
  neg_mean_loss: -0.4757836498809659
  node_ip: 127.0.0.1
  pid: 47189
  time_since_restore: 10.703420877456665
  time_this_iter_s: 0.10684394836425781
  time_total_s: 10.703420877456665
  timestamp: 1658500296
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: f5b1ec08
  warmup_time: 0.0031630992889404297
  

Result for objective_f5aedf9a:
  date: 2022-07-22_15-31-36
  done: true
  experiment_id: 9f7847b7440648cb86e73bf1be0ccc05
  experiment_tag: 7_activation=tanh,height=48.7060,steps=100,width=19.7352
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 4.9215262379377105
  neg_mean_loss: -4.9215262379377105
  node_ip: 127.0.0.1
  pid: 47188
  time_since_restore: 10.734119176864624
  time_this_iter_s: 0.11568498611450195
  time_total_s: 10.734119176864624
  timestamp: 1658500296
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: f5aedf9a
  warmup_time: 0.002830028533935547
  
Result for objective_fd91e28e:
  date: 2022-07-22_15-31-38
  done: false
  experiment_id: 5c3693eeb55b476088ce1de9127c5a39
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 9.70376987633604
  neg_mean_loss: -9.70376987633604
  node_ip: 127.0.0.1
  pid: 47214
  time_since_restore: 0.10451984405517578
  time_this_iter_s: 0.10451984405517578
  time_total_s: 0.10451984405517578
  timestamp: 1658500298
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: fd91e28e
  warmup_time: 0.00302886962890625
  
Result for objective_fb926d32:
  date: 2022-07-22_15-31-40
  done: false
  experiment_id: c6af4463441e4a3a8ca1298a043e6151
  hostname: Kais-MacBook-Pro.local
  iterations: 25
  iterations_since_restore: 26
  mean_loss: 4.859752718513368
  neg_mean_loss: -4.859752718513368
  node_ip: 127.0.0.1
  pid: 47205
  time_since_restore: 5.151448011398315
  time_this_iter_s: 0.11450004577636719
  time_total_s: 5.151448011398315
  timestamp: 1658500300
  timesteps_since_restore: 0
  training_iteration: 26
  trial_id: fb926d32
  warmup_time: 0.0029900074005126953
  
Result for objective_fd91e28e:
  date: 2022-07-22_15-31-43
  done: false
  experiment_id: 5c3693eeb55b476088ce1de9127c5a39
  hostname: Kais-MacBook-Pro.local
  iterations: 46
  iterations_since_restore: 47
  mean_loss: -0.11565561363589152
  neg_mean_loss: 0.11565561363589152
  node_ip: 127.0.0.1
  pid: 47214
  time_since_restore: 5.126538991928101
  time_this_iter_s: 0.10815834999084473
  time_total_s: 5.126538991928101
  timestamp: 1658500303
  timesteps_since_restore: 0
  training_iteration: 47
  trial_id: fd91e28e
  warmup_time: 0.00302886962890625
  
Result for objective_fb926d32:
  date: 2022-07-22_15-31-45
  done: false
  experiment_id: c6af4463441e4a3a8ca1298a043e6151
  hostname: Kais-MacBook-Pro.local
  iterations: 72
  iterations_since_restore: 73
  mean_loss: 4.613811037170093
  neg_mean_loss: -4.613811037170093
  node_ip: 127.0.0.1
  pid: 47205
  time_since_restore: 10.208579063415527
  time_this_iter_s: 0.10763001441955566
  time_total_s: 10.208579063415527
  timestamp: 1658500305
  timesteps_since_restore: 0
  training_iteration: 73
  trial_id: fb926d32
  warmup_time: 0.0029900074005126953
  
Result for objective_fb926d32:
  date: 2022-07-22_15-31-48
  done: true
  experiment_id: c6af4463441e4a3a8ca1298a043e6151
  experiment_tag: 9_activation=tanh,height=44.7780,steps=100,width=10.0724
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 4.577084283144497
  neg_mean_loss: -4.577084283144497
  node_ip: 127.0.0.1
  pid: 47205
  time_since_restore: 13.110869884490967
  time_this_iter_s: 0.10806989669799805
  time_total_s: 13.110869884490967
  timestamp: 1658500308
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: fb926d32
  warmup_time: 0.0029900074005126953
  
Result for objective_fd91e28e:
  date: 2022-07-22_15-31-48
  done: false
  experiment_id: 5c3693eeb55b476088ce1de9127c5a39
  hostname: Kais-MacBook-Pro.local
  iterations: 93
  iterations_since_restore: 94
  mean_loss: -0.20609110794676003
  neg_mean_loss: 0.20609110794676003
  node_ip: 127.0.0.1
  pid: 47214
  time_since_restore: 10.14166784286499
  time_this_iter_s: 0.10436320304870605
  time_total_s: 10.14166784286499
  timestamp: 1658500308
  timesteps_since_restore: 0
  training_iteration: 94
  trial_id: fd91e28e
  warmup_time: 0.00302886962890625
  
Result for objective_fd91e28e:
  date: 2022-07-22_15-31-49
  done: true
  experiment_id: 5c3693eeb55b476088ce1de9127c5a39
  experiment_tag: 10_activation=relu,height=-2.9623,steps=100,width=11.8215
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -0.21150779503682235
  neg_mean_loss: 0.21150779503682235
  node_ip: 127.0.0.1
  pid: 47214
  time_since_restore: 10.793406963348389
  time_this_iter_s: 0.10776925086975098
  time_total_s: 10.793406963348389
  timestamp: 1658500309
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: fd91e28e
  warmup_time: 0.00302886962890625
  

Here are the hyperparamters found to minimize the mean loss of the defined objective.

print("Best hyperparameters found were: ", results.get_best_result().config)

Best hyperparameters found were:  {'steps': 100, 'width': 5.584304853357766, 'height': -27.49576980043919, 'activation': 'tanh'}

Conditional search spaces#

Sometimes we may want to build a more complicated search space that has conditional dependencies on other hyperparameters. In this case, we pass a nested dictionary to objective_two, which has been slightly adjusted from objective to deal with the conditional search space.

def evaluation_fn(step, width, height, mult=1):
    return (0.1 + width * step / 100) ** (-1) + height * 0.1 * mult

def objective_two(config):
    width, height = config["width"], config["height"]
    sub_dict = config["activation"]
    mult = sub_dict.get("mult", 1)
    
    for step in range(config["steps"]):
        intermediate_score = evaluation_fn(step, width, height, mult)
        train.report({"iterations": step, "mean_loss": intermediate_score})
        time.sleep(0.1)

conditional_space = {
    "activation": hp.choice(
        "activation",
        [
            {"activation": "relu", "mult": hp.uniform("mult", 1, 2)},
            {"activation": "tanh"},
        ],
    ),
    "width": hp.uniform("width", 0, 20),
    "height": hp.uniform("height", -100, 100),
    "steps": 100,
}

Now we the define the search algorithm built from HyperOptSearch constrained by ConcurrencyLimiter. When the hyperparameter search space is conditional, we pass it (conditional_space) into HyperOptSearch.

algo = HyperOptSearch(space=conditional_space, metric="mean_loss", mode="min")
algo = ConcurrencyLimiter(algo, max_concurrent=4)

Now we run the experiment, this time with an empty config because we instead provided space to the HyperOptSearch search_alg.

tuner = tune.Tuner(
    objective_two,
    tune_config=tune.TuneConfig(
        metric="mean_loss",
        mode="min",
        search_alg=algo,
        num_samples=num_samples,
    ),
)
results = tuner.fit()

== Status ==
Current time: 2022-07-22 15:32:33 (running for 00:00:44.21)
Memory usage on this node: 10.7/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.29 GiB heap, 0.0/2.0 GiB objects
Current best trial: 0de7d38c with mean_loss=-9.200364093875208 and parameters={'activation': {'activation': 'relu', 'mult': 1.3982639549501585}, 'height': -66.3136247260571, 'steps': 100, 'width': 13.922128223483856}
Result logdir: /Users/kai/ray_results/objective_two_2022-07-22_15-31-49
Number of trials: 10/10 (10 TERMINATED)

Trial name	status	loc	activation/activa...	activation/mult	height	steps	width	loss	iter	total time (s)	iterations	neg_mean_loss
objective_two_05a0ad52	TERMINATED	127.0.0.1:47229	tanh		-63.0091	100	10.1954	-6.20281	100	10.8547	99	6.20281
objective_two_075f0d78	TERMINATED	127.0.0.1:47236	relu	1.22098	46.4977	100	13.6093	5.75095	100	12.0174	99	-5.75095
objective_two_07617f54	TERMINATED	127.0.0.1:47237	tanh		-41.2706	100	7.34134	-3.99133	100	11.8677	99	3.99133
objective_two_07636cd8	TERMINATED	127.0.0.1:47238	tanh		49.0313	100	0.828826	5.98945	100	12.0258	99	-5.98945
objective_two_0de7d38c	TERMINATED	127.0.0.1:47256	relu	1.39826	-66.3136	100	13.9221	-9.20036	100	10.7072	99	9.20036
objective_two_100ebe3c	TERMINATED	127.0.0.1:47265	tanh		-38.8555	100	15.9966	-3.82281	100	10.6571	99	3.82281
objective_two_101e702a	TERMINATED	127.0.0.1:47268	relu	1.51022	5.84901	100	0.431039	2.78184	100	10.7202	99	-2.78184
objective_two_1022212a	TERMINATED	127.0.0.1:47269	relu	1.32257	71.2885	100	11.4466	9.51591	100	10.7072	99	-9.51591
objective_two_15fbf8dc	TERMINATED	127.0.0.1:47288	relu	1.22166	-32.1584	100	9.43222	-3.82272	100	10.5991	99	3.82272
objective_two_18090732	TERMINATED	127.0.0.1:47302	tanh		52.7613	100	16.7268	5.33616	100	10.6336	99	-5.33616

Result for objective_two_05a0ad52:
  date: 2022-07-22_15-31-52
  done: false
  experiment_id: af7041e0b2c947aa8a30c2e30f94ba83
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 3.699090470918877
  neg_mean_loss: -3.699090470918877
  node_ip: 127.0.0.1
  pid: 47229
  time_since_restore: 0.00011491775512695312
  time_this_iter_s: 0.00011491775512695312
  time_total_s: 0.00011491775512695312
  timestamp: 1658500312
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 05a0ad52
  warmup_time: 0.0027589797973632812
  
Result for objective_two_07636cd8:
  date: 2022-07-22_15-31-55
  done: false
  experiment_id: 5c9d75b036ae410b9fdea303f14fe4cd
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 14.903126957374846
  neg_mean_loss: -14.903126957374846
  node_ip: 127.0.0.1
  pid: 47238
  time_since_restore: 0.00011110305786132812
  time_this_iter_s: 0.00011110305786132812
  time_total_s: 0.00011110305786132812
  timestamp: 1658500315
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 07636cd8
  warmup_time: 0.0037419795989990234
  
Result for objective_two_07617f54:
  date: 2022-07-22_15-31-55
  done: false
  experiment_id: 32155cb3b7f04dc3b7b5b679eecd8c46
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 5.872941741395985
  neg_mean_loss: -5.872941741395985
  node_ip: 127.0.0.1
  pid: 47237
  time_since_restore: 0.00015211105346679688
  time_this_iter_s: 0.00015211105346679688
  time_total_s: 0.00015211105346679688
  timestamp: 1658500315
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 07617f54
  warmup_time: 0.00436091423034668
  
Result for objective_two_075f0d78:
  date: 2022-07-22_15-31-55
  done: false
  experiment_id: 5a46a57ef1164628bac1618e39078e0e
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 15.677277832165048
  neg_mean_loss: -15.677277832165048
  node_ip: 127.0.0.1
  pid: 47236
  time_since_restore: 0.00011396408081054688
  time_this_iter_s: 0.00011396408081054688
  time_total_s: 0.00011396408081054688
  timestamp: 1658500315
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 075f0d78
  warmup_time: 0.0035758018493652344
  
Result for objective_two_05a0ad52:
  date: 2022-07-22_15-31-57
  done: false
  experiment_id: af7041e0b2c947aa8a30c2e30f94ba83
  hostname: Kais-MacBook-Pro.local
  iterations: 45
  iterations_since_restore: 46
  mean_loss: -6.087595964816368
  neg_mean_loss: 6.087595964816368
  node_ip: 127.0.0.1
  pid: 47229
  time_since_restore: 5.055715799331665
  time_this_iter_s: 0.10677886009216309
  time_total_s: 5.055715799331665
  timestamp: 1658500317
  timesteps_since_restore: 0
  training_iteration: 46
  trial_id: 05a0ad52
  warmup_time: 0.0027589797973632812
  
Result for objective_two_07617f54:
  date: 2022-07-22_15-32-00
  done: false
  experiment_id: 32155cb3b7f04dc3b7b5b679eecd8c46
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: -3.8454022145530584
  neg_mean_loss: 3.8454022145530584
  node_ip: 127.0.0.1
  pid: 47237
  time_since_restore: 5.048717021942139
  time_this_iter_s: 0.10686898231506348
  time_total_s: 5.048717021942139
  timestamp: 1658500320
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 07617f54
  warmup_time: 0.00436091423034668
  
Result for objective_two_07636cd8:
  date: 2022-07-22_15-32-00
  done: false
  experiment_id: 5c9d75b036ae410b9fdea303f14fe4cd
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 6.9458266379188185
  neg_mean_loss: -6.9458266379188185
  node_ip: 127.0.0.1
  pid: 47238
  time_since_restore: 5.097726821899414
  time_this_iter_s: 0.1087799072265625
  time_total_s: 5.097726821899414
  timestamp: 1658500320
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 07636cd8
  warmup_time: 0.0037419795989990234
  
Result for objective_two_075f0d78:
  date: 2022-07-22_15-32-00
  done: false
  experiment_id: 5a46a57ef1164628bac1618e39078e0e
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 5.83121027007688
  neg_mean_loss: -5.83121027007688
  node_ip: 127.0.0.1
  pid: 47236
  time_since_restore: 5.118211269378662
  time_this_iter_s: 0.10918712615966797
  time_total_s: 5.118211269378662
  timestamp: 1658500320
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 075f0d78
  warmup_time: 0.0035758018493652344
  
Result for objective_two_05a0ad52:
  date: 2022-07-22_15-32-02
  done: false
  experiment_id: af7041e0b2c947aa8a30c2e30f94ba83
  hostname: Kais-MacBook-Pro.local
  iterations: 92
  iterations_since_restore: 93
  mean_loss: -6.195421815991769
  neg_mean_loss: 6.195421815991769
  node_ip: 127.0.0.1
  pid: 47229
  time_since_restore: 10.10265588760376
  time_this_iter_s: 0.10532593727111816
  time_total_s: 10.10265588760376
  timestamp: 1658500322
  timesteps_since_restore: 0
  training_iteration: 93
  trial_id: 05a0ad52
  warmup_time: 0.0027589797973632812
  
Result for objective_two_05a0ad52:
  date: 2022-07-22_15-32-03
  done: true
  experiment_id: af7041e0b2c947aa8a30c2e30f94ba83
  experiment_tag: 1_activation=tanh,height=-63.0091,steps=100,width=10.1954
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -6.202807371456877
  neg_mean_loss: 6.202807371456877
  node_ip: 127.0.0.1
  pid: 47229
  time_since_restore: 10.854680061340332
  time_this_iter_s: 0.10699224472045898
  time_total_s: 10.854680061340332
  timestamp: 1658500323
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 05a0ad52
  warmup_time: 0.0027589797973632812
  
Result for objective_two_075f0d78:
  date: 2022-07-22_15-32-04
  done: false
  experiment_id: 5a46a57ef1164628bac1618e39078e0e
  hostname: Kais-MacBook-Pro.local
  iterations: 91
  iterations_since_restore: 92
  mean_loss: 5.757377572321986
  neg_mean_loss: -5.757377572321986
  node_ip: 127.0.0.1
  pid: 47236
  time_since_restore: 9.820771932601929
  time_this_iter_s: 0.10582685470581055
  time_total_s: 9.820771932601929
  timestamp: 1658500324
  timesteps_since_restore: 0
  training_iteration: 92
  trial_id: 075f0d78
  warmup_time: 0.0035758018493652344
  
Result for objective_two_07617f54:
  date: 2022-07-22_15-32-04
  done: false
  experiment_id: 32155cb3b7f04dc3b7b5b679eecd8c46
  hostname: Kais-MacBook-Pro.local
  iterations: 92
  iterations_since_restore: 93
  mean_loss: -3.981158750752472
  neg_mean_loss: 3.981158750752472
  node_ip: 127.0.0.1
  pid: 47237
  time_since_restore: 9.865068197250366
  time_this_iter_s: 0.1065821647644043
  time_total_s: 9.865068197250366
  timestamp: 1658500324
  timesteps_since_restore: 0
  training_iteration: 93
  trial_id: 07617f54
  warmup_time: 0.00436091423034668
  
Result for objective_two_0de7d38c:
  date: 2022-07-22_15-32-06
  done: false
  experiment_id: 81d62307312a447e97c5d1f1cdad2687
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 0.7276048823462773
  neg_mean_loss: -0.7276048823462773
  node_ip: 127.0.0.1
  pid: 47256
  time_since_restore: 0.00012803077697753906
  time_this_iter_s: 0.00012803077697753906
  time_total_s: 0.00012803077697753906
  timestamp: 1658500326
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 0de7d38c
  warmup_time: 0.0027358531951904297
  
Result for objective_two_07636cd8:
  date: 2022-07-22_15-32-04
  done: false
  experiment_id: 5c9d75b036ae410b9fdea303f14fe4cd
  hostname: Kais-MacBook-Pro.local
  iterations: 91
  iterations_since_restore: 92
  mean_loss: 6.073769581173668
  neg_mean_loss: -6.073769581173668
  node_ip: 127.0.0.1
  pid: 47238
  time_since_restore: 9.813458919525146
  time_this_iter_s: 0.10477709770202637
  time_total_s: 9.813458919525146
  timestamp: 1658500324
  timesteps_since_restore: 0
  training_iteration: 92
  trial_id: 07636cd8
  warmup_time: 0.0037419795989990234
  
Result for objective_two_07617f54:
  date: 2022-07-22_15-32-06
  done: true
  experiment_id: 32155cb3b7f04dc3b7b5b679eecd8c46
  experiment_tag: 3_activation=tanh,height=-41.2706,steps=100,width=7.3413
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -3.9913348635947523
  neg_mean_loss: 3.9913348635947523
  node_ip: 127.0.0.1
  pid: 47237
  time_since_restore: 11.867748260498047
  time_this_iter_s: 0.10774397850036621
  time_total_s: 11.867748260498047
  timestamp: 1658500326
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 07617f54
  warmup_time: 0.00436091423034668
  
Result for objective_two_075f0d78:
  date: 2022-07-22_15-32-07
  done: true
  experiment_id: 5a46a57ef1164628bac1618e39078e0e
  experiment_tag: 2_activation=relu,mult=1.2210,height=46.4977,steps=100,width=13.6093
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 5.7509525535298085
  neg_mean_loss: -5.7509525535298085
  node_ip: 127.0.0.1
  pid: 47236
  time_since_restore: 12.017354011535645
  time_this_iter_s: 0.10736513137817383
  time_total_s: 12.017354011535645
  timestamp: 1658500327
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 075f0d78
  warmup_time: 0.0035758018493652344
  
Result for objective_two_07636cd8:
  date: 2022-07-22_15-32-07
  done: true
  experiment_id: 5c9d75b036ae410b9fdea303f14fe4cd
  experiment_tag: 4_activation=tanh,height=49.0313,steps=100,width=0.8288
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 5.989448515159168
  neg_mean_loss: -5.989448515159168
  node_ip: 127.0.0.1
  pid: 47238
  time_since_restore: 12.025846004486084
  time_this_iter_s: 0.10526180267333984
  time_total_s: 12.025846004486084
  timestamp: 1658500327
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 07636cd8
  warmup_time: 0.0037419795989990234
  

Result for objective_two_100ebe3c:
  date: 2022-07-22_15-32-09
  done: false
  experiment_id: f31e00c7c87c4884bef04183585473d1
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 6.11444579492732
  neg_mean_loss: -6.11444579492732
  node_ip: 127.0.0.1
  pid: 47265
  time_since_restore: 0.00012183189392089844
  time_this_iter_s: 0.00012183189392089844
  time_total_s: 0.00012183189392089844
  timestamp: 1658500329
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 100ebe3c
  warmup_time: 0.002948284149169922
  
Result for objective_two_101e702a:
  date: 2022-07-22_15-32-09
  done: false
  experiment_id: 8c013d0ff6434dfe9719fa317f95feb8
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 10.883328850459264
  neg_mean_loss: -10.883328850459264
  node_ip: 127.0.0.1
  pid: 47268
  time_since_restore: 0.00011587142944335938
  time_this_iter_s: 0.00011587142944335938
  time_total_s: 0.00011587142944335938
  timestamp: 1658500329
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 101e702a
  warmup_time: 0.003309965133666992
  
Result for objective_two_1022212a:
  date: 2022-07-22_15-32-09
  done: false
  experiment_id: c3c0bdb8b68146c68f2c34e08fa2f184
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 19.42843735686162
  neg_mean_loss: -19.42843735686162
  node_ip: 127.0.0.1
  pid: 47269
  time_since_restore: 0.00011324882507324219
  time_this_iter_s: 0.00011324882507324219
  time_total_s: 0.00011324882507324219
  timestamp: 1658500329
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 1022212a
  warmup_time: 0.0029020309448242188
  
Result for objective_two_0de7d38c:
  date: 2022-07-22_15-32-11
  done: false
  experiment_id: 81d62307312a447e97c5d1f1cdad2687
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: -9.121869790245293
  neg_mean_loss: 9.121869790245293
  node_ip: 127.0.0.1
  pid: 47256
  time_since_restore: 5.047628164291382
  time_this_iter_s: 0.10473132133483887
  time_total_s: 5.047628164291382
  timestamp: 1658500331
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 0de7d38c
  warmup_time: 0.0027358531951904297
  
Result for objective_two_100ebe3c:
  date: 2022-07-22_15-32-14
  done: false
  experiment_id: f31e00c7c87c4884bef04183585473d1
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: -3.7542934113649324
  neg_mean_loss: 3.7542934113649324
  node_ip: 127.0.0.1
  pid: 47265
  time_since_restore: 5.057713985443115
  time_this_iter_s: 0.10337996482849121
  time_total_s: 5.057713985443115
  timestamp: 1658500334
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 100ebe3c
  warmup_time: 0.002948284149169922
  
Result for objective_two_101e702a:
  date: 2022-07-22_15-32-14
  done: false
  experiment_id: 8c013d0ff6434dfe9719fa317f95feb8
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 4.188150411466976
  neg_mean_loss: -4.188150411466976
  node_ip: 127.0.0.1
  pid: 47268
  time_since_restore: 5.101155757904053
  time_this_iter_s: 0.10826706886291504
  time_total_s: 5.101155757904053
  timestamp: 1658500334
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 101e702a
  warmup_time: 0.003309965133666992
  
Result for objective_two_1022212a:
  date: 2022-07-22_15-32-14
  done: false
  experiment_id: c3c0bdb8b68146c68f2c34e08fa2f184
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 9.610922254324745
  neg_mean_loss: -9.610922254324745
  node_ip: 127.0.0.1
  pid: 47269
  time_since_restore: 5.09367823600769
  time_this_iter_s: 0.11145401000976562
  time_total_s: 5.09367823600769
  timestamp: 1658500334
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 1022212a
  warmup_time: 0.0029020309448242188
  
Result for objective_two_0de7d38c:
  date: 2022-07-22_15-32-16
  done: false
  experiment_id: 81d62307312a447e97c5d1f1cdad2687
  hostname: Kais-MacBook-Pro.local
  iterations: 93
  iterations_since_restore: 94
  mean_loss: -9.195752548100788
  neg_mean_loss: 9.195752548100788
  node_ip: 127.0.0.1
  pid: 47256
  time_since_restore: 10.068511009216309
  time_this_iter_s: 0.10715794563293457
  time_total_s: 10.068511009216309
  timestamp: 1658500336
  timesteps_since_restore: 0
  training_iteration: 94
  trial_id: 0de7d38c
  warmup_time: 0.0027358531951904297
  
Result for objective_two_0de7d38c:
  date: 2022-07-22_15-32-16
  done: true
  experiment_id: 81d62307312a447e97c5d1f1cdad2687
  experiment_tag: 5_activation=relu,mult=1.3983,height=-66.3136,steps=100,width=13.9221
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -9.200364093875208
  neg_mean_loss: 9.200364093875208
  node_ip: 127.0.0.1
  pid: 47256
  time_since_restore: 10.707159996032715
  time_this_iter_s: 0.1063990592956543
  time_total_s: 10.707159996032715
  timestamp: 1658500336
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 0de7d38c
  warmup_time: 0.0027358531951904297
  
Result for objective_two_100ebe3c:
  date: 2022-07-22_15-32-19
  done: false
  experiment_id: f31e00c7c87c4884bef04183585473d1
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: -3.8194902277136236
  neg_mean_loss: 3.8194902277136236
  node_ip: 127.0.0.1
  pid: 47265
  time_since_restore: 10.118366956710815
  time_this_iter_s: 0.10590505599975586
  time_total_s: 10.118366956710815
  timestamp: 1658500339
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: 100ebe3c
  warmup_time: 0.002948284149169922
  
Result for objective_two_15fbf8dc:
  date: 2022-07-22_15-32-19
  done: false
  experiment_id: 825ceb15d1134810ba19f09bb90c2b35
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 6.071327317808535
  neg_mean_loss: -6.071327317808535
  node_ip: 127.0.0.1
  pid: 47288
  time_since_restore: 0.0002067089080810547
  time_this_iter_s: 0.0002067089080810547
  time_total_s: 0.0002067089080810547
  timestamp: 1658500339
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 15fbf8dc
  warmup_time: 0.005033969879150391
  
Result for objective_two_101e702a:
  date: 2022-07-22_15-32-19
  done: false
  experiment_id: 8c013d0ff6434dfe9719fa317f95feb8
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: 2.86283543260466
  neg_mean_loss: -2.86283543260466
  node_ip: 127.0.0.1
  pid: 47268
  time_since_restore: 10.181840896606445
  time_this_iter_s: 0.10625004768371582
  time_total_s: 10.181840896606445
  timestamp: 1658500339
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: 101e702a
  warmup_time: 0.003309965133666992
  
Result for objective_two_1022212a:
  date: 2022-07-22_15-32-19
  done: false
  experiment_id: c3c0bdb8b68146c68f2c34e08fa2f184
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: 9.520519990087298
  neg_mean_loss: -9.520519990087298
  node_ip: 127.0.0.1
  pid: 47269
  time_since_restore: 10.166353225708008
  time_this_iter_s: 0.10583710670471191
  time_total_s: 10.166353225708008
  timestamp: 1658500339
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: 1022212a
  warmup_time: 0.0029020309448242188
  
Result for objective_two_100ebe3c:
  date: 2022-07-22_15-32-20
  done: true
  experiment_id: f31e00c7c87c4884bef04183585473d1
  experiment_tag: 6_activation=tanh,height=-38.8555,steps=100,width=15.9966
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -3.8228058558351754
  neg_mean_loss: 3.8228058558351754
  node_ip: 127.0.0.1
  pid: 47265
  time_since_restore: 10.657065153121948
  time_this_iter_s: 0.10721731185913086
  time_total_s: 10.657065153121948
  timestamp: 1658500340
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 100ebe3c
  warmup_time: 0.002948284149169922
  
Result for objective_two_101e702a:
  date: 2022-07-22_15-32-20
  done: true
  experiment_id: 8c013d0ff6434dfe9719fa317f95feb8
  experiment_tag: 7_activation=relu,mult=1.5102,height=5.8490,steps=100,width=0.4310
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 2.781840740489214
  neg_mean_loss: -2.781840740489214
  node_ip: 127.0.0.1
  pid: 47268
  time_since_restore: 10.72019100189209
  time_this_iter_s: 0.1079869270324707
  time_total_s: 10.72019100189209
  timestamp: 1658500340
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 101e702a
  warmup_time: 0.003309965133666992
  
Result for objective_two_1022212a:
  date: 2022-07-22_15-32-20
  done: true
  experiment_id: c3c0bdb8b68146c68f2c34e08fa2f184
  experiment_tag: 8_activation=relu,mult=1.3226,height=71.2885,steps=100,width=11.4466
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 9.515910032420853
  neg_mean_loss: -9.515910032420853
  node_ip: 127.0.0.1
  pid: 47269
  time_since_restore: 10.707203149795532
  time_this_iter_s: 0.10514593124389648
  time_total_s: 10.707203149795532
  timestamp: 1658500340
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 1022212a
  warmup_time: 0.0029020309448242188
  

Result for objective_two_18090732:
  date: 2022-07-22_15-32-23
  done: false
  experiment_id: 7cb1145f46214bc4a5dd35e796969e53
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 15.276134004304078
  neg_mean_loss: -15.276134004304078
  node_ip: 127.0.0.1
  pid: 47302
  time_since_restore: 0.00011110305786132812
  time_this_iter_s: 0.00011110305786132812
  time_total_s: 0.00011110305786132812
  timestamp: 1658500343
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: '18090732'
  warmup_time: 0.0028650760650634766
  
Result for objective_two_15fbf8dc:
  date: 2022-07-22_15-32-24
  done: false
  experiment_id: 825ceb15d1134810ba19f09bb90c2b35
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: -3.708075105876967
  neg_mean_loss: 3.708075105876967
  node_ip: 127.0.0.1
  pid: 47288
  time_since_restore: 5.015958786010742
  time_this_iter_s: 0.10610795021057129
  time_total_s: 5.015958786010742
  timestamp: 1658500344
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 15fbf8dc
  warmup_time: 0.005033969879150391
  
Result for objective_two_18090732:
  date: 2022-07-22_15-32-28
  done: false
  experiment_id: 7cb1145f46214bc4a5dd35e796969e53
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 5.4017373166361775
  neg_mean_loss: -5.4017373166361775
  node_ip: 127.0.0.1
  pid: 47302
  time_since_restore: 5.047500133514404
  time_this_iter_s: 0.10670995712280273
  time_total_s: 5.047500133514404
  timestamp: 1658500348
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: '18090732'
  warmup_time: 0.0028650760650634766
  
Result for objective_two_15fbf8dc:
  date: 2022-07-22_15-32-29
  done: false
  experiment_id: 825ceb15d1134810ba19f09bb90c2b35
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: -3.8171437433543964
  neg_mean_loss: 3.8171437433543964
  node_ip: 127.0.0.1
  pid: 47288
  time_since_restore: 10.057979822158813
  time_this_iter_s: 0.10869002342224121
  time_total_s: 10.057979822158813
  timestamp: 1658500349
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: 15fbf8dc
  warmup_time: 0.005033969879150391
  
Result for objective_two_15fbf8dc:
  date: 2022-07-22_15-32-30
  done: true
  experiment_id: 825ceb15d1134810ba19f09bb90c2b35
  experiment_tag: 9_activation=relu,mult=1.2217,height=-32.1584,steps=100,width=9.4322
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -3.8227168355020016
  neg_mean_loss: 3.8227168355020016
  node_ip: 127.0.0.1
  pid: 47288
  time_since_restore: 10.599114894866943
  time_this_iter_s: 0.10727405548095703
  time_total_s: 10.599114894866943
  timestamp: 1658500350
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 15fbf8dc
  warmup_time: 0.005033969879150391
  
Result for objective_two_18090732:
  date: 2022-07-22_15-32-33
  done: false
  experiment_id: 7cb1145f46214bc4a5dd35e796969e53
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: 5.339332557853146
  neg_mean_loss: -5.339332557853146
  node_ip: 127.0.0.1
  pid: 47302
  time_since_restore: 10.092173099517822
  time_this_iter_s: 0.1067051887512207
  time_total_s: 10.092173099517822
  timestamp: 1658500353
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: '18090732'
  warmup_time: 0.0028650760650634766
  
Result for objective_two_18090732:
  date: 2022-07-22_15-32-33
  done: true
  experiment_id: 7cb1145f46214bc4a5dd35e796969e53
  experiment_tag: 10_activation=tanh,height=52.7613,steps=100,width=16.7268
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 5.336159871047403
  neg_mean_loss: -5.336159871047403
  node_ip: 127.0.0.1
  pid: 47302
  time_since_restore: 10.633639097213745
  time_this_iter_s: 0.1067359447479248
  time_total_s: 10.633639097213745
  timestamp: 1658500353
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: '18090732'
  warmup_time: 0.0028650760650634766
  

Finally, we again show the hyperparameters that minimize the mean loss defined by the score of the objective function above.

print("Best hyperparameters found were: ", results.get_best_result().config)

Best hyperparameters found were:  {'activation': {'activation': 'relu', 'mult': 1.3982639549501585}, 'height': -66.3136247260571, 'steps': 100, 'width': 13.922128223483856}