Running Tune experiments with Skopt

In this tutorial we introduce Skopt, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with Skopt and, as a result, allow you to seamlessly scale up a Skopt optimization process - without sacrificing performance.

Scikit-Optimize, or skopt, is a simple and efficient library to optimize expensive and noisy black-box functions, e.g. large-scale ML experiments. It implements several methods for sequential model-based optimization. Noteably, skopt does not perform gradient-based optimization, and instead uses computationally cheap surrogate models to approximate the expensive function. In this example we minimize a simple objective to briefly demonstrate the usage of Skopt with Ray Tune via SkOptSearch. It’s useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume scikit-opitmize==0.8.1 library is installed. To learn more, please refer to the Scikit-Optimize website.

Click below to see all the imports we need for this example. You can also launch directly into a Binder instance to run this notebook yourself. Just click on the rocket symbol at the top of the navigation.

import time
from typing import Dict, Optional, Any

import ray

import skopt
from ray import tune
from ray.air import session
from ray.tune.search import ConcurrencyLimiter
from ray.tune.search.skopt import SkOptSearch

Let’s start by defining a simple evaluation function. Again, an explicit math formula is queried here for demonstration, yet in practice this is typically a black-box function– e.g. the performance results after training an ML model. We artificially sleep for a bit (0.1 seconds) to simulate a long-running ML experiment. This setup assumes that we’re running multiple steps of an experiment while tuning three hyperparameters, namely width, height, and activation.

def evaluate(step, width, height, activation):
    time.sleep(0.1)
    activation_boost = 10 if activation=="relu" else 0
    return (0.1 + width * step / 100) ** (-1) + height * 0.1 + activation_boost

Next, our objective function to be optimized takes a Tune config, evaluates the score of your experiment in a training loop, and uses session.report to report the score back to Tune.

def objective(config):
    for step in range(config["steps"]):
        score = evaluate(step, config["width"], config["height"], config["activation"])
        session.report({"iterations": step, "mean_loss": score})

Next we define a search space. The critical assumption is that the optimal hyperparamters live within this space. Yet, if the space is very large, then those hyperparameters may be difficult to find in a short amount of time.

search_space = {
    "steps": 100,
    "width": tune.uniform(0, 20),
    "height": tune.uniform(-100, 100),
    "activation": tune.choice(["relu", "tanh"]),
}

The search algorithm is instantiated from the SkOptSearch class. We also constrain the the number of concurrent trials to 4 with a ConcurrencyLimiter.

algo = SkOptSearch()
algo = ConcurrencyLimiter(algo, max_concurrent=4)

The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to 1000 samples. (you can decrease this if it takes too long on your machine).

num_samples = 1000

Finally, we run the experiment to "min"imize the “mean_loss” of the objective by searching search_config via algo, num_samples times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute tuner.fit().

tuner = tune.Tuner(
    objective,
    tune_config=tune.TuneConfig(
        metric="mean_loss",
        mode="min",
        search_alg=algo,
        num_samples=num_samples,
    ),
    param_space=search_space,
)
results = tuner.fit()
Function checkpointing is disabled. This may result in unexpected behavior when using checkpointing features or certain schedulers. To enable, set the train function arguments to be `func(config, checkpoint_dir=None)`.
== Status ==
Current time: 2022-07-22 15:47:07 (running for 00:00:44.11)
Memory usage on this node: 9.9/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.42 GiB heap, 0.0/2.0 GiB objects
Current best trial: 20e93604 with mean_loss=-8.869507755259557 and parameters={'steps': 100, 'width': 8.246480084328711, 'height': -89.90514307848873, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-46-22
Number of trials: 10/10 (10 TERMINATED)
Trial name status loc activation height width loss iter total time (s) iterations neg_mean_loss
objective_0ec1ec00TERMINATED127.0.0.1:48435tanh 10.9976 8.60092 1.21584 100 10.941 99 -1.21584
objective_107c17faTERMINATED127.0.0.1:48446relu 49.545511.8085 15.0394 100 11.5877 99 -15.0394
objective_107d75f0TERMINATED127.0.0.1:48447tanh -7.6131 3.10991-0.446727 100 11.7109 99 0.446727
objective_107ef722TERMINATED127.0.0.1:48448relu -75.8366 5.30975 2.60302 100 11.6501 99 -2.60302
objective_17033b12TERMINATED127.0.0.1:48470tanh -89.189112.1463 -8.83643 100 10.7255 99 8.83643
objective_18fbb20aTERMINATED127.0.0.1:48475tanh 33.5568 9.75437 3.45818 100 10.7563 99 -3.45818
objective_190ab4d0TERMINATED127.0.0.1:48478relu -65.148 4.19546 3.7203 100 10.7123 99 -3.7203
objective_191ca3deTERMINATED127.0.0.1:48481tanh -13.4892 6.74078-1.20128 100 13.219 99 1.20128
objective_1eec012eTERMINATED127.0.0.1:48494tanh -50.281914.8287 -4.96053 100 13.2232 99 4.96053
objective_20e93604TERMINATED127.0.0.1:48502tanh -89.9051 8.24648-8.86951 100 10.7925 99 8.86951


Result for objective_0ec1ec00:
  date: 2022-07-22_15-46-26
  done: false
  experiment_id: c3e1e5ac1d6c4d67878bf793d020f99e
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 11.099760178357887
  neg_mean_loss: -11.099760178357887
  node_ip: 127.0.0.1
  pid: 48435
  time_since_restore: 0.10384106636047363
  time_this_iter_s: 0.10384106636047363
  time_total_s: 0.10384106636047363
  timestamp: 1658501186
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 0ec1ec00
  warmup_time: 0.003099679946899414
  
Result for objective_107c17fa:
  date: 2022-07-22_15-46-29
  done: false
  experiment_id: f2a0e8ad4ea64c489424406b75b5c051
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 24.95455186350225
  neg_mean_loss: -24.95455186350225
  node_ip: 127.0.0.1
  pid: 48446
  time_since_restore: 0.10309791564941406
  time_this_iter_s: 0.10309791564941406
  time_total_s: 0.10309791564941406
  timestamp: 1658501189
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 107c17fa
  warmup_time: 0.0029959678649902344
  
Result for objective_107ef722:
  date: 2022-07-22_15-46-29
  done: false
  experiment_id: 286beb82d0f0493dacbafb0ceb9bfe2d
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 12.416336029599446
  neg_mean_loss: -12.416336029599446
  node_ip: 127.0.0.1
  pid: 48448
  time_since_restore: 0.10424971580505371
  time_this_iter_s: 0.10424971580505371
  time_total_s: 0.10424971580505371
  timestamp: 1658501189
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 107ef722
  warmup_time: 0.0032057762145996094
  
Result for objective_107d75f0:
  date: 2022-07-22_15-46-29
  done: false
  experiment_id: a44e7e5deb2e4bc68b31ca0466db50f5
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 9.238689666819806
  neg_mean_loss: -9.238689666819806
  node_ip: 127.0.0.1
  pid: 48447
  time_since_restore: 0.10475015640258789
  time_this_iter_s: 0.10475015640258789
  time_total_s: 0.10475015640258789
  timestamp: 1658501189
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 107d75f0
  warmup_time: 0.0029137134552001953
  
Result for objective_0ec1ec00:
  date: 2022-07-22_15-46-31
  done: false
  experiment_id: c3e1e5ac1d6c4d67878bf793d020f99e
  hostname: Kais-MacBook-Pro.local
  iterations: 45
  iterations_since_restore: 46
  mean_loss: 1.3516231012646176
  neg_mean_loss: -1.3516231012646176
  node_ip: 127.0.0.1
  pid: 48435
  time_since_restore: 5.127159833908081
  time_this_iter_s: 0.10582566261291504
  time_total_s: 5.127159833908081
  timestamp: 1658501191
  timesteps_since_restore: 0
  training_iteration: 46
  trial_id: 0ec1ec00
  warmup_time: 0.003099679946899414
  
Result for objective_107c17fa:
  date: 2022-07-22_15-46-34
  done: false
  experiment_id: f2a0e8ad4ea64c489424406b75b5c051
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 15.131543197018463
  neg_mean_loss: -15.131543197018463
  node_ip: 127.0.0.1
  pid: 48446
  time_since_restore: 5.161302089691162
  time_this_iter_s: 0.10862898826599121
  time_total_s: 5.161302089691162
  timestamp: 1658501194
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 107c17fa
  warmup_time: 0.0029959678649902344
  
Result for objective_107ef722:
  date: 2022-07-22_15-46-34
  done: false
  experiment_id: 286beb82d0f0493dacbafb0ceb9bfe2d
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 2.8016060611446587
  neg_mean_loss: -2.8016060611446587
  node_ip: 127.0.0.1
  pid: 48448
  time_since_restore: 5.157445907592773
  time_this_iter_s: 0.10624003410339355
  time_total_s: 5.157445907592773
  timestamp: 1658501194
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 107ef722
  warmup_time: 0.0032057762145996094
  
Result for objective_107d75f0:
  date: 2022-07-22_15-46-34
  done: false
  experiment_id: a44e7e5deb2e4bc68b31ca0466db50f5
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: -0.12096483365591593
  neg_mean_loss: 0.12096483365591593
  node_ip: 127.0.0.1
  pid: 48447
  time_since_restore: 5.159098863601685
  time_this_iter_s: 0.10865378379821777
  time_total_s: 5.159098863601685
  timestamp: 1658501194
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 107d75f0
  warmup_time: 0.0029137134552001953
  
Result for objective_0ec1ec00:
  date: 2022-07-22_15-46-36
  done: false
  experiment_id: c3e1e5ac1d6c4d67878bf793d020f99e
  hostname: Kais-MacBook-Pro.local
  iterations: 92
  iterations_since_restore: 93
  mean_loss: 1.2245597846713034
  neg_mean_loss: -1.2245597846713034
  node_ip: 127.0.0.1
  pid: 48435
  time_since_restore: 10.187216997146606
  time_this_iter_s: 0.10774493217468262
  time_total_s: 10.187216997146606
  timestamp: 1658501196
  timesteps_since_restore: 0
  training_iteration: 93
  trial_id: 0ec1ec00
  warmup_time: 0.003099679946899414
  
Result for objective_0ec1ec00:
  date: 2022-07-22_15-46-37
  done: true
  experiment_id: c3e1e5ac1d6c4d67878bf793d020f99e
  experiment_tag: 1_activation=tanh,height=10.9976,steps=100,width=8.6009
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 1.2158379996438775
  neg_mean_loss: -1.2158379996438775
  node_ip: 127.0.0.1
  pid: 48435
  time_since_restore: 10.9409658908844
  time_this_iter_s: 0.10762286186218262
  time_total_s: 10.9409658908844
  timestamp: 1658501197
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 0ec1ec00
  warmup_time: 0.003099679946899414
  
Result for objective_107ef722:
  date: 2022-07-22_15-46-39
  done: false
  experiment_id: 286beb82d0f0493dacbafb0ceb9bfe2d
  hostname: Kais-MacBook-Pro.local
  iterations: 91
  iterations_since_restore: 92
  mean_loss: 2.619098804697928
  neg_mean_loss: -2.619098804697928
  node_ip: 127.0.0.1
  pid: 48448
  time_since_restore: 9.851147890090942
  time_this_iter_s: 0.1064901351928711
  time_total_s: 9.851147890090942
  timestamp: 1658501199
  timesteps_since_restore: 0
  training_iteration: 92
  trial_id: 107ef722
  warmup_time: 0.0032057762145996094
  
Result for objective_107c17fa:
  date: 2022-07-22_15-46-39
  done: false
  experiment_id: f2a0e8ad4ea64c489424406b75b5c051
  hostname: Kais-MacBook-Pro.local
  iterations: 92
  iterations_since_restore: 93
  mean_loss: 15.0457609631331
  neg_mean_loss: -15.0457609631331
  node_ip: 127.0.0.1
  pid: 48446
  time_since_restore: 9.976460695266724
  time_this_iter_s: 0.10694289207458496
  time_total_s: 9.976460695266724
  timestamp: 1658501199
  timesteps_since_restore: 0
  training_iteration: 93
  trial_id: 107c17fa
  warmup_time: 0.0029959678649902344
  
Result for objective_107d75f0:
  date: 2022-07-22_15-46-39
  done: false
  experiment_id: a44e7e5deb2e4bc68b31ca0466db50f5
  hostname: Kais-MacBook-Pro.local
  iterations: 90
  iterations_since_restore: 91
  mean_loss: -0.41635399078466045
  neg_mean_loss: 0.41635399078466045
  node_ip: 127.0.0.1
  pid: 48447
  time_since_restore: 9.776082992553711
  time_this_iter_s: 0.10715007781982422
  time_total_s: 9.776082992553711
  timestamp: 1658501199
  timesteps_since_restore: 0
  training_iteration: 91
  trial_id: 107d75f0
  warmup_time: 0.0029137134552001953
  
Result for objective_17033b12:
  date: 2022-07-22_15-46-40
  done: false
  experiment_id: b5f76922e6424055b6aa255677e2f4b8
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 1.0810921171053653
  neg_mean_loss: -1.0810921171053653
  node_ip: 127.0.0.1
  pid: 48470
  time_since_restore: 0.10409784317016602
  time_this_iter_s: 0.10409784317016602
  time_total_s: 0.10409784317016602
  timestamp: 1658501200
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 17033b12
  warmup_time: 0.0029649734497070312
  
Result for objective_107c17fa:
  date: 2022-07-22_15-46-41
  done: true
  experiment_id: f2a0e8ad4ea64c489424406b75b5c051
  experiment_tag: 2_activation=relu,height=49.5455,steps=100,width=11.8085
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 15.039366532944502
  neg_mean_loss: -15.039366532944502
  node_ip: 127.0.0.1
  pid: 48446
  time_since_restore: 11.587661743164062
  time_this_iter_s: 0.10614180564880371
  time_total_s: 11.587661743164062
  timestamp: 1658501201
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 107c17fa
  warmup_time: 0.0029959678649902344
  
Result for objective_107ef722:
  date: 2022-07-22_15-46-41
  done: true
  experiment_id: 286beb82d0f0493dacbafb0ceb9bfe2d
  experiment_tag: 4_activation=relu,height=-75.8366,steps=100,width=5.3097
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 2.603019812948376
  neg_mean_loss: -2.603019812948376
  node_ip: 127.0.0.1
  pid: 48448
  time_since_restore: 11.65014386177063
  time_this_iter_s: 0.10755181312561035
  time_total_s: 11.65014386177063
  timestamp: 1658501201
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 107ef722
  warmup_time: 0.0032057762145996094
  
Result for objective_107d75f0:
  date: 2022-07-22_15-46-41
  done: true
  experiment_id: a44e7e5deb2e4bc68b31ca0466db50f5
  experiment_tag: 3_activation=tanh,height=-7.6131,steps=100,width=3.1099
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -0.44672713582845847
  neg_mean_loss: 0.44672713582845847
  node_ip: 127.0.0.1
  pid: 48447
  time_since_restore: 11.710935115814209
  time_this_iter_s: 0.1116342544555664
  time_total_s: 11.710935115814209
  timestamp: 1658501201
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 107d75f0
  warmup_time: 0.0029137134552001953
  
Result for objective_18fbb20a:
  date: 2022-07-22_15-46-43
  done: false
  experiment_id: 5e399bc125a548739dd433f4153019fe
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 13.355684066332913
  neg_mean_loss: -13.355684066332913
  node_ip: 127.0.0.1
  pid: 48475
  time_since_restore: 0.10402607917785645
  time_this_iter_s: 0.10402607917785645
  time_total_s: 0.10402607917785645
  timestamp: 1658501203
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 18fbb20a
  warmup_time: 0.003000974655151367
  
Result for objective_190ab4d0:
  date: 2022-07-22_15-46-43
  done: false
  experiment_id: 1b007a40cf6d425085379a4314739912
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 13.48520025532623
  neg_mean_loss: -13.48520025532623
  node_ip: 127.0.0.1
  pid: 48478
  time_since_restore: 0.10344505310058594
  time_this_iter_s: 0.10344505310058594
  time_total_s: 0.10344505310058594
  timestamp: 1658501203
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 190ab4d0
  warmup_time: 0.0028760433197021484
  
Result for objective_191ca3de:
  date: 2022-07-22_15-46-43
  done: false
  experiment_id: e3e72bbeb8ae46f0a4f1624db62bc730
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 8.651082223836086
  neg_mean_loss: -8.651082223836086
  node_ip: 127.0.0.1
  pid: 48481
  time_since_restore: 0.1036369800567627
  time_this_iter_s: 0.1036369800567627
  time_total_s: 0.1036369800567627
  timestamp: 1658501203
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 191ca3de
  warmup_time: 0.0029621124267578125
  
Result for objective_17033b12:
  date: 2022-07-22_15-46-45
  done: false
  experiment_id: b5f76922e6424055b6aa255677e2f4b8
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: -8.746754140047871
  neg_mean_loss: 8.746754140047871
  node_ip: 127.0.0.1
  pid: 48470
  time_since_restore: 5.144078969955444
  time_this_iter_s: 0.10620617866516113
  time_total_s: 5.144078969955444
  timestamp: 1658501205
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 17033b12
  warmup_time: 0.0029649734497070312
  
Result for objective_18fbb20a:
  date: 2022-07-22_15-46-48
  done: false
  experiment_id: 5e399bc125a548739dd433f4153019fe
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 3.5691516291476226
  neg_mean_loss: -3.5691516291476226
  node_ip: 127.0.0.1
  pid: 48475
  time_since_restore: 5.178721189498901
  time_this_iter_s: 0.10647201538085938
  time_total_s: 5.178721189498901
  timestamp: 1658501208
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 18fbb20a
  warmup_time: 0.003000974655151367
  
Result for objective_190ab4d0:
  date: 2022-07-22_15-46-48
  done: false
  experiment_id: 1b007a40cf6d425085379a4314739912
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 3.9678564240341583
  neg_mean_loss: -3.9678564240341583
  node_ip: 127.0.0.1
  pid: 48478
  time_since_restore: 5.156033754348755
  time_this_iter_s: 0.10652899742126465
  time_total_s: 5.156033754348755
  timestamp: 1658501208
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 190ab4d0
  warmup_time: 0.0028760433197021484
  
Result for objective_191ca3de:
  date: 2022-07-22_15-46-48
  done: false
  experiment_id: e3e72bbeb8ae46f0a4f1624db62bc730
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: -1.04293599865772
  neg_mean_loss: 1.04293599865772
  node_ip: 127.0.0.1
  pid: 48481
  time_since_restore: 5.1839048862457275
  time_this_iter_s: 0.10760998725891113
  time_total_s: 5.1839048862457275
  timestamp: 1658501208
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 191ca3de
  warmup_time: 0.0029621124267578125
  
Result for objective_17033b12:
  date: 2022-07-22_15-46-50
  done: false
  experiment_id: b5f76922e6424055b6aa255677e2f4b8
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: -8.832083655687075
  neg_mean_loss: 8.832083655687075
  node_ip: 127.0.0.1
  pid: 48470
  time_since_restore: 10.193100929260254
  time_this_iter_s: 0.10694503784179688
  time_total_s: 10.193100929260254
  timestamp: 1658501210
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: 17033b12
  warmup_time: 0.0029649734497070312
  
Result for objective_17033b12:
  date: 2022-07-22_15-46-50
  done: true
  experiment_id: b5f76922e6424055b6aa255677e2f4b8
  experiment_tag: 5_activation=tanh,height=-89.1891,steps=100,width=12.1463
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -8.836432551723352
  neg_mean_loss: 8.836432551723352
  node_ip: 127.0.0.1
  pid: 48470
  time_since_restore: 10.725535869598389
  time_this_iter_s: 0.10705089569091797
  time_total_s: 10.725535869598389
  timestamp: 1658501210
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 17033b12
  warmup_time: 0.0029649734497070312
  
Result for objective_1eec012e:
  date: 2022-07-22_15-46-53
  done: false
  experiment_id: a3cab04eb23543ddbf9b1daa9e915851
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 4.9718091604083385
  neg_mean_loss: -4.9718091604083385
  node_ip: 127.0.0.1
  pid: 48494
  time_since_restore: 0.10102462768554688
  time_this_iter_s: 0.10102462768554688
  time_total_s: 0.10102462768554688
  timestamp: 1658501213
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 1eec012e
  warmup_time: 0.002748250961303711
  
Result for objective_18fbb20a:
  date: 2022-07-22_15-46-53
  done: false
  experiment_id: 5e399bc125a548739dd433f4153019fe
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: 3.4635693481489094
  neg_mean_loss: -3.4635693481489094
  node_ip: 127.0.0.1
  pid: 48475
  time_since_restore: 10.22235107421875
  time_this_iter_s: 0.10835385322570801
  time_total_s: 10.22235107421875
  timestamp: 1658501213
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: 18fbb20a
  warmup_time: 0.003000974655151367
  
Result for objective_190ab4d0:
  date: 2022-07-22_15-46-53
  done: false
  experiment_id: 1b007a40cf6d425085379a4314739912
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: 3.732496287450277
  neg_mean_loss: -3.732496287450277
  node_ip: 127.0.0.1
  pid: 48478
  time_since_restore: 10.17775583267212
  time_this_iter_s: 0.10712075233459473
  time_total_s: 10.17775583267212
  timestamp: 1658501213
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: 190ab4d0
  warmup_time: 0.0028760433197021484
  
Result for objective_191ca3de:
  date: 2022-07-22_15-46-54
  done: false
  experiment_id: e3e72bbeb8ae46f0a4f1624db62bc730
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: -1.193549900472242
  neg_mean_loss: 1.193549900472242
  node_ip: 127.0.0.1
  pid: 48481
  time_since_restore: 10.230173826217651
  time_this_iter_s: 0.10641312599182129
  time_total_s: 10.230173826217651
  timestamp: 1658501214
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: 191ca3de
  warmup_time: 0.0029621124267578125
  
Result for objective_18fbb20a:
  date: 2022-07-22_15-46-54
  done: true
  experiment_id: 5e399bc125a548739dd433f4153019fe
  experiment_tag: 6_activation=tanh,height=33.5568,steps=100,width=9.7544
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 3.4581764420917516
  neg_mean_loss: -3.4581764420917516
  node_ip: 127.0.0.1
  pid: 48475
  time_since_restore: 10.756349325180054
  time_this_iter_s: 0.10465312004089355
  time_total_s: 10.756349325180054
  timestamp: 1658501214
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 18fbb20a
  warmup_time: 0.003000974655151367
  
Result for objective_190ab4d0:
  date: 2022-07-22_15-46-54
  done: true
  experiment_id: 1b007a40cf6d425085379a4314739912
  experiment_tag: 7_activation=relu,height=-65.1480,steps=100,width=4.1955
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 3.7203002216299694
  neg_mean_loss: -3.7203002216299694
  node_ip: 127.0.0.1
  pid: 48478
  time_since_restore: 10.712332725524902
  time_this_iter_s: 0.10630083084106445
  time_total_s: 10.712332725524902
  timestamp: 1658501214
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 190ab4d0
  warmup_time: 0.0028760433197021484
  
Result for objective_20e93604:
  date: 2022-07-22_15-46-56
  done: false
  experiment_id: e2c12d5518de42acb5f8413171242ef8
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 1.0094856921511255
  neg_mean_loss: -1.0094856921511255
  node_ip: 127.0.0.1
  pid: 48502
  time_since_restore: 0.10361671447753906
  time_this_iter_s: 0.10361671447753906
  time_total_s: 0.10361671447753906
  timestamp: 1658501216
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 20e93604
  warmup_time: 0.0035440921783447266
  
Result for objective_191ca3de:
  date: 2022-07-22_15-46-57
  done: true
  experiment_id: e3e72bbeb8ae46f0a4f1624db62bc730
  experiment_tag: 8_activation=tanh,height=-13.4892,steps=100,width=6.7408
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -1.2012809142667369
  neg_mean_loss: 1.2012809142667369
  node_ip: 127.0.0.1
  pid: 48481
  time_since_restore: 13.218971729278564
  time_this_iter_s: 2.560670852661133
  time_total_s: 13.218971729278564
  timestamp: 1658501217
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 191ca3de
  warmup_time: 0.0029621124267578125
  
Result for objective_1eec012e:
  date: 2022-07-22_15-46-58
  done: false
  experiment_id: a3cab04eb23543ddbf9b1daa9e915851
  hostname: Kais-MacBook-Pro.local
  iterations: 24
  iterations_since_restore: 25
  mean_loss: -4.754884423931921
  neg_mean_loss: 4.754884423931921
  node_ip: 127.0.0.1
  pid: 48494
  time_since_restore: 5.151007890701294
  time_this_iter_s: 0.1075141429901123
  time_total_s: 5.151007890701294
  timestamp: 1658501218
  timesteps_since_restore: 0
  training_iteration: 25
  trial_id: 1eec012e
  warmup_time: 0.002748250961303711
  
Result for objective_20e93604:
  date: 2022-07-22_15-47-02
  done: false
  experiment_id: e2c12d5518de42acb5f8413171242ef8
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: -8.738995488800374
  neg_mean_loss: 8.738995488800374
  node_ip: 127.0.0.1
  pid: 48502
  time_since_restore: 5.183563709259033
  time_this_iter_s: 0.10645675659179688
  time_total_s: 5.183563709259033
  timestamp: 1658501222
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 20e93604
  warmup_time: 0.0035440921783447266
  
Result for objective_1eec012e:
  date: 2022-07-22_15-47-03
  done: false
  experiment_id: a3cab04eb23543ddbf9b1daa9e915851
  hostname: Kais-MacBook-Pro.local
  iterations: 71
  iterations_since_restore: 72
  mean_loss: -4.9341033303033965
  neg_mean_loss: 4.9341033303033965
  node_ip: 127.0.0.1
  pid: 48494
  time_since_restore: 10.200856685638428
  time_this_iter_s: 0.10717487335205078
  time_total_s: 10.200856685638428
  timestamp: 1658501223
  timesteps_since_restore: 0
  training_iteration: 72
  trial_id: 1eec012e
  warmup_time: 0.002748250961303711
  
Result for objective_1eec012e:
  date: 2022-07-22_15-47-06
  done: true
  experiment_id: a3cab04eb23543ddbf9b1daa9e915851
  experiment_tag: 9_activation=tanh,height=-50.2819,steps=100,width=14.8287
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -4.960533899755889
  neg_mean_loss: 4.960533899755889
  node_ip: 127.0.0.1
  pid: 48494
  time_since_restore: 13.22315764427185
  time_this_iter_s: 0.10771775245666504
  time_total_s: 13.22315764427185
  timestamp: 1658501226
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 1eec012e
  warmup_time: 0.002748250961303711
  
Result for objective_20e93604:
  date: 2022-07-22_15-47-07
  done: false
  experiment_id: e2c12d5518de42acb5f8413171242ef8
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: -8.863153212712554
  neg_mean_loss: 8.863153212712554
  node_ip: 127.0.0.1
  pid: 48502
  time_since_restore: 10.251462936401367
  time_this_iter_s: 0.1052391529083252
  time_total_s: 10.251462936401367
  timestamp: 1658501227
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: 20e93604
  warmup_time: 0.0035440921783447266
  
Result for objective_20e93604:
  date: 2022-07-22_15-47-07
  done: true
  experiment_id: e2c12d5518de42acb5f8413171242ef8
  experiment_tag: 10_activation=tanh,height=-89.9051,steps=100,width=8.2465
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -8.869507755259557
  neg_mean_loss: 8.869507755259557
  node_ip: 127.0.0.1
  pid: 48502
  time_since_restore: 10.792518854141235
  time_this_iter_s: 0.10802984237670898
  time_total_s: 10.792518854141235
  timestamp: 1658501227
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 20e93604
  warmup_time: 0.0035440921783447266
  

We now have hyperparameters found to minimize the mean loss.

print("Best hyperparameters found were: ", results.get_best_result().config)
Best hyperparameters found were:  {'steps': 100, 'width': 8.246480084328711, 'height': -89.90514307848873, 'activation': 'tanh'}

Providing an initial set of hyperparameters

While defining the search algorithm, we may choose to provide an initial set of hyperparameters that we believe are especially promising or informative, and pass this information as a helpful starting point for the SkOptSearch object. We also can pass the known rewards for these initial params to save on unnecessary computation.

initial_params = [
    {"width": 10, "height": 0, "activation": "relu"},
    {"width": 15, "height": -20, "activation": "tanh"}
]
known_rewards = [-189, -1144]

Now the search_alg built using SkOptSearch takes points_to_evaluate.

algo = SkOptSearch(points_to_evaluate=initial_params)
algo = ConcurrencyLimiter(algo, max_concurrent=4)

And again run the experiment, this time with initial hyperparameter evaluations:

tuner = tune.Tuner(
    objective,
    tune_config=tune.TuneConfig(
        metric="mean_loss",
        mode="min",
        search_alg=algo,
        num_samples=num_samples,
    ),
    param_space=search_space,
)
results = tuner.fit()
== Status ==
Current time: 2022-07-22 15:49:08 (running for 00:00:43.84)
Memory usage on this node: 10.5/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.42 GiB heap, 0.0/2.0 GiB objects
Current best trial: 60e6657e with mean_loss=-6.564839315448637 and parameters={'steps': 100, 'width': 11.45060059898019, 'height': -66.52281754112352, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-48-24
Number of trials: 10/10 (10 TERMINATED)
Trial name status loc activation height width loss iter total time (s) iterations neg_mean_loss
objective_56a79fceTERMINATED127.0.0.1:48590relu 0 10 10.1 100 10.9393 99 -10.1
objective_583ee2caTERMINATED127.0.0.1:48605tanh -20 15 -1.93311 100 11.6366 99 1.93311
objective_58405952TERMINATED127.0.0.1:48606relu -87.802519.608 1.271 100 11.752 99 -1.271
objective_5841ebe6TERMINATED127.0.0.1:48607relu 49.51 15.0894 15.0175 100 11.6761 99 -15.0175
objective_5ec4da6eTERMINATED127.0.0.1:48629relu 76.083713.7995 17.681 100 10.7667 99 -17.681
objective_60d7fb56TERMINATED127.0.0.1:48634relu -50.453416.3133 5.0162 100 10.75 99 -5.0162
objective_60e6657eTERMINATED127.0.0.1:48637tanh -66.522811.4506 -6.56484 100 10.759 99 6.56484
objective_60f84e42TERMINATED127.0.0.1:48640tanh 99.5965 9.0442510.0701 100 10.71 99 -10.0701
objective_66bdb9e8TERMINATED127.0.0.1:48653tanh -61.954712.3263 -6.11419 100 10.724 99 6.11419
objective_68c346b8TERMINATED127.0.0.1:48660tanh 23.7916 2.108 2.83643 100 10.7375 99 -2.83643


Result for objective_56a79fce:
  date: 2022-07-22_15-48-27
  done: false
  experiment_id: 7046e554c1db465bbed80331b034b40f
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 20.0
  neg_mean_loss: -20.0
  node_ip: 127.0.0.1
  pid: 48590
  time_since_restore: 0.10438704490661621
  time_this_iter_s: 0.10438704490661621
  time_total_s: 0.10438704490661621
  timestamp: 1658501307
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 56a79fce
  warmup_time: 0.002905130386352539
  
Result for objective_583ee2ca:
  date: 2022-07-22_15-48-30
  done: false
  experiment_id: 59c76129c7074e41ab39b330d898ba03
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 8.0
  neg_mean_loss: -8.0
  node_ip: 127.0.0.1
  pid: 48605
  time_since_restore: 0.10483098030090332
  time_this_iter_s: 0.10483098030090332
  time_total_s: 0.10483098030090332
  timestamp: 1658501310
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 583ee2ca
  warmup_time: 0.0031731128692626953
  
Result for objective_5841ebe6:
  date: 2022-07-22_15-48-30
  done: false
  experiment_id: fde2877ded13416195ab1af070eda91a
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 24.951000429599397
  neg_mean_loss: -24.951000429599397
  node_ip: 127.0.0.1
  pid: 48607
  time_since_restore: 0.10440397262573242
  time_this_iter_s: 0.10440397262573242
  time_total_s: 0.10440397262573242
  timestamp: 1658501310
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 5841ebe6
  warmup_time: 0.0033330917358398438
  
Result for objective_58405952:
  date: 2022-07-22_15-48-30
  done: false
  experiment_id: 5902a2d532b3437e8b94d02919e1604c
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 11.21975075417543
  neg_mean_loss: -11.21975075417543
  node_ip: 127.0.0.1
  pid: 48606
  time_since_restore: 0.10400986671447754
  time_this_iter_s: 0.10400986671447754
  time_total_s: 0.10400986671447754
  timestamp: 1658501310
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: '58405952'
  warmup_time: 0.0027670860290527344
  
Result for objective_56a79fce:
  date: 2022-07-22_15-48-32
  done: false
  experiment_id: 7046e554c1db465bbed80331b034b40f
  hostname: Kais-MacBook-Pro.local
  iterations: 45
  iterations_since_restore: 46
  mean_loss: 10.217391304347826
  neg_mean_loss: -10.217391304347826
  node_ip: 127.0.0.1
  pid: 48590
  time_since_restore: 5.135921239852905
  time_this_iter_s: 0.10727715492248535
  time_total_s: 5.135921239852905
  timestamp: 1658501312
  timesteps_since_restore: 0
  training_iteration: 46
  trial_id: 56a79fce
  warmup_time: 0.002905130386352539
  
Result for objective_583ee2ca:
  date: 2022-07-22_15-48-35
  done: false
  experiment_id: 59c76129c7074e41ab39b330d898ba03
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: -1.8601398601398602
  neg_mean_loss: 1.8601398601398602
  node_ip: 127.0.0.1
  pid: 48605
  time_since_restore: 5.154622793197632
  time_this_iter_s: 0.10759091377258301
  time_total_s: 5.154622793197632
  timestamp: 1658501315
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 583ee2ca
  warmup_time: 0.0031731128692626953
  
Result for objective_5841ebe6:
  date: 2022-07-22_15-48-35
  done: false
  experiment_id: fde2877ded13416195ab1af070eda91a
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 15.090043780907216
  neg_mean_loss: -15.090043780907216
  node_ip: 127.0.0.1
  pid: 48607
  time_since_restore: 5.153692960739136
  time_this_iter_s: 0.10715007781982422
  time_total_s: 5.153692960739136
  timestamp: 1658501315
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 5841ebe6
  warmup_time: 0.0033330917358398438
  
Result for objective_58405952:
  date: 2022-07-22_15-48-35
  done: false
  experiment_id: 5902a2d532b3437e8b94d02919e1604c
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 1.3270955262548618
  neg_mean_loss: -1.3270955262548618
  node_ip: 127.0.0.1
  pid: 48606
  time_since_restore: 5.145922660827637
  time_this_iter_s: 0.10549187660217285
  time_total_s: 5.145922660827637
  timestamp: 1658501315
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: '58405952'
  warmup_time: 0.0027670860290527344
  
Result for objective_56a79fce:
  date: 2022-07-22_15-48-37
  done: false
  experiment_id: 7046e554c1db465bbed80331b034b40f
  hostname: Kais-MacBook-Pro.local
  iterations: 92
  iterations_since_restore: 93
  mean_loss: 10.10752688172043
  neg_mean_loss: -10.10752688172043
  node_ip: 127.0.0.1
  pid: 48590
  time_since_restore: 10.195838212966919
  time_this_iter_s: 0.10654616355895996
  time_total_s: 10.195838212966919
  timestamp: 1658501317
  timesteps_since_restore: 0
  training_iteration: 93
  trial_id: 56a79fce
  warmup_time: 0.002905130386352539
  
Result for objective_56a79fce:
  date: 2022-07-22_15-48-38
  done: true
  experiment_id: 7046e554c1db465bbed80331b034b40f
  experiment_tag: 1_activation=relu,height=0,steps=100,width=10
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 10.1
  neg_mean_loss: -10.1
  node_ip: 127.0.0.1
  pid: 48590
  time_since_restore: 10.939304113388062
  time_this_iter_s: 0.10537099838256836
  time_total_s: 10.939304113388062
  timestamp: 1658501318
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 56a79fce
  warmup_time: 0.002905130386352539
  
Result for objective_5841ebe6:
  date: 2022-07-22_15-48-39
  done: false
  experiment_id: fde2877ded13416195ab1af070eda91a
  hostname: Kais-MacBook-Pro.local
  iterations: 90
  iterations_since_restore: 91
  mean_loss: 15.024097555037425
  neg_mean_loss: -15.024097555037425
  node_ip: 127.0.0.1
  pid: 48607
  time_since_restore: 9.817665100097656
  time_this_iter_s: 0.10596418380737305
  time_total_s: 9.817665100097656
  timestamp: 1658501319
  timesteps_since_restore: 0
  training_iteration: 91
  trial_id: 5841ebe6
  warmup_time: 0.0033330917358398438
  
Result for objective_58405952:
  date: 2022-07-22_15-48-39
  done: false
  experiment_id: 5902a2d532b3437e8b94d02919e1604c
  hostname: Kais-MacBook-Pro.local
  iterations: 89
  iterations_since_restore: 90
  mean_loss: 1.2767270681910414
  neg_mean_loss: -1.2767270681910414
  node_ip: 127.0.0.1
  pid: 48606
  time_since_restore: 9.72554087638855
  time_this_iter_s: 0.10558199882507324
  time_total_s: 9.72554087638855
  timestamp: 1658501319
  timesteps_since_restore: 0
  training_iteration: 90
  trial_id: '58405952'
  warmup_time: 0.0027670860290527344
  
Result for objective_583ee2ca:
  date: 2022-07-22_15-48-39
  done: false
  experiment_id: 59c76129c7074e41ab39b330d898ba03
  hostname: Kais-MacBook-Pro.local
  iterations: 91
  iterations_since_restore: 92
  mean_loss: -1.9272727272727272
  neg_mean_loss: 1.9272727272727272
  node_ip: 127.0.0.1
  pid: 48605
  time_since_restore: 9.925616025924683
  time_this_iter_s: 0.1071469783782959
  time_total_s: 9.925616025924683
  timestamp: 1658501319
  timesteps_since_restore: 0
  training_iteration: 92
  trial_id: 583ee2ca
  warmup_time: 0.0031731128692626953
  
Result for objective_5ec4da6e:
  date: 2022-07-22_15-48-40
  done: false
  experiment_id: 24cf29dbb0e44fa7a2dbffe353a56c35
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 27.60836685236658
  neg_mean_loss: -27.60836685236658
  node_ip: 127.0.0.1
  pid: 48629
  time_since_restore: 0.10513186454772949
  time_this_iter_s: 0.10513186454772949
  time_total_s: 0.10513186454772949
  timestamp: 1658501320
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 5ec4da6e
  warmup_time: 0.002888917922973633
  
Result for objective_583ee2ca:
  date: 2022-07-22_15-48-41
  done: true
  experiment_id: 59c76129c7074e41ab39b330d898ba03
  experiment_tag: 2_activation=tanh,height=-20,steps=100,width=15
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -1.9331103678929766
  neg_mean_loss: 1.9331103678929766
  node_ip: 127.0.0.1
  pid: 48605
  time_since_restore: 11.636637926101685
  time_this_iter_s: 0.10811305046081543
  time_total_s: 11.636637926101685
  timestamp: 1658501321
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 583ee2ca
  warmup_time: 0.0031731128692626953
  
Result for objective_5841ebe6:
  date: 2022-07-22_15-48-41
  done: true
  experiment_id: fde2877ded13416195ab1af070eda91a
  experiment_tag: 4_activation=relu,height=49.5100,steps=100,width=15.0894
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 15.017496549772488
  neg_mean_loss: -15.017496549772488
  node_ip: 127.0.0.1
  pid: 48607
  time_since_restore: 11.676102876663208
  time_this_iter_s: 0.1074519157409668
  time_total_s: 11.676102876663208
  timestamp: 1658501321
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 5841ebe6
  warmup_time: 0.0033330917358398438
  
Result for objective_58405952:
  date: 2022-07-22_15-48-41
  done: true
  experiment_id: 5902a2d532b3437e8b94d02919e1604c
  experiment_tag: 3_activation=relu,height=-87.8025,steps=100,width=19.6080
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 1.271001380631752
  neg_mean_loss: -1.271001380631752
  node_ip: 127.0.0.1
  pid: 48606
  time_since_restore: 11.751991033554077
  time_this_iter_s: 0.1101231575012207
  time_total_s: 11.751991033554077
  timestamp: 1658501321
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: '58405952'
  warmup_time: 0.0027670860290527344
  
Result for objective_60d7fb56:
  date: 2022-07-22_15-48-44
  done: false
  experiment_id: 26d42c9dcc6e4354b3fa7355fc806522
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 14.954664969255782
  neg_mean_loss: -14.954664969255782
  node_ip: 127.0.0.1
  pid: 48634
  time_since_restore: 0.10394525527954102
  time_this_iter_s: 0.10394525527954102
  time_total_s: 0.10394525527954102
  timestamp: 1658501324
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 60d7fb56
  warmup_time: 0.003077268600463867
  
Result for objective_60e6657e:
  date: 2022-07-22_15-48-44
  done: false
  experiment_id: 3f2014eb8b41451bb277cabed9951d90
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 3.3477182458876484
  neg_mean_loss: -3.3477182458876484
  node_ip: 127.0.0.1
  pid: 48637
  time_since_restore: 0.10410618782043457
  time_this_iter_s: 0.10410618782043457
  time_total_s: 0.10410618782043457
  timestamp: 1658501324
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 60e6657e
  warmup_time: 0.0029239654541015625
  
Result for objective_60f84e42:
  date: 2022-07-22_15-48-44
  done: false
  experiment_id: 481882ef37194e6f995c4294a3453fd2
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 19.959649161689548
  neg_mean_loss: -19.959649161689548
  node_ip: 127.0.0.1
  pid: 48640
  time_since_restore: 0.10413289070129395
  time_this_iter_s: 0.10413289070129395
  time_total_s: 0.10413289070129395
  timestamp: 1658501324
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 60f84e42
  warmup_time: 0.0028841495513916016
  
Result for objective_5ec4da6e:
  date: 2022-07-22_15-48-45
  done: false
  experiment_id: 24cf29dbb0e44fa7a2dbffe353a56c35
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 17.760209862707853
  neg_mean_loss: -17.760209862707853
  node_ip: 127.0.0.1
  pid: 48629
  time_since_restore: 5.151746988296509
  time_this_iter_s: 0.10759496688842773
  time_total_s: 5.151746988296509
  timestamp: 1658501325
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 5ec4da6e
  warmup_time: 0.002888917922973633
  
Result for objective_60d7fb56:
  date: 2022-07-22_15-48-49
  done: false
  experiment_id: 26d42c9dcc6e4354b3fa7355fc806522
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 5.08341058857216
  neg_mean_loss: -5.08341058857216
  node_ip: 127.0.0.1
  pid: 48634
  time_since_restore: 5.146520137786865
  time_this_iter_s: 0.10820508003234863
  time_total_s: 5.146520137786865
  timestamp: 1658501329
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 60d7fb56
  warmup_time: 0.003077268600463867
  
Result for objective_60e6657e:
  date: 2022-07-22_15-48-49
  done: false
  experiment_id: 3f2014eb8b41451bb277cabed9951d90
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: -6.469859332234883
  neg_mean_loss: 6.469859332234883
  node_ip: 127.0.0.1
  pid: 48637
  time_since_restore: 5.153999090194702
  time_this_iter_s: 0.10596871376037598
  time_total_s: 5.153999090194702
  timestamp: 1658501329
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 60e6657e
  warmup_time: 0.0029239654541015625
  
Result for objective_60f84e42:
  date: 2022-07-22_15-48-49
  done: false
  experiment_id: 481882ef37194e6f995c4294a3453fd2
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 10.189492101039557
  neg_mean_loss: -10.189492101039557
  node_ip: 127.0.0.1
  pid: 48640
  time_since_restore: 5.137938976287842
  time_this_iter_s: 0.10651087760925293
  time_total_s: 5.137938976287842
  timestamp: 1658501329
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 60f84e42
  warmup_time: 0.0028841495513916016
  
Result for objective_5ec4da6e:
  date: 2022-07-22_15-48-50
  done: false
  experiment_id: 24cf29dbb0e44fa7a2dbffe353a56c35
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: 17.684869174683147
  neg_mean_loss: -17.684869174683147
  node_ip: 127.0.0.1
  pid: 48629
  time_since_restore: 10.229070901870728
  time_this_iter_s: 0.11342406272888184
  time_total_s: 10.229070901870728
  timestamp: 1658501330
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: 5ec4da6e
  warmup_time: 0.002888917922973633
  
Result for objective_5ec4da6e:
  date: 2022-07-22_15-48-51
  done: true
  experiment_id: 24cf29dbb0e44fa7a2dbffe353a56c35
  experiment_tag: 5_activation=relu,height=76.0837,steps=100,width=13.7995
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 17.6810334976324
  neg_mean_loss: -17.6810334976324
  node_ip: 127.0.0.1
  pid: 48629
  time_since_restore: 10.766729831695557
  time_this_iter_s: 0.10697484016418457
  time_total_s: 10.766729831695557
  timestamp: 1658501331
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 5ec4da6e
  warmup_time: 0.002888917922973633
  
Result for objective_66bdb9e8:
  date: 2022-07-22_15-48-54
  done: false
  experiment_id: fae00fc97db044e9965698562b20039b
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 3.804528837217097
  neg_mean_loss: -3.804528837217097
  node_ip: 127.0.0.1
  pid: 48653
  time_since_restore: 0.10318303108215332
  time_this_iter_s: 0.10318303108215332
  time_total_s: 0.10318303108215332
  timestamp: 1658501334
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 66bdb9e8
  warmup_time: 0.004212856292724609
  
Result for objective_60d7fb56:
  date: 2022-07-22_15-48-54
  done: false
  experiment_id: 26d42c9dcc6e4354b3fa7355fc806522
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: 5.019454849577342
  neg_mean_loss: -5.019454849577342
  node_ip: 127.0.0.1
  pid: 48634
  time_since_restore: 10.212354183197021
  time_this_iter_s: 0.10782504081726074
  time_total_s: 10.212354183197021
  timestamp: 1658501334
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: 60d7fb56
  warmup_time: 0.003077268600463867
  
Result for objective_60e6657e:
  date: 2022-07-22_15-48-54
  done: false
  experiment_id: 3f2014eb8b41451bb277cabed9951d90
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: -6.560230936519563
  neg_mean_loss: 6.560230936519563
  node_ip: 127.0.0.1
  pid: 48637
  time_since_restore: 10.219763278961182
  time_this_iter_s: 0.10767722129821777
  time_total_s: 10.219763278961182
  timestamp: 1658501334
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: 60e6657e
  warmup_time: 0.0029239654541015625
  
Result for objective_60f84e42:
  date: 2022-07-22_15-48-54
  done: false
  experiment_id: 481882ef37194e6f995c4294a3453fd2
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: 10.075906679849322
  neg_mean_loss: -10.075906679849322
  node_ip: 127.0.0.1
  pid: 48640
  time_since_restore: 10.16911792755127
  time_this_iter_s: 0.10426807403564453
  time_total_s: 10.16911792755127
  timestamp: 1658501334
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: 60f84e42
  warmup_time: 0.0028841495513916016
  
Result for objective_60d7fb56:
  date: 2022-07-22_15-48-54
  done: true
  experiment_id: 26d42c9dcc6e4354b3fa7355fc806522
  experiment_tag: 6_activation=relu,height=-50.4534,steps=100,width=16.3133
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 5.016202769898167
  neg_mean_loss: -5.016202769898167
  node_ip: 127.0.0.1
  pid: 48634
  time_since_restore: 10.75002908706665
  time_this_iter_s: 0.10717272758483887
  time_total_s: 10.75002908706665
  timestamp: 1658501334
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 60d7fb56
  warmup_time: 0.003077268600463867
  
Result for objective_60e6657e:
  date: 2022-07-22_15-48-54
  done: true
  experiment_id: 3f2014eb8b41451bb277cabed9951d90
  experiment_tag: 7_activation=tanh,height=-66.5228,steps=100,width=11.4506
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -6.564839315448637
  neg_mean_loss: 6.564839315448637
  node_ip: 127.0.0.1
  pid: 48637
  time_since_restore: 10.759015321731567
  time_this_iter_s: 0.10623526573181152
  time_total_s: 10.759015321731567
  timestamp: 1658501334
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 60e6657e
  warmup_time: 0.0029239654541015625
  
Result for objective_60f84e42:
  date: 2022-07-22_15-48-55
  done: true
  experiment_id: 481882ef37194e6f995c4294a3453fd2
  experiment_tag: 8_activation=tanh,height=99.5965,steps=100,width=9.0442
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 10.070099940211907
  neg_mean_loss: -10.070099940211907
  node_ip: 127.0.0.1
  pid: 48640
  time_since_restore: 10.710038900375366
  time_this_iter_s: 0.1064748764038086
  time_total_s: 10.710038900375366
  timestamp: 1658501335
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 60f84e42
  warmup_time: 0.0028841495513916016
  
Result for objective_68c346b8:
  date: 2022-07-22_15-48-57
  done: false
  experiment_id: fe241f21de6a4466bcdf713da6d4cd92
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 12.37916148081603
  neg_mean_loss: -12.37916148081603
  node_ip: 127.0.0.1
  pid: 48660
  time_since_restore: 0.10358977317810059
  time_this_iter_s: 0.10358977317810059
  time_total_s: 0.10358977317810059
  timestamp: 1658501337
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 68c346b8
  warmup_time: 0.0028688907623291016
  
Result for objective_66bdb9e8:
  date: 2022-07-22_15-48-59
  done: false
  experiment_id: fae00fc97db044e9965698562b20039b
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: -6.025789125550826
  neg_mean_loss: 6.025789125550826
  node_ip: 127.0.0.1
  pid: 48653
  time_since_restore: 5.1440629959106445
  time_this_iter_s: 0.10767483711242676
  time_total_s: 5.1440629959106445
  timestamp: 1658501339
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 66bdb9e8
  warmup_time: 0.004212856292724609
  
Result for objective_68c346b8:
  date: 2022-07-22_15-49-02
  done: false
  experiment_id: fe241f21de6a4466bcdf713da6d4cd92
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 3.295952835154158
  neg_mean_loss: -3.295952835154158
  node_ip: 127.0.0.1
  pid: 48660
  time_since_restore: 5.15015983581543
  time_this_iter_s: 0.10423922538757324
  time_total_s: 5.15015983581543
  timestamp: 1658501342
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 68c346b8
  warmup_time: 0.0028688907623291016
  
Result for objective_66bdb9e8:
  date: 2022-07-22_15-49-04
  done: false
  experiment_id: fae00fc97db044e9965698562b20039b
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: -6.109904185213174
  neg_mean_loss: 6.109904185213174
  node_ip: 127.0.0.1
  pid: 48653
  time_since_restore: 10.1879301071167
  time_this_iter_s: 0.10626506805419922
  time_total_s: 10.1879301071167
  timestamp: 1658501344
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: 66bdb9e8
  warmup_time: 0.004212856292724609
  
Result for objective_66bdb9e8:
  date: 2022-07-22_15-49-04
  done: true
  experiment_id: fae00fc97db044e9965698562b20039b
  experiment_tag: 9_activation=tanh,height=-61.9547,steps=100,width=12.3263
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -6.114190623827535
  neg_mean_loss: 6.114190623827535
  node_ip: 127.0.0.1
  pid: 48653
  time_since_restore: 10.724038124084473
  time_this_iter_s: 0.1088099479675293
  time_total_s: 10.724038124084473
  timestamp: 1658501344
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 66bdb9e8
  warmup_time: 0.004212856292724609
  
Result for objective_68c346b8:
  date: 2022-07-22_15-49-07
  done: false
  experiment_id: fe241f21de6a4466bcdf713da6d4cd92
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: 2.859579303290837
  neg_mean_loss: -2.859579303290837
  node_ip: 127.0.0.1
  pid: 48660
  time_since_restore: 10.199071884155273
  time_this_iter_s: 0.10729789733886719
  time_total_s: 10.199071884155273
  timestamp: 1658501347
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: 68c346b8
  warmup_time: 0.0028688907623291016
  
Result for objective_68c346b8:
  date: 2022-07-22_15-49-08
  done: true
  experiment_id: fe241f21de6a4466bcdf713da6d4cd92
  experiment_tag: 10_activation=tanh,height=23.7916,steps=100,width=2.1080
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 2.836425260080649
  neg_mean_loss: -2.836425260080649
  node_ip: 127.0.0.1
  pid: 48660
  time_since_restore: 10.7375328540802
  time_this_iter_s: 0.1081850528717041
  time_total_s: 10.7375328540802
  timestamp: 1658501348
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 68c346b8
  warmup_time: 0.0028688907623291016
  

And we again show the ideal hyperparameters.

print("Best hyperparameters found were: ", results.get_best_result().config)
Best hyperparameters found were:  {'steps': 100, 'width': 11.45060059898019, 'height': -66.52281754112352, 'activation': 'tanh'}