Running Tune experiments with Optuna#

In this tutorial we introduce Optuna, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with Optuna and, as a result, allow you to seamlessly scale up a Optuna optimization process - without sacrificing performance.

Similar to Ray Tune, Optuna is an automatic hyperparameter optimization software framework, particularly designed for machine learning. It features an imperative (“how” over “what” emphasis), define-by-run style user API. With Optuna, a user has the ability to dynamically construct the search spaces for the hyperparameters. Optuna falls in the domain of “derivative-free optimization” and “black-box optimization”.

In this example we minimize a simple objective to briefly demonstrate the usage of Optuna with Ray Tune via OptunaSearch, including examples of conditional search spaces (string together relationships between hyperparameters), and the multi-objective problem (measure trade-offs among all important metrics). It’s useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume optuna==2.9.1 library is installed. To learn more, please refer to Optuna website.

Please note that sophisticated schedulers, such as AsyncHyperBandScheduler, may not work correctly with multi-objective optimization, since they typically expect a scalar score to compare fitness among trials.

Click below to see all the imports we need for this example. You can also launch directly into a Binder instance to run this notebook yourself. Just click on the rocket symbol at the top of the navigation.

Let’s start by defining a simple evaluation function. An explicit math formula is queried here for demonstration, yet in practice this is typically a black-box function– e.g. the performance results after training an ML model. We artificially sleep for a bit (0.1 seconds) to simulate a long-running ML experiment. This setup assumes that we’re running multiple steps of an experiment while tuning three hyperparameters, namely width, height, and activation.

def evaluate(step, width, height, activation):
    time.sleep(0.1)
    activation_boost = 10 if activation=="relu" else 0
    return (0.1 + width * step / 100) ** (-1) + height * 0.1 + activation_boost

Next, our objective function to be optimized takes a Tune config, evaluates the score of your experiment in a training loop, and uses train.report to report the score back to Tune.

def objective(config):
    for step in range(config["steps"]):
        score = evaluate(step, config["width"], config["height"], config["activation"])
        train.report({"iterations": step, "mean_loss": score})

Next we define a search space. The critical assumption is that the optimal hyperparamters live within this space. Yet, if the space is very large, then those hyperparamters may be difficult to find in a short amount of time.

The simplest case is a search space with independent dimensions. In this case, a config dictionary will suffice.

search_space = {
    "steps": 100,
    "width": tune.uniform(0, 20),
    "height": tune.uniform(-100, 100),
    "activation": tune.choice(["relu", "tanh"]),
}

Here we define the Optuna search algorithm:

algo = OptunaSearch()

We also constrain the number of concurrent trials to 4 with a ConcurrencyLimiter.

algo = ConcurrencyLimiter(algo, max_concurrent=4)

The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to 1000 samples. (you can decrease this if it takes too long on your machine).

num_samples = 1000

Finally, we run the experiment to "min"imize the “mean_loss” of the objective by searching search_space via algo, num_samples times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute tuner.fit().

tuner = tune.Tuner(
    objective,
    tune_config=tune.TuneConfig(
        metric="mean_loss",
        mode="min",
        search_alg=algo,
        num_samples=num_samples,
    ),
    param_space=search_space,
)
results = tuner.fit()

[I 2022-07-22 15:21:47,769] A new study created in memory with name: optuna

== Status ==
Current time: 2022-07-22 15:22:32 (running for 00:00:43.89)
Memory usage on this node: 10.1/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects
Current best trial: b1bad2fe with mean_loss=-1.5166144695235813 and parameters={'steps': 100, 'width': 7.688465886501621, 'height': -16.46289560854555, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/optuna_example
Number of trials: 10/10 (10 TERMINATED)

Trial name	status	loc	activation	height	width	loss	iter	total time (s)	iterations	neg_mean_loss
objective_9f689668	TERMINATED	127.0.0.1:46141	tanh	32.9806	16.0406	3.36064	100	11.0409	99	-3.36064
objective_a11d2104	TERMINATED	127.0.0.1:46149	relu	72.2627	0.763881	18.3942	100	11.7018	99	-18.3942
objective_a11eef8e	TERMINATED	127.0.0.1:46150	relu	-32.8474	13.1144	6.79169	100	11.8071	99	-6.79169
objective_a1209e56	TERMINATED	127.0.0.1:46151	tanh	75.6408	10.4415	7.65989	100	11.6987	99	-7.65989
objective_a7b383e6	TERMINATED	127.0.0.1:46174	relu	50.1501	3.4612	15.2986	100	10.7015	99	-15.2986
objective_a9c844c8	TERMINATED	127.0.0.1:46183	relu	-40.3931	14.0525	6.03205	100	10.6687	99	-6.03205
objective_a9cb308e	TERMINATED	127.0.0.1:46184	tanh	48.4802	12.7746	4.92647	100	10.7122	99	-4.92647
objective_a9d8332e	TERMINATED	127.0.0.1:46189	tanh	-1.26682	10.9788	-0.0355157	100	10.7089	99	0.0355157
objective_af948a1a	TERMINATED	127.0.0.1:46204	tanh	61.6381	11.0105	6.25472	100	10.744	99	-6.25472
objective_b1bad2fe	TERMINATED	127.0.0.1:46209	tanh	-16.4629	7.68847	-1.51661	100	10.7274	99	1.51661

Result for objective_9f689668:
  date: 2022-07-22_15-21-51
  done: false
  experiment_id: 7d29e5c95aa44d77becc24b23a326b9b
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 13.298063134872326
  neg_mean_loss: -13.298063134872326
  node_ip: 127.0.0.1
  pid: 46141
  time_since_restore: 0.10468196868896484
  time_this_iter_s: 0.10468196868896484
  time_total_s: 0.10468196868896484
  timestamp: 1658499711
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 9f689668
  warmup_time: 0.003930091857910156
  
Result for objective_a1209e56:
  date: 2022-07-22_15-21-54
  done: false
  experiment_id: 70d9788c327c43fb94450e5084416b81
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 17.564083106485064
  neg_mean_loss: -17.564083106485064
  node_ip: 127.0.0.1
  pid: 46151
  time_since_restore: 0.10416483879089355
  time_this_iter_s: 0.10416483879089355
  time_total_s: 0.10416483879089355
  timestamp: 1658499714
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: a1209e56
  warmup_time: 0.0026938915252685547
  
Result for objective_a11d2104:
  date: 2022-07-22_15-21-54
  done: false
  experiment_id: af2598364a004cd28f37ede19afd1ffb
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 27.2262748487915
  neg_mean_loss: -27.2262748487915
  node_ip: 127.0.0.1
  pid: 46149
  time_since_restore: 0.10507392883300781
  time_this_iter_s: 0.10507392883300781
  time_total_s: 0.10507392883300781
  timestamp: 1658499714
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: a11d2104
  warmup_time: 0.0031232833862304688
  
Result for objective_a11eef8e:
  date: 2022-07-22_15-21-54
  done: false
  experiment_id: e4a5922603d44b459c23607a42d5e574
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 16.71525618736072
  neg_mean_loss: -16.71525618736072
  node_ip: 127.0.0.1
  pid: 46150
  time_since_restore: 0.10363006591796875
  time_this_iter_s: 0.10363006591796875
  time_total_s: 0.10363006591796875
  timestamp: 1658499714
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: a11eef8e
  warmup_time: 0.0028891563415527344
  
Result for objective_9f689668:
  date: 2022-07-22_15-21-56
  done: false
  experiment_id: 7d29e5c95aa44d77becc24b23a326b9b
  hostname: Kais-MacBook-Pro.local
  iterations: 45
  iterations_since_restore: 46
  mean_loss: 3.434707064844201
  neg_mean_loss: -3.434707064844201
  node_ip: 127.0.0.1
  pid: 46141
  time_since_restore: 5.202661752700806
  time_this_iter_s: 0.10527586936950684
  time_total_s: 5.202661752700806
  timestamp: 1658499716
  timesteps_since_restore: 0
  training_iteration: 46
  trial_id: 9f689668
  warmup_time: 0.003930091857910156
  
Result for objective_a1209e56:
  date: 2022-07-22_15-21-59
  done: false
  experiment_id: 70d9788c327c43fb94450e5084416b81
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 7.763782433167051
  neg_mean_loss: -7.763782433167051
  node_ip: 127.0.0.1
  pid: 46151
  time_since_restore: 5.146381139755249
  time_this_iter_s: 0.10691022872924805
  time_total_s: 5.146381139755249
  timestamp: 1658499719
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: a1209e56
  warmup_time: 0.0026938915252685547
  
Result for objective_a11eef8e:
  date: 2022-07-22_15-21-59
  done: false
  experiment_id: e4a5922603d44b459c23607a42d5e574
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 6.874905099790281
  neg_mean_loss: -6.874905099790281
  node_ip: 127.0.0.1
  pid: 46150
  time_since_restore: 5.1618921756744385
  time_this_iter_s: 0.10776615142822266
  time_total_s: 5.1618921756744385
  timestamp: 1658499719
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: a11eef8e
  warmup_time: 0.0028891563415527344
  
Result for objective_a11d2104:
  date: 2022-07-22_15-21-59
  done: false
  experiment_id: af2598364a004cd28f37ede19afd1ffb
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 19.4048102011043
  neg_mean_loss: -19.4048102011043
  node_ip: 127.0.0.1
  pid: 46149
  time_since_restore: 5.170748233795166
  time_this_iter_s: 0.11556220054626465
  time_total_s: 5.170748233795166
  timestamp: 1658499719
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: a11d2104
  warmup_time: 0.0031232833862304688
  
Result for objective_9f689668:
  date: 2022-07-22_15-22-01
  done: false
  experiment_id: 7d29e5c95aa44d77becc24b23a326b9b
  hostname: Kais-MacBook-Pro.local
  iterations: 92
  iterations_since_restore: 93
  mean_loss: 3.3653696881211697
  neg_mean_loss: -3.3653696881211697
  node_ip: 127.0.0.1
  pid: 46141
  time_since_restore: 10.248206853866577
  time_this_iter_s: 0.10745501518249512
  time_total_s: 10.248206853866577
  timestamp: 1658499721
  timesteps_since_restore: 0
  training_iteration: 93
  trial_id: 9f689668
  warmup_time: 0.003930091857910156
  
Result for objective_9f689668:
  date: 2022-07-22_15-22-02
  done: true
  experiment_id: 7d29e5c95aa44d77becc24b23a326b9b
  experiment_tag: 1_activation=tanh,height=32.9806,steps=100,width=16.0406
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 3.3606404197394255
  neg_mean_loss: -3.3606404197394255
  node_ip: 127.0.0.1
  pid: 46141
  time_since_restore: 11.04089903831482
  time_this_iter_s: 0.10824704170227051
  time_total_s: 11.04089903831482
  timestamp: 1658499722
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 9f689668
  warmup_time: 0.003930091857910156
  
Result for objective_a1209e56:
  date: 2022-07-22_15-22-04
  done: false
  experiment_id: 70d9788c327c43fb94450e5084416b81
  hostname: Kais-MacBook-Pro.local
  iterations: 90
  iterations_since_restore: 91
  mean_loss: 7.669375143981996
  neg_mean_loss: -7.669375143981996
  node_ip: 127.0.0.1
  pid: 46151
  time_since_restore: 9.724387168884277
  time_this_iter_s: 0.10670638084411621
  time_total_s: 9.724387168884277
  timestamp: 1658499724
  timesteps_since_restore: 0
  training_iteration: 91
  trial_id: a1209e56
  warmup_time: 0.0026938915252685547
  
Result for objective_a11d2104:
  date: 2022-07-22_15-22-04
  done: false
  experiment_id: af2598364a004cd28f37ede19afd1ffb
  hostname: Kais-MacBook-Pro.local
  iterations: 90
  iterations_since_restore: 91
  mean_loss: 18.49612779999574
  neg_mean_loss: -18.49612779999574
  node_ip: 127.0.0.1
  pid: 46149
  time_since_restore: 9.818917274475098
  time_this_iter_s: 0.10725140571594238
  time_total_s: 9.818917274475098
  timestamp: 1658499724
  timesteps_since_restore: 0
  training_iteration: 91
  trial_id: a11d2104
  warmup_time: 0.0031232833862304688
  
Result for objective_a11eef8e:
  date: 2022-07-22_15-22-03
  done: false
  experiment_id: e4a5922603d44b459c23607a42d5e574
  hostname: Kais-MacBook-Pro.local
  iterations: 89
  iterations_since_restore: 90
  mean_loss: 6.800205168670805
  neg_mean_loss: -6.800205168670805
  node_ip: 127.0.0.1
  pid: 46150
  time_since_restore: 9.714852094650269
  time_this_iter_s: 0.10286402702331543
  time_total_s: 9.714852094650269
  timestamp: 1658499723
  timesteps_since_restore: 0
  training_iteration: 90
  trial_id: a11eef8e
  warmup_time: 0.0028891563415527344
  
Result for objective_a7b383e6:
  date: 2022-07-22_15-22-05
  done: false
  experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 25.015010793208496
  neg_mean_loss: -25.015010793208496
  node_ip: 127.0.0.1
  pid: 46174
  time_since_restore: 0.10384607315063477
  time_this_iter_s: 0.10384607315063477
  time_total_s: 0.10384607315063477
  timestamp: 1658499725
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: a7b383e6
  warmup_time: 0.002817869186401367
  
Result for objective_a1209e56:
  date: 2022-07-22_15-22-05
  done: true
  experiment_id: 70d9788c327c43fb94450e5084416b81
  experiment_tag: 4_activation=tanh,height=75.6408,steps=100,width=10.4415
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 7.659894851608276
  neg_mean_loss: -7.659894851608276
  node_ip: 127.0.0.1
  pid: 46151
  time_since_restore: 11.698678970336914
  time_this_iter_s: 0.10874581336975098
  time_total_s: 11.698678970336914
  timestamp: 1658499725
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: a1209e56
  warmup_time: 0.0026938915252685547
  
Result for objective_a11d2104:
  date: 2022-07-22_15-22-05
  done: true
  experiment_id: af2598364a004cd28f37ede19afd1ffb
  experiment_tag: 2_activation=relu,height=72.2627,steps=100,width=0.7639
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 18.394168929385685
  neg_mean_loss: -18.394168929385685
  node_ip: 127.0.0.1
  pid: 46149
  time_since_restore: 11.701792001724243
  time_this_iter_s: 0.10734391212463379
  time_total_s: 11.701792001724243
  timestamp: 1658499725
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: a11d2104
  warmup_time: 0.0031232833862304688
  
Result for objective_a11eef8e:
  date: 2022-07-22_15-22-06
  done: true
  experiment_id: e4a5922603d44b459c23607a42d5e574
  experiment_tag: 3_activation=relu,height=-32.8474,steps=100,width=13.1144
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 6.791690049131352
  neg_mean_loss: -6.791690049131352
  node_ip: 127.0.0.1
  pid: 46150
  time_since_restore: 11.807135105133057
  time_this_iter_s: 0.10672497749328613
  time_total_s: 11.807135105133057
  timestamp: 1658499726
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: a11eef8e
  warmup_time: 0.0028891563415527344
  

Result for objective_a9cb308e:
  date: 2022-07-22_15-22-08
  done: false
  experiment_id: fcc8b8f4880c471c8e8545228d7e591a
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 14.84802392036395
  neg_mean_loss: -14.84802392036395
  node_ip: 127.0.0.1
  pid: 46184
  time_since_restore: 0.1039888858795166
  time_this_iter_s: 0.1039888858795166
  time_total_s: 0.1039888858795166
  timestamp: 1658499728
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: a9cb308e
  warmup_time: 0.0027310848236083984
  
Result for objective_a9c844c8:
  date: 2022-07-22_15-22-08
  done: false
  experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 15.960686077942514
  neg_mean_loss: -15.960686077942514
  node_ip: 127.0.0.1
  pid: 46183
  time_since_restore: 0.1004641056060791
  time_this_iter_s: 0.1004641056060791
  time_total_s: 0.1004641056060791
  timestamp: 1658499728
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: a9c844c8
  warmup_time: 0.002599954605102539
  
Result for objective_a9d8332e:
  date: 2022-07-22_15-22-08
  done: false
  experiment_id: 9c2f8aa7e4ed45c8867025673b146195
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 9.873318211243287
  neg_mean_loss: -9.873318211243287
  node_ip: 127.0.0.1
  pid: 46189
  time_since_restore: 0.10463905334472656
  time_this_iter_s: 0.10463905334472656
  time_total_s: 0.10463905334472656
  timestamp: 1658499728
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: a9d8332e
  warmup_time: 0.002753019332885742
  
Result for objective_a7b383e6:
  date: 2022-07-22_15-22-10
  done: false
  experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 15.594128340598196
  neg_mean_loss: -15.594128340598196
  node_ip: 127.0.0.1
  pid: 46174
  time_since_restore: 5.12034797668457
  time_this_iter_s: 0.10577988624572754
  time_total_s: 5.12034797668457
  timestamp: 1658499730
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: a7b383e6
  warmup_time: 0.002817869186401367
  
Result for objective_a9cb308e:
  date: 2022-07-22_15-22-13
  done: false
  experiment_id: fcc8b8f4880c471c8e8545228d7e591a
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 5.011849709111575
  neg_mean_loss: -5.011849709111575
  node_ip: 127.0.0.1
  pid: 46184
  time_since_restore: 5.150436162948608
  time_this_iter_s: 0.10827016830444336
  time_total_s: 5.150436162948608
  timestamp: 1658499733
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: a9cb308e
  warmup_time: 0.0027310848236083984
  
Result for objective_a9c844c8:
  date: 2022-07-22_15-22-13
  done: false
  experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 6.109835222238134
  neg_mean_loss: -6.109835222238134
  node_ip: 127.0.0.1
  pid: 46183
  time_since_restore: 5.136260032653809
  time_this_iter_s: 0.10730218887329102
  time_total_s: 5.136260032653809
  timestamp: 1658499733
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: a9c844c8
  warmup_time: 0.002599954605102539
  
Result for objective_a9d8332e:
  date: 2022-07-22_15-22-13
  done: false
  experiment_id: 9c2f8aa7e4ed45c8867025673b146195
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 0.06343129971586822
  neg_mean_loss: -0.06343129971586822
  node_ip: 127.0.0.1
  pid: 46189
  time_since_restore: 5.141700983047485
  time_this_iter_s: 0.1074991226196289
  time_total_s: 5.141700983047485
  timestamp: 1658499733
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: a9d8332e
  warmup_time: 0.002753019332885742
  
Result for objective_a7b383e6:
  date: 2022-07-22_15-22-15
  done: false
  experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: 15.313204013214245
  neg_mean_loss: -15.313204013214245
  node_ip: 127.0.0.1
  pid: 46174
  time_since_restore: 10.163710832595825
  time_this_iter_s: 0.10754895210266113
  time_total_s: 10.163710832595825
  timestamp: 1658499735
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: a7b383e6
  warmup_time: 0.002817869186401367
  
Result for objective_a7b383e6:
  date: 2022-07-22_15-22-15
  done: true
  experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa
  experiment_tag: 5_activation=relu,height=50.1501,steps=100,width=3.4612
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 15.298570798421121
  neg_mean_loss: -15.298570798421121
  node_ip: 127.0.0.1
  pid: 46174
  time_since_restore: 10.701509952545166
  time_this_iter_s: 0.10684585571289062
  time_total_s: 10.701509952545166
  timestamp: 1658499735
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: a7b383e6
  warmup_time: 0.002817869186401367
  
Result for objective_af948a1a:
  date: 2022-07-22_15-22-18
  done: false
  experiment_id: d27f58d6527a4d47ae4c12496f5275cf
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 16.16381214127525
  neg_mean_loss: -16.16381214127525
  node_ip: 127.0.0.1
  pid: 46204
  time_since_restore: 0.10407209396362305
  time_this_iter_s: 0.10407209396362305
  time_total_s: 0.10407209396362305
  timestamp: 1658499738
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: af948a1a
  warmup_time: 0.002669095993041992
  
Result for objective_a9cb308e:
  date: 2022-07-22_15-22-18
  done: false
  experiment_id: fcc8b8f4880c471c8e8545228d7e591a
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: 4.930613328484136
  neg_mean_loss: -4.930613328484136
  node_ip: 127.0.0.1
  pid: 46184
  time_since_restore: 10.175731897354126
  time_this_iter_s: 0.10618400573730469
  time_total_s: 10.175731897354126
  timestamp: 1658499738
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: a9cb308e
  warmup_time: 0.0027310848236083984
  
Result for objective_a9d8332e:
  date: 2022-07-22_15-22-18
  done: false
  experiment_id: 9c2f8aa7e4ed45c8867025673b146195
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: -0.030712998363490673
  neg_mean_loss: 0.030712998363490673
  node_ip: 127.0.0.1
  pid: 46189
  time_since_restore: 10.171820163726807
  time_this_iter_s: 0.10807609558105469
  time_total_s: 10.171820163726807
  timestamp: 1658499738
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: a9d8332e
  warmup_time: 0.002753019332885742
  
Result for objective_a9c844c8:
  date: 2022-07-22_15-22-18
  done: false
  experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e
  hostname: Kais-MacBook-Pro.local
  iterations: 95
  iterations_since_restore: 96
  mean_loss: 6.035035952030598
  neg_mean_loss: -6.035035952030598
  node_ip: 127.0.0.1
  pid: 46183
  time_since_restore: 10.23954701423645
  time_this_iter_s: 0.10863399505615234
  time_total_s: 10.23954701423645
  timestamp: 1658499738
  timesteps_since_restore: 0
  training_iteration: 96
  trial_id: a9c844c8
  warmup_time: 0.002599954605102539
  
Result for objective_a9cb308e:
  date: 2022-07-22_15-22-19
  done: true
  experiment_id: fcc8b8f4880c471c8e8545228d7e591a
  experiment_tag: 7_activation=tanh,height=48.4802,steps=100,width=12.7746
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 4.926474869576359
  neg_mean_loss: -4.926474869576359
  node_ip: 127.0.0.1
  pid: 46184
  time_since_restore: 10.712197065353394
  time_this_iter_s: 0.10726809501647949
  time_total_s: 10.712197065353394
  timestamp: 1658499739
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: a9cb308e
  warmup_time: 0.0027310848236083984
  
Result for objective_a9c844c8:
  date: 2022-07-22_15-22-19
  done: true
  experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e
  experiment_tag: 6_activation=relu,height=-40.3931,steps=100,width=14.0525
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 6.03205335569619
  neg_mean_loss: -6.03205335569619
  node_ip: 127.0.0.1
  pid: 46183
  time_since_restore: 10.668684959411621
  time_this_iter_s: 0.1074681282043457
  time_total_s: 10.668684959411621
  timestamp: 1658499739
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: a9c844c8
  warmup_time: 0.002599954605102539
  
Result for objective_a9d8332e:
  date: 2022-07-22_15-22-19
  done: true
  experiment_id: 9c2f8aa7e4ed45c8867025673b146195
  experiment_tag: 8_activation=tanh,height=-1.2668,steps=100,width=10.9788
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -0.035515719608698634
  neg_mean_loss: 0.035515719608698634
  node_ip: 127.0.0.1
  pid: 46189
  time_since_restore: 10.708936929702759
  time_this_iter_s: 0.10948991775512695
  time_total_s: 10.708936929702759
  timestamp: 1658499739
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: a9d8332e
  warmup_time: 0.002753019332885742
  

Result for objective_b1bad2fe:
  date: 2022-07-22_15-22-21
  done: false
  experiment_id: f58c8e48e2324dd69c28f526bf96497c
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 8.353710439145445
  neg_mean_loss: -8.353710439145445
  node_ip: 127.0.0.1
  pid: 46209
  time_since_restore: 0.10369992256164551
  time_this_iter_s: 0.10369992256164551
  time_total_s: 0.10369992256164551
  timestamp: 1658499741
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: b1bad2fe
  warmup_time: 0.0026597976684570312
  
Result for objective_af948a1a:
  date: 2022-07-22_15-22-23
  done: false
  experiment_id: d27f58d6527a4d47ae4c12496f5275cf
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 6.353387644558537
  neg_mean_loss: -6.353387644558537
  node_ip: 127.0.0.1
  pid: 46204
  time_since_restore: 5.161527872085571
  time_this_iter_s: 0.10784792900085449
  time_total_s: 5.161527872085571
  timestamp: 1658499743
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: af948a1a
  warmup_time: 0.002669095993041992
  
Result for objective_b1bad2fe:
  date: 2022-07-22_15-22-26
  done: false
  experiment_id: f58c8e48e2324dd69c28f526bf96497c
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: -1.3770075531258914
  neg_mean_loss: 1.3770075531258914
  node_ip: 127.0.0.1
  pid: 46209
  time_since_restore: 5.158898115158081
  time_this_iter_s: 0.10701417922973633
  time_total_s: 5.158898115158081
  timestamp: 1658499746
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: b1bad2fe
  warmup_time: 0.0026597976684570312
  
Result for objective_af948a1a:
  date: 2022-07-22_15-22-28
  done: false
  experiment_id: d27f58d6527a4d47ae4c12496f5275cf
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: 6.259506962612728
  neg_mean_loss: -6.259506962612728
  node_ip: 127.0.0.1
  pid: 46204
  time_since_restore: 10.20361876487732
  time_this_iter_s: 0.10824179649353027
  time_total_s: 10.20361876487732
  timestamp: 1658499748
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: af948a1a
  warmup_time: 0.002669095993041992
  
Result for objective_af948a1a:
  date: 2022-07-22_15-22-28
  done: true
  experiment_id: d27f58d6527a4d47ae4c12496f5275cf
  experiment_tag: 9_activation=tanh,height=61.6381,steps=100,width=11.0105
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 6.254717826198094
  neg_mean_loss: -6.254717826198094
  node_ip: 127.0.0.1
  pid: 46204
  time_since_restore: 10.744001865386963
  time_this_iter_s: 0.10536718368530273
  time_total_s: 10.744001865386963
  timestamp: 1658499748
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: af948a1a
  warmup_time: 0.002669095993041992
  
Result for objective_b1bad2fe:
  date: 2022-07-22_15-22-31
  done: false
  experiment_id: f58c8e48e2324dd69c28f526bf96497c
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: -1.5098109958909571
  neg_mean_loss: 1.5098109958909571
  node_ip: 127.0.0.1
  pid: 46209
  time_since_restore: 10.190011024475098
  time_this_iter_s: 0.10639405250549316
  time_total_s: 10.190011024475098
  timestamp: 1658499751
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: b1bad2fe
  warmup_time: 0.0026597976684570312
  
Result for objective_b1bad2fe:
  date: 2022-07-22_15-22-32
  done: true
  experiment_id: f58c8e48e2324dd69c28f526bf96497c
  experiment_tag: 10_activation=tanh,height=-16.4629,steps=100,width=7.6885
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -1.5166144695235813
  neg_mean_loss: 1.5166144695235813
  node_ip: 127.0.0.1
  pid: 46209
  time_since_restore: 10.727396011352539
  time_this_iter_s: 0.10742807388305664
  time_total_s: 10.727396011352539
  timestamp: 1658499752
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: b1bad2fe
  warmup_time: 0.0026597976684570312
  

And now we have the hyperparameters found to minimize the mean loss.

print("Best hyperparameters found were: ", results.get_best_result().config)

Best hyperparameters found were:  {'steps': 100, 'width': 7.688465886501621, 'height': -16.46289560854555, 'activation': 'tanh'}

Providing an initial set of hyperparameters#

While defining the search algorithm, we may choose to provide an initial set of hyperparameters that we believe are especially promising or informative, and pass this information as a helpful starting point for the OptunaSearch object.

initial_params = [
    {"width": 1, "height": 2, "activation": "relu"},
    {"width": 4, "height": 2, "activation": "relu"},
]

Now the search_alg built using OptunaSearch takes points_to_evaluate.

searcher = OptunaSearch(points_to_evaluate=initial_params)
algo = ConcurrencyLimiter(searcher, max_concurrent=4)

And run the experiment with initial hyperparameter evaluations:

tuner = tune.Tuner(
    objective,
    tune_config=tune.TuneConfig(
        metric="mean_loss",
        mode="min",
        search_alg=algo,
        num_samples=num_samples,
    ),
    param_space=search_space,
)
results = tuner.fit()

[I 2022-07-22 15:22:32,644] A new study created in memory with name: optuna
/Users/kai/coding/ray/python/ray/tune/search/optuna/optuna_search.py:389: ExperimentalWarning: enqueue_trial is experimental (supported from v1.2.0). The interface can change in the future.
  self._ot_study.enqueue_trial(point)
/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/optuna/study/study.py:857: ExperimentalWarning: create_trial is experimental (supported from v2.0.0). The interface can change in the future.
  create_trial(state=TrialState.WAITING, system_attrs={"fixed_params": params})
/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/optuna/study/study.py:857: ExperimentalWarning: add_trial is experimental (supported from v2.0.0). The interface can change in the future.
  create_trial(state=TrialState.WAITING, system_attrs={"fixed_params": params})

== Status ==
Current time: 2022-07-22 15:23:15 (running for 00:00:42.97)
Memory usage on this node: 10.1/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects
Current best trial: c3762426 with mean_loss=-8.621481572769046 and parameters={'steps': 100, 'width': 0.7236705053153614, 'height': -98.46320622400306, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/optuna_example_2
Number of trials: 10/10 (10 TERMINATED)

Trial name	status	loc	activation	height	width	loss	iter	total time (s)	iterations	neg_mean_loss
objective_b9acff32	TERMINATED	127.0.0.1:46220	relu	2	1	11.1174	100	10.6837	99	-11.1174
objective_bb3b2702	TERMINATED	127.0.0.1:46227	relu	2	4	10.4463	100	11.2816	99	-10.4463
objective_bb3c8714	TERMINATED	127.0.0.1:46228	relu	-90.7016	0.566597	2.44285	100	11.2766	99	-2.44285
objective_bb3e0210	TERMINATED	127.0.0.1:46229	tanh	-81.4501	14.783	-8.07715	100	11.2196	99	8.07715
objective_c19b9dde	TERMINATED	127.0.0.1:46248	relu	86.466	0.308659	21.1122	100	10.7226	99	-21.1122
objective_c3762426	TERMINATED	127.0.0.1:46253	tanh	-98.4632	0.723671	-8.62148	100	10.7271	99	8.62148
objective_c385528e	TERMINATED	127.0.0.1:46256	relu	-54.8368	11.0765	4.60669	100	10.7498	99	-4.60669
objective_c387a7c8	TERMINATED	127.0.0.1:46257	tanh	42.9466	8.01708	4.41909	100	10.7214	99	-4.41909
objective_c97678f8	TERMINATED	127.0.0.1:46272	tanh	9.25214	15.5288	0.989841	100	12.4591	99	-0.989841
objective_cb6180fe	TERMINATED	127.0.0.1:46277	relu	41.6647	8.0585	14.2903	100	10.752	99	-14.2903

Result for objective_b9acff32:
  date: 2022-07-22_15-22-35
  done: false
  experiment_id: ec3c260f0ad348fa9b188f9c93c01eba
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 20.2
  neg_mean_loss: -20.2
  node_ip: 127.0.0.1
  pid: 46220
  time_since_restore: 0.10049605369567871
  time_this_iter_s: 0.10049605369567871
  time_total_s: 0.10049605369567871
  timestamp: 1658499755
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: b9acff32
  warmup_time: 0.0029239654541015625
  
Result for objective_bb3e0210:
  date: 2022-07-22_15-22-37
  done: false
  experiment_id: 08a7bc47c140405296e58a049cd2f6b4
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 1.8549886153795079
  neg_mean_loss: -1.8549886153795079
  node_ip: 127.0.0.1
  pid: 46229
  time_since_restore: 0.10361814498901367
  time_this_iter_s: 0.10361814498901367
  time_total_s: 0.10361814498901367
  timestamp: 1658499757
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: bb3e0210
  warmup_time: 0.002833843231201172
  
Result for objective_bb3b2702:
  date: 2022-07-22_15-22-37
  done: false
  experiment_id: 2f984120624b4e9aa3392fcc1b16b811
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 20.2
  neg_mean_loss: -20.2
  node_ip: 127.0.0.1
  pid: 46227
  time_since_restore: 0.10420107841491699
  time_this_iter_s: 0.10420107841491699
  time_total_s: 0.10420107841491699
  timestamp: 1658499757
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: bb3b2702
  warmup_time: 0.002672910690307617
  
Result for objective_bb3c8714:
  date: 2022-07-22_15-22-37
  done: false
  experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 10.929835533284928
  neg_mean_loss: -10.929835533284928
  node_ip: 127.0.0.1
  pid: 46228
  time_since_restore: 0.1048588752746582
  time_this_iter_s: 0.1048588752746582
  time_total_s: 0.1048588752746582
  timestamp: 1658499757
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: bb3c8714
  warmup_time: 0.0027511119842529297
  
Result for objective_b9acff32:
  date: 2022-07-22_15-22-40
  done: false
  experiment_id: ec3c260f0ad348fa9b188f9c93c01eba
  hostname: Kais-MacBook-Pro.local
  iterations: 48
  iterations_since_restore: 49
  mean_loss: 11.924137931034483
  neg_mean_loss: -11.924137931034483
  node_ip: 127.0.0.1
  pid: 46220
  time_since_restore: 5.207761287689209
  time_this_iter_s: 0.10753417015075684
  time_total_s: 5.207761287689209
  timestamp: 1658499760
  timesteps_since_restore: 0
  training_iteration: 49
  trial_id: b9acff32
  warmup_time: 0.0029239654541015625
  
Result for objective_bb3e0210:
  date: 2022-07-22_15-22-43
  done: false
  experiment_id: 08a7bc47c140405296e58a049cd2f6b4
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: -8.003127819283387
  neg_mean_loss: 8.003127819283387
  node_ip: 127.0.0.1
  pid: 46229
  time_since_restore: 5.188760042190552
  time_this_iter_s: 0.10853385925292969
  time_total_s: 5.188760042190552
  timestamp: 1658499763
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: bb3e0210
  warmup_time: 0.002833843231201172
  
Result for objective_bb3b2702:
  date: 2022-07-22_15-22-43
  done: false
  experiment_id: 2f984120624b4e9aa3392fcc1b16b811
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 10.705050505050504
  neg_mean_loss: -10.705050505050504
  node_ip: 127.0.0.1
  pid: 46227
  time_since_restore: 5.1574530601501465
  time_this_iter_s: 0.10719585418701172
  time_total_s: 5.1574530601501465
  timestamp: 1658499763
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: bb3b2702
  warmup_time: 0.002672910690307617
  
Result for objective_bb3c8714:
  date: 2022-07-22_15-22-43
  done: false
  experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 3.6598337799021596
  neg_mean_loss: -3.6598337799021596
  node_ip: 127.0.0.1
  pid: 46228
  time_since_restore: 5.163677930831909
  time_this_iter_s: 0.10699272155761719
  time_total_s: 5.163677930831909
  timestamp: 1658499763
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: bb3c8714
  warmup_time: 0.0027511119842529297
  
Result for objective_b9acff32:
  date: 2022-07-22_15-22-45
  done: false
  experiment_id: ec3c260f0ad348fa9b188f9c93c01eba
  hostname: Kais-MacBook-Pro.local
  iterations: 95
  iterations_since_restore: 96
  mean_loss: 11.152380952380952
  neg_mean_loss: -11.152380952380952
  node_ip: 127.0.0.1
  pid: 46220
  time_since_restore: 10.250982284545898
  time_this_iter_s: 0.1073312759399414
  time_total_s: 10.250982284545898
  timestamp: 1658499765
  timesteps_since_restore: 0
  training_iteration: 96
  trial_id: b9acff32
  warmup_time: 0.0029239654541015625
  
Result for objective_b9acff32:
  date: 2022-07-22_15-22-45
  done: true
  experiment_id: ec3c260f0ad348fa9b188f9c93c01eba
  experiment_tag: 1_activation=relu,height=2,steps=100,width=1
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 11.11743119266055
  neg_mean_loss: -11.11743119266055
  node_ip: 127.0.0.1
  pid: 46220
  time_since_restore: 10.68371295928955
  time_this_iter_s: 0.10785365104675293
  time_total_s: 10.68371295928955
  timestamp: 1658499765
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: b9acff32
  warmup_time: 0.0029239654541015625
  
Result for objective_bb3e0210:
  date: 2022-07-22_15-22-47
  done: false
  experiment_id: 08a7bc47c140405296e58a049cd2f6b4
  hostname: Kais-MacBook-Pro.local
  iterations: 93
  iterations_since_restore: 94
  mean_loss: -8.072800015147221
  neg_mean_loss: 8.072800015147221
  node_ip: 127.0.0.1
  pid: 46229
  time_since_restore: 10.099772214889526
  time_this_iter_s: 0.10250687599182129
  time_total_s: 10.099772214889526
  timestamp: 1658499767
  timesteps_since_restore: 0
  training_iteration: 94
  trial_id: bb3e0210
  warmup_time: 0.002833843231201172
  
Result for objective_bb3c8714:
  date: 2022-07-22_15-22-47
  done: false
  experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf
  hostname: Kais-MacBook-Pro.local
  iterations: 92
  iterations_since_restore: 93
  mean_loss: 2.53944357306219
  neg_mean_loss: -2.53944357306219
  node_ip: 127.0.0.1
  pid: 46228
  time_since_restore: 9.980368852615356
  time_this_iter_s: 0.10658979415893555
  time_total_s: 9.980368852615356
  timestamp: 1658499767
  timesteps_since_restore: 0
  training_iteration: 93
  trial_id: bb3c8714
  warmup_time: 0.0027511119842529297
  
Result for objective_bb3b2702:
  date: 2022-07-22_15-22-47
  done: false
  experiment_id: 2f984120624b4e9aa3392fcc1b16b811
  hostname: Kais-MacBook-Pro.local
  iterations: 92
  iterations_since_restore: 93
  mean_loss: 10.464550264550265
  neg_mean_loss: -10.464550264550265
  node_ip: 127.0.0.1
  pid: 46227
  time_since_restore: 9.964121103286743
  time_this_iter_s: 0.10547995567321777
  time_total_s: 9.964121103286743
  timestamp: 1658499767
  timesteps_since_restore: 0
  training_iteration: 93
  trial_id: bb3b2702
  warmup_time: 0.002672910690307617
  
Result for objective_c19b9dde:
  date: 2022-07-22_15-22-48
  done: false
  experiment_id: fab35a2f68c347c4bb815c268e9500a6
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 28.646596578422436
  neg_mean_loss: -28.646596578422436
  node_ip: 127.0.0.1
  pid: 46248
  time_since_restore: 0.10230708122253418
  time_this_iter_s: 0.10230708122253418
  time_total_s: 0.10230708122253418
  timestamp: 1658499768
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: c19b9dde
  warmup_time: 0.0026540756225585938
  
Result for objective_bb3e0210:
  date: 2022-07-22_15-22-49
  done: true
  experiment_id: 08a7bc47c140405296e58a049cd2f6b4
  experiment_tag: 4_activation=tanh,height=-81.4501,steps=100,width=14.7830
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -8.077146761191683
  neg_mean_loss: 8.077146761191683
  node_ip: 127.0.0.1
  pid: 46229
  time_since_restore: 11.219618082046509
  time_this_iter_s: 0.1108238697052002
  time_total_s: 11.219618082046509
  timestamp: 1658499769
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: bb3e0210
  warmup_time: 0.002833843231201172
  
Result for objective_bb3c8714:
  date: 2022-07-22_15-22-49
  done: true
  experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf
  experiment_tag: 3_activation=relu,height=-90.7016,steps=100,width=0.5666
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 2.442852652712544
  neg_mean_loss: -2.442852652712544
  node_ip: 127.0.0.1
  pid: 46228
  time_since_restore: 11.276630163192749
  time_this_iter_s: 0.10824012756347656
  time_total_s: 11.276630163192749
  timestamp: 1658499769
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: bb3c8714
  warmup_time: 0.0027511119842529297
  
Result for objective_bb3b2702:
  date: 2022-07-22_15-22-49
  done: true
  experiment_id: 2f984120624b4e9aa3392fcc1b16b811
  experiment_tag: 2_activation=relu,height=2,steps=100,width=4
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 10.446305418719211
  neg_mean_loss: -10.446305418719211
  node_ip: 127.0.0.1
  pid: 46227
  time_since_restore: 11.281576871871948
  time_this_iter_s: 0.1068716049194336
  time_total_s: 11.281576871871948
  timestamp: 1658499769
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: bb3b2702
  warmup_time: 0.002672910690307617
  

Result for objective_c3762426:
  date: 2022-07-22_15-22-51
  done: false
  experiment_id: 034596724b2a4c9f86726e9a15fb386b
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 0.15367937759969408
  neg_mean_loss: -0.15367937759969408
  node_ip: 127.0.0.1
  pid: 46253
  time_since_restore: 0.1050560474395752
  time_this_iter_s: 0.1050560474395752
  time_total_s: 0.1050560474395752
  timestamp: 1658499771
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: c3762426
  warmup_time: 0.0028579235076904297
  
Result for objective_c385528e:
  date: 2022-07-22_15-22-51
  done: false
  experiment_id: 3044e8f1500f4b16aba04b72105c67be
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 14.516317633001044
  neg_mean_loss: -14.516317633001044
  node_ip: 127.0.0.1
  pid: 46256
  time_since_restore: 0.10402488708496094
  time_this_iter_s: 0.10402488708496094
  time_total_s: 0.10402488708496094
  timestamp: 1658499771
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: c385528e
  warmup_time: 0.0027010440826416016
  
Result for objective_c387a7c8:
  date: 2022-07-22_15-22-51
  done: false
  experiment_id: fc7f3d46b136437cab526c2fa6ea2944
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 14.294664350689382
  neg_mean_loss: -14.294664350689382
  node_ip: 127.0.0.1
  pid: 46257
  time_since_restore: 0.10461091995239258
  time_this_iter_s: 0.10461091995239258
  time_total_s: 0.10461091995239258
  timestamp: 1658499771
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: c387a7c8
  warmup_time: 0.0025899410247802734
  
Result for objective_c19b9dde:
  date: 2022-07-22_15-22-53
  done: false
  experiment_id: fab35a2f68c347c4bb815c268e9500a6
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 22.72706638821166
  neg_mean_loss: -22.72706638821166
  node_ip: 127.0.0.1
  pid: 46248
  time_since_restore: 5.118642330169678
  time_this_iter_s: 0.10826611518859863
  time_total_s: 5.118642330169678
  timestamp: 1658499773
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: c19b9dde
  warmup_time: 0.0026540756225585938
  
Result for objective_c3762426:
  date: 2022-07-22_15-22-56
  done: false
  experiment_id: 034596724b2a4c9f86726e9a15fb386b
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: -7.5742395372701825
  neg_mean_loss: 7.5742395372701825
  node_ip: 127.0.0.1
  pid: 46253
  time_since_restore: 5.14322304725647
  time_this_iter_s: 0.10359907150268555
  time_total_s: 5.14322304725647
  timestamp: 1658499776
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: c3762426
  warmup_time: 0.0028579235076904297
  
Result for objective_c385528e:
  date: 2022-07-22_15-22-56
  done: false
  experiment_id: 3044e8f1500f4b16aba04b72105c67be
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 4.704784733826337
  neg_mean_loss: -4.704784733826337
  node_ip: 127.0.0.1
  pid: 46256
  time_since_restore: 5.16935396194458
  time_this_iter_s: 0.10719513893127441
  time_total_s: 5.16935396194458
  timestamp: 1658499776
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: c385528e
  warmup_time: 0.0027010440826416016
  
Result for objective_c387a7c8:
  date: 2022-07-22_15-22-56
  done: false
  experiment_id: fc7f3d46b136437cab526c2fa6ea2944
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 4.553194134463865
  neg_mean_loss: -4.553194134463865
  node_ip: 127.0.0.1
  pid: 46257
  time_since_restore: 5.1458728313446045
  time_this_iter_s: 0.10374999046325684
  time_total_s: 5.1458728313446045
  timestamp: 1658499776
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: c387a7c8
  warmup_time: 0.0025899410247802734
  
Result for objective_c19b9dde:
  date: 2022-07-22_15-22-58
  done: false
  experiment_id: fab35a2f68c347c4bb815c268e9500a6
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: 21.20978139671461
  neg_mean_loss: -21.20978139671461
  node_ip: 127.0.0.1
  pid: 46248
  time_since_restore: 10.183105230331421
  time_this_iter_s: 0.10718917846679688
  time_total_s: 10.183105230331421
  timestamp: 1658499778
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: c19b9dde
  warmup_time: 0.0026540756225585938
  
Result for objective_c19b9dde:
  date: 2022-07-22_15-22-59
  done: true
  experiment_id: fab35a2f68c347c4bb815c268e9500a6
  experiment_tag: 5_activation=relu,height=86.4660,steps=100,width=0.3087
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 21.112246386473707
  neg_mean_loss: -21.112246386473707
  node_ip: 127.0.0.1
  pid: 46248
  time_since_restore: 10.722583055496216
  time_this_iter_s: 0.10788774490356445
  time_total_s: 10.722583055496216
  timestamp: 1658499779
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: c19b9dde
  warmup_time: 0.0026540756225585938
  
Result for objective_c97678f8:
  date: 2022-07-22_15-23-01
  done: false
  experiment_id: 7c4187f99b9246bdbae534ce20f33bfc
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 10.925214276212655
  neg_mean_loss: -10.925214276212655
  node_ip: 127.0.0.1
  pid: 46272
  time_since_restore: 0.1039581298828125
  time_this_iter_s: 0.1039581298828125
  time_total_s: 0.1039581298828125
  timestamp: 1658499781
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: c97678f8
  warmup_time: 0.002711057662963867
  
Result for objective_c3762426:
  date: 2022-07-22_15-23-01
  done: false
  experiment_id: 034596724b2a4c9f86726e9a15fb386b
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: -8.564680574270156
  neg_mean_loss: 8.564680574270156
  node_ip: 127.0.0.1
  pid: 46253
  time_since_restore: 10.190189123153687
  time_this_iter_s: 0.10848307609558105
  time_total_s: 10.190189123153687
  timestamp: 1658499781
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: c3762426
  warmup_time: 0.0028579235076904297
  
Result for objective_c387a7c8:
  date: 2022-07-22_15-23-01
  done: false
  experiment_id: fc7f3d46b136437cab526c2fa6ea2944
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: 4.425622066068654
  neg_mean_loss: -4.425622066068654
  node_ip: 127.0.0.1
  pid: 46257
  time_since_restore: 10.181554794311523
  time_this_iter_s: 0.10594582557678223
  time_total_s: 10.181554794311523
  timestamp: 1658499781
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: c387a7c8
  warmup_time: 0.0025899410247802734
  
Result for objective_c385528e:
  date: 2022-07-22_15-23-01
  done: false
  experiment_id: 3044e8f1500f4b16aba04b72105c67be
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: 4.611447627123378
  neg_mean_loss: -4.611447627123378
  node_ip: 127.0.0.1
  pid: 46256
  time_since_restore: 10.214791059494019
  time_this_iter_s: 0.10866808891296387
  time_total_s: 10.214791059494019
  timestamp: 1658499781
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: c385528e
  warmup_time: 0.0027010440826416016
  
Result for objective_c3762426:
  date: 2022-07-22_15-23-02
  done: true
  experiment_id: 034596724b2a4c9f86726e9a15fb386b
  experiment_tag: 6_activation=tanh,height=-98.4632,steps=100,width=0.7237
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -8.621481572769046
  neg_mean_loss: 8.621481572769046
  node_ip: 127.0.0.1
  pid: 46253
  time_since_restore: 10.72708010673523
  time_this_iter_s: 0.10719490051269531
  time_total_s: 10.72708010673523
  timestamp: 1658499782
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: c3762426
  warmup_time: 0.0028579235076904297
  
Result for objective_c387a7c8:
  date: 2022-07-22_15-23-02
  done: true
  experiment_id: fc7f3d46b136437cab526c2fa6ea2944
  experiment_tag: 8_activation=tanh,height=42.9466,steps=100,width=8.0171
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 4.419090335709762
  neg_mean_loss: -4.419090335709762
  node_ip: 127.0.0.1
  pid: 46257
  time_since_restore: 10.721372842788696
  time_this_iter_s: 0.10873699188232422
  time_total_s: 10.721372842788696
  timestamp: 1658499782
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: c387a7c8
  warmup_time: 0.0025899410247802734
  
Result for objective_c385528e:
  date: 2022-07-22_15-23-02
  done: true
  experiment_id: 3044e8f1500f4b16aba04b72105c67be
  experiment_tag: 7_activation=relu,height=-54.8368,steps=100,width=11.0765
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 4.606686500095965
  neg_mean_loss: -4.606686500095965
  node_ip: 127.0.0.1
  pid: 46256
  time_since_restore: 10.749844074249268
  time_this_iter_s: 0.10710501670837402
  time_total_s: 10.749844074249268
  timestamp: 1658499782
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: c385528e
  warmup_time: 0.0027010440826416016
  

Result for objective_cb6180fe:
  date: 2022-07-22_15-23-04
  done: false
  experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 24.166473702046588
  neg_mean_loss: -24.166473702046588
  node_ip: 127.0.0.1
  pid: 46277
  time_since_restore: 0.10389995574951172
  time_this_iter_s: 0.10389995574951172
  time_total_s: 0.10389995574951172
  timestamp: 1658499784
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: cb6180fe
  warmup_time: 0.0027260780334472656
  
Result for objective_c97678f8:
  date: 2022-07-22_15-23-06
  done: false
  experiment_id: 7c4187f99b9246bdbae534ce20f33bfc
  hostname: Kais-MacBook-Pro.local
  iterations: 31
  iterations_since_restore: 32
  mean_loss: 1.1287174007116527
  neg_mean_loss: -1.1287174007116527
  node_ip: 127.0.0.1
  pid: 46272
  time_since_restore: 5.140314340591431
  time_this_iter_s: 0.10663819313049316
  time_total_s: 5.140314340591431
  timestamp: 1658499786
  timesteps_since_restore: 0
  training_iteration: 32
  trial_id: c97678f8
  warmup_time: 0.002711057662963867
  
Result for objective_cb6180fe:
  date: 2022-07-22_15-23-09
  done: false
  experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 14.42370878193022
  neg_mean_loss: -14.42370878193022
  node_ip: 127.0.0.1
  pid: 46277
  time_since_restore: 5.139087915420532
  time_this_iter_s: 0.10686802864074707
  time_total_s: 5.139087915420532
  timestamp: 1658499789
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: cb6180fe
  warmup_time: 0.0027260780334472656
  
Result for objective_c97678f8:
  date: 2022-07-22_15-23-11
  done: false
  experiment_id: 7c4187f99b9246bdbae534ce20f33bfc
  hostname: Kais-MacBook-Pro.local
  iterations: 78
  iterations_since_restore: 79
  mean_loss: 1.0070978091228147
  neg_mean_loss: -1.0070978091228147
  node_ip: 127.0.0.1
  pid: 46272
  time_since_restore: 10.206932067871094
  time_this_iter_s: 0.10774993896484375
  time_total_s: 10.206932067871094
  timestamp: 1658499791
  timesteps_since_restore: 0
  training_iteration: 79
  trial_id: c97678f8
  warmup_time: 0.002711057662963867
  
Result for objective_c97678f8:
  date: 2022-07-22_15-23-14
  done: true
  experiment_id: 7c4187f99b9246bdbae534ce20f33bfc
  experiment_tag: 9_activation=tanh,height=9.2521,steps=100,width=15.5288
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 0.9898408262377837
  neg_mean_loss: -0.9898408262377837
  node_ip: 127.0.0.1
  pid: 46272
  time_since_restore: 12.459095239639282
  time_this_iter_s: 0.10785508155822754
  time_total_s: 12.459095239639282
  timestamp: 1658499794
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: c97678f8
  warmup_time: 0.002711057662963867
  
Result for objective_cb6180fe:
  date: 2022-07-22_15-23-15
  done: false
  experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: 14.296767042885353
  neg_mean_loss: -14.296767042885353
  node_ip: 127.0.0.1
  pid: 46277
  time_since_restore: 10.220713138580322
  time_this_iter_s: 0.10750126838684082
  time_total_s: 10.220713138580322
  timestamp: 1658499795
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: cb6180fe
  warmup_time: 0.0027260780334472656
  
Result for objective_cb6180fe:
  date: 2022-07-22_15-23-15
  done: true
  experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea
  experiment_tag: 10_activation=relu,height=41.6647,steps=100,width=8.0585
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 14.29026803363612
  neg_mean_loss: -14.29026803363612
  node_ip: 127.0.0.1
  pid: 46277
  time_since_restore: 10.752035856246948
  time_this_iter_s: 0.1066899299621582
  time_total_s: 10.752035856246948
  timestamp: 1658499795
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: cb6180fe
  warmup_time: 0.0027260780334472656
  

We take another look at the optimal hyperparamters.

print("Best hyperparameters found were: ", results.get_best_result().config)

Best hyperparameters found were:  {'steps': 100, 'width': 0.7236705053153614, 'height': -98.46320622400306, 'activation': 'tanh'}

Conditional search spaces#

Sometimes we may want to build a more complicated search space that has conditional dependencies on other hyperparameters. In this case, we pass a define-by-run function to the search_alg argument in ray.tune().

def define_by_run_func(trial) -> Optional[Dict[str, Any]]:
    """Define-by-run function to create the search space.

    Ensure no actual computation takes place here. That should go into
    the trainable passed to ``Tuner()`` (in this example, that's
    ``objective``).

    For more information, see https://optuna.readthedocs.io/en/stable\
    /tutorial/10_key_features/002_configurations.html

    This function should either return None or a dict with constant values.
    """

    activation = trial.suggest_categorical("activation", ["relu", "tanh"])

    # Define-by-run allows for conditional search spaces.
    if activation == "relu":
        trial.suggest_float("width", 0, 20)
        trial.suggest_float("height", -100, 100)
    else:
        trial.suggest_float("width", -1, 21)
        trial.suggest_float("height", -101, 101)
        
    # Return all constants in a dictionary.
    return {"steps": 100}

As before, we create the search_alg from OptunaSearch and ConcurrencyLimiter, this time we define the scope of search via the space argument and provide no initialization. We also must specific metric and mode when using space.

searcher = OptunaSearch(space=define_by_run_func, metric="mean_loss", mode="min")
algo = ConcurrencyLimiter(searcher, max_concurrent=4)

[I 2022-07-22 15:23:15,784] A new study created in memory with name: optuna

Running the experiment with a define-by-run search space:

tuner = tune.Tuner(
    objective,
    tune_config=tune.TuneConfig(
        search_alg=algo,
        num_samples=num_samples,
    ),
)
results = tuner.fit()

== Status ==
Current time: 2022-07-22 15:23:58 (running for 00:00:43.15)
Memory usage on this node: 10.4/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects
Result logdir: /Users/kai/ray_results/optuna_example_3
Number of trials: 10/10 (10 TERMINATED)

Trial name	status	loc	activation	height	steps	width	loss	iter	total time (s)	iterations	neg_mean_loss
objective_d363eed6	TERMINATED	127.0.0.1:46290	relu	96.9804	100	6.47293	19.8517	100	10.735	99	-19.8517
objective_d4f3a700	TERMINATED	127.0.0.1:46298	tanh	-76.4387	100	17.7358	-7.58724	100	11.396	99	7.58724
objective_d4f50ce4	TERMINATED	127.0.0.1:46299	relu	-89.8609	100	6.28321	1.17212	100	11.4182	99	-1.17212
objective_d4f67908	TERMINATED	127.0.0.1:46300	relu	67.9266	100	14.0599	16.864	100	11.341	99	-16.864
objective_db5c0402	TERMINATED	127.0.0.1:46317	tanh	-54.13	100	11.4712	-5.32571	100	10.705	99	5.32571
objective_dd487b56	TERMINATED	127.0.0.1:46322	relu	-25.9818	100	0.758497	8.57703	100	10.6916	99	-8.57703
objective_dd4b4e94	TERMINATED	127.0.0.1:46323	relu	54.6085	100	1.2361	16.2163	100	10.6711	99	-16.2163
objective_dd5a8bca	TERMINATED	127.0.0.1:46328	tanh	5.22131	100	17.0952	0.580871	100	10.7017	99	-0.580871
objective_e3482178	TERMINATED	127.0.0.1:46341	tanh	78.844	100	15.5079	7.94912	100	12.6417	99	-7.94912
objective_e532d6e0	TERMINATED	127.0.0.1:46353	tanh	-66.9988	100	7.15087	-6.56059	100	10.7486	99	6.56059

Result for objective_d363eed6:
  date: 2022-07-22_15-23-18
  done: false
  experiment_id: af8f846e60254bc794827e76909df4f0
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 29.698044266420816
  neg_mean_loss: -29.698044266420816
  node_ip: 127.0.0.1
  pid: 46290
  time_since_restore: 0.10497498512268066
  time_this_iter_s: 0.10497498512268066
  time_total_s: 0.10497498512268066
  timestamp: 1658499798
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: d363eed6
  warmup_time: 0.0027971267700195312
  
Result for objective_d4f3a700:
  date: 2022-07-22_15-23-21
  done: false
  experiment_id: 52e5f2a557cf4466939b6c80bb9ff905
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 2.3561338711040003
  neg_mean_loss: -2.3561338711040003
  node_ip: 127.0.0.1
  pid: 46298
  time_since_restore: 0.10145115852355957
  time_this_iter_s: 0.10145115852355957
  time_total_s: 0.10145115852355957
  timestamp: 1658499801
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: d4f3a700
  warmup_time: 0.0033049583435058594
  
Result for objective_d4f67908:
  date: 2022-07-22_15-23-21
  done: false
  experiment_id: 230c6da41a834386be865bb19780b3a5
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 26.792658178391687
  neg_mean_loss: -26.792658178391687
  node_ip: 127.0.0.1
  pid: 46300
  time_since_restore: 0.10428690910339355
  time_this_iter_s: 0.10428690910339355
  time_total_s: 0.10428690910339355
  timestamp: 1658499801
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: d4f67908
  warmup_time: 0.0030798912048339844
  
Result for objective_d4f50ce4:
  date: 2022-07-22_15-23-21
  done: false
  experiment_id: f2396d75f50b417f9fae16d1c9ddcda4
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 11.013906266700227
  neg_mean_loss: -11.013906266700227
  node_ip: 127.0.0.1
  pid: 46299
  time_since_restore: 0.1020050048828125
  time_this_iter_s: 0.1020050048828125
  time_total_s: 0.1020050048828125
  timestamp: 1658499801
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: d4f50ce4
  warmup_time: 0.0027348995208740234
  
Result for objective_d363eed6:
  date: 2022-07-22_15-23-23
  done: false
  experiment_id: af8f846e60254bc794827e76909df4f0
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 20.016284953435374
  neg_mean_loss: -20.016284953435374
  node_ip: 127.0.0.1
  pid: 46290
  time_since_restore: 5.138504981994629
  time_this_iter_s: 0.10720086097717285
  time_total_s: 5.138504981994629
  timestamp: 1658499803
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: d363eed6
  warmup_time: 0.0027971267700195312
  
Result for objective_d4f3a700:
  date: 2022-07-22_15-23-26
  done: false
  experiment_id: 52e5f2a557cf4466939b6c80bb9ff905
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: -7.525324371061028
  neg_mean_loss: 7.525324371061028
  node_ip: 127.0.0.1
  pid: 46298
  time_since_restore: 5.153075218200684
  time_this_iter_s: 0.10515189170837402
  time_total_s: 5.153075218200684
  timestamp: 1658499806
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: d4f3a700
  warmup_time: 0.0033049583435058594
  
Result for objective_d4f67908:
  date: 2022-07-22_15-23-26
  done: false
  experiment_id: 230c6da41a834386be865bb19780b3a5
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 16.94173032408916
  neg_mean_loss: -16.94173032408916
  node_ip: 127.0.0.1
  pid: 46300
  time_since_restore: 5.153704881668091
  time_this_iter_s: 0.10537075996398926
  time_total_s: 5.153704881668091
  timestamp: 1658499806
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: d4f67908
  warmup_time: 0.0030798912048339844
  
Result for objective_d4f50ce4:
  date: 2022-07-22_15-23-26
  done: false
  experiment_id: f2396d75f50b417f9fae16d1c9ddcda4
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 1.3414410369095506
  neg_mean_loss: -1.3414410369095506
  node_ip: 127.0.0.1
  pid: 46299
  time_since_restore: 5.1622138023376465
  time_this_iter_s: 0.10337996482849121
  time_total_s: 5.1622138023376465
  timestamp: 1658499806
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: d4f50ce4
  warmup_time: 0.0027348995208740234
  
Result for objective_d363eed6:
  date: 2022-07-22_15-23-28
  done: false
  experiment_id: af8f846e60254bc794827e76909df4f0
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: 19.85973747786284
  neg_mean_loss: -19.85973747786284
  node_ip: 127.0.0.1
  pid: 46290
  time_since_restore: 10.1943039894104
  time_this_iter_s: 0.12109899520874023
  time_total_s: 10.1943039894104
  timestamp: 1658499808
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: d363eed6
  warmup_time: 0.0027971267700195312
  
Result for objective_d363eed6:
  date: 2022-07-22_15-23-29
  done: true
  experiment_id: af8f846e60254bc794827e76909df4f0
  experiment_tag: 1_activation=relu,height=96.9804,steps=100,width=6.4729
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 19.851696631549423
  neg_mean_loss: -19.851696631549423
  node_ip: 127.0.0.1
  pid: 46290
  time_since_restore: 10.734967947006226
  time_this_iter_s: 0.10534882545471191
  time_total_s: 10.734967947006226
  timestamp: 1658499809
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: d363eed6
  warmup_time: 0.0027971267700195312
  
Result for objective_d4f67908:
  date: 2022-07-22_15-23-31
  done: false
  experiment_id: 230c6da41a834386be865bb19780b3a5
  hostname: Kais-MacBook-Pro.local
  iterations: 93
  iterations_since_restore: 94
  mean_loss: 16.86855533914186
  neg_mean_loss: -16.86855533914186
  node_ip: 127.0.0.1
  pid: 46300
  time_since_restore: 10.096334218978882
  time_this_iter_s: 0.10534310340881348
  time_total_s: 10.096334218978882
  timestamp: 1658499811
  timesteps_since_restore: 0
  training_iteration: 94
  trial_id: d4f67908
  warmup_time: 0.0030798912048339844
  
Result for objective_d4f50ce4:
  date: 2022-07-22_15-23-31
  done: false
  experiment_id: f2396d75f50b417f9fae16d1c9ddcda4
  hostname: Kais-MacBook-Pro.local
  iterations: 92
  iterations_since_restore: 93
  mean_loss: 1.183958166642718
  neg_mean_loss: -1.183958166642718
  node_ip: 127.0.0.1
  pid: 46299
  time_since_restore: 9.976859092712402
  time_this_iter_s: 0.1057121753692627
  time_total_s: 9.976859092712402
  timestamp: 1658499811
  timesteps_since_restore: 0
  training_iteration: 93
  trial_id: d4f50ce4
  warmup_time: 0.0027348995208740234
  
Result for objective_d4f3a700:
  date: 2022-07-22_15-23-31
  done: false
  experiment_id: 52e5f2a557cf4466939b6c80bb9ff905
  hostname: Kais-MacBook-Pro.local
  iterations: 93
  iterations_since_restore: 94
  mean_loss: -7.583604593881161
  neg_mean_loss: 7.583604593881161
  node_ip: 127.0.0.1
  pid: 46298
  time_since_restore: 10.091833114624023
  time_this_iter_s: 0.10643887519836426
  time_total_s: 10.091833114624023
  timestamp: 1658499811
  timesteps_since_restore: 0
  training_iteration: 94
  trial_id: d4f3a700
  warmup_time: 0.0033049583435058594
  
Result for objective_db5c0402:
  date: 2022-07-22_15-23-31
  done: false
  experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 4.587004175501654
  neg_mean_loss: -4.587004175501654
  node_ip: 127.0.0.1
  pid: 46317
  time_since_restore: 0.10449004173278809
  time_this_iter_s: 0.10449004173278809
  time_total_s: 0.10449004173278809
  timestamp: 1658499811
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: db5c0402
  warmup_time: 0.003406047821044922
  
Result for objective_d4f67908:
  date: 2022-07-22_15-23-32
  done: true
  experiment_id: 230c6da41a834386be865bb19780b3a5
  experiment_tag: 4_activation=relu,height=67.9266,steps=100,width=14.0599
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 16.86398832185618
  neg_mean_loss: -16.86398832185618
  node_ip: 127.0.0.1
  pid: 46300
  time_since_restore: 11.341041088104248
  time_this_iter_s: 0.10513019561767578
  time_total_s: 11.341041088104248
  timestamp: 1658499812
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: d4f67908
  warmup_time: 0.0030798912048339844
  
Result for objective_d4f3a700:
  date: 2022-07-22_15-23-32
  done: true
  experiment_id: 52e5f2a557cf4466939b6c80bb9ff905
  experiment_tag: 2_activation=tanh,height=-76.4387,steps=100,width=17.7358
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -7.587236125634034
  neg_mean_loss: 7.587236125634034
  node_ip: 127.0.0.1
  pid: 46298
  time_since_restore: 11.396040916442871
  time_this_iter_s: 0.10666394233703613
  time_total_s: 11.396040916442871
  timestamp: 1658499812
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: d4f3a700
  warmup_time: 0.0033049583435058594
  
Result for objective_d4f50ce4:
  date: 2022-07-22_15-23-32
  done: true
  experiment_id: f2396d75f50b417f9fae16d1c9ddcda4
  experiment_tag: 3_activation=relu,height=-89.8609,steps=100,width=6.2832
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 1.1721245345451354
  neg_mean_loss: -1.1721245345451354
  node_ip: 127.0.0.1
  pid: 46299
  time_since_restore: 11.418162822723389
  time_this_iter_s: 0.11828303337097168
  time_total_s: 11.418162822723389
  timestamp: 1658499812
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: d4f50ce4
  warmup_time: 0.0027348995208740234
  

Result for objective_dd4b4e94:
  date: 2022-07-22_15-23-35
  done: false
  experiment_id: c6036d4a26674f86946fd7c8c895e85c
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 25.460850093028082
  neg_mean_loss: -25.460850093028082
  node_ip: 127.0.0.1
  pid: 46323
  time_since_restore: 0.10411691665649414
  time_this_iter_s: 0.10411691665649414
  time_total_s: 0.10411691665649414
  timestamp: 1658499815
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: dd4b4e94
  warmup_time: 0.003142118453979492
  
Result for objective_dd487b56:
  date: 2022-07-22_15-23-35
  done: false
  experiment_id: 75e118a5733e454fab54f4a20c036852
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 17.401819651158622
  neg_mean_loss: -17.401819651158622
  node_ip: 127.0.0.1
  pid: 46322
  time_since_restore: 0.1048891544342041
  time_this_iter_s: 0.1048891544342041
  time_total_s: 0.1048891544342041
  timestamp: 1658499815
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: dd487b56
  warmup_time: 0.0028679370880126953
  
Result for objective_dd5a8bca:
  date: 2022-07-22_15-23-35
  done: false
  experiment_id: 775683e6d95a4705846a37e809b474d7
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 10.522131438315931
  neg_mean_loss: -10.522131438315931
  node_ip: 127.0.0.1
  pid: 46328
  time_since_restore: 0.10494732856750488
  time_this_iter_s: 0.10494732856750488
  time_total_s: 0.10494732856750488
  timestamp: 1658499815
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: dd5a8bca
  warmup_time: 0.002931833267211914
  
Result for objective_db5c0402:
  date: 2022-07-22_15-23-36
  done: false
  experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: -5.230895562358142
  neg_mean_loss: 5.230895562358142
  node_ip: 127.0.0.1
  pid: 46317
  time_since_restore: 5.120880842208862
  time_this_iter_s: 0.1076059341430664
  time_total_s: 5.120880842208862
  timestamp: 1658499816
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: db5c0402
  warmup_time: 0.003406047821044922
  
Result for objective_dd4b4e94:
  date: 2022-07-22_15-23-40
  done: false
  experiment_id: c6036d4a26674f86946fd7c8c895e85c
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 16.929346028466988
  neg_mean_loss: -16.929346028466988
  node_ip: 127.0.0.1
  pid: 46323
  time_since_restore: 5.137623071670532
  time_this_iter_s: 0.10670709609985352
  time_total_s: 5.137623071670532
  timestamp: 1658499820
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: dd4b4e94
  warmup_time: 0.003142118453979492
  
Result for objective_dd487b56:
  date: 2022-07-22_15-23-40
  done: false
  experiment_id: 75e118a5733e454fab54f4a20c036852
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 9.592430453048937
  neg_mean_loss: -9.592430453048937
  node_ip: 127.0.0.1
  pid: 46322
  time_since_restore: 5.140692949295044
  time_this_iter_s: 0.1086587905883789
  time_total_s: 5.140692949295044
  timestamp: 1658499820
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: dd487b56
  warmup_time: 0.0028679370880126953
  
Result for objective_dd5a8bca:
  date: 2022-07-22_15-23-40
  done: false
  experiment_id: 775683e6d95a4705846a37e809b474d7
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  mean_loss: 0.6450610360030851
  neg_mean_loss: -0.6450610360030851
  node_ip: 127.0.0.1
  pid: 46328
  time_since_restore: 5.138496160507202
  time_this_iter_s: 0.10809993743896484
  time_total_s: 5.138496160507202
  timestamp: 1658499820
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: dd5a8bca
  warmup_time: 0.002931833267211914
  
Result for objective_db5c0402:
  date: 2022-07-22_15-23-41
  done: false
  experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: -5.321109063262834
  neg_mean_loss: 5.321109063262834
  node_ip: 127.0.0.1
  pid: 46317
  time_since_restore: 10.162945985794067
  time_this_iter_s: 0.10671615600585938
  time_total_s: 10.162945985794067
  timestamp: 1658499821
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: db5c0402
  warmup_time: 0.003406047821044922
  
Result for objective_db5c0402:
  date: 2022-07-22_15-23-42
  done: true
  experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8
  experiment_tag: 5_activation=tanh,height=-54.1300,steps=100,width=11.4712
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -5.325709301325668
  neg_mean_loss: 5.325709301325668
  node_ip: 127.0.0.1
  pid: 46317
  time_since_restore: 10.704953908920288
  time_this_iter_s: 0.1082158088684082
  time_total_s: 10.704953908920288
  timestamp: 1658499822
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: db5c0402
  warmup_time: 0.003406047821044922
  
Result for objective_dd4b4e94:
  date: 2022-07-22_15-23-45
  done: false
  experiment_id: c6036d4a26674f86946fd7c8c895e85c
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: 16.253282229309562
  neg_mean_loss: -16.253282229309562
  node_ip: 127.0.0.1
  pid: 46323
  time_since_restore: 10.140441179275513
  time_this_iter_s: 0.10501408576965332
  time_total_s: 10.140441179275513
  timestamp: 1658499825
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: dd4b4e94
  warmup_time: 0.003142118453979492
  
Result for objective_dd487b56:
  date: 2022-07-22_15-23-45
  done: false
  experiment_id: 75e118a5733e454fab54f4a20c036852
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: 8.631851056235014
  neg_mean_loss: -8.631851056235014
  node_ip: 127.0.0.1
  pid: 46322
  time_since_restore: 10.156053066253662
  time_this_iter_s: 0.10682201385498047
  time_total_s: 10.156053066253662
  timestamp: 1658499825
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: dd487b56
  warmup_time: 0.0028679370880126953
  
Result for objective_e3482178:
  date: 2022-07-22_15-23-45
  done: false
  experiment_id: 6709b686be234eb996457210f3efbe08
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 17.884403838898738
  neg_mean_loss: -17.884403838898738
  node_ip: 127.0.0.1
  pid: 46341
  time_since_restore: 0.10499191284179688
  time_this_iter_s: 0.10499191284179688
  time_total_s: 0.10499191284179688
  timestamp: 1658499825
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: e3482178
  warmup_time: 0.002877950668334961
  
Result for objective_dd5a8bca:
  date: 2022-07-22_15-23-45
  done: false
  experiment_id: 775683e6d95a4705846a37e809b474d7
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  mean_loss: 0.583976365762005
  neg_mean_loss: -0.583976365762005
  node_ip: 127.0.0.1
  pid: 46328
  time_since_restore: 10.165286302566528
  time_this_iter_s: 0.10989904403686523
  time_total_s: 10.165286302566528
  timestamp: 1658499825
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: dd5a8bca
  warmup_time: 0.002931833267211914
  
Result for objective_dd4b4e94:
  date: 2022-07-22_15-23-45
  done: true
  experiment_id: c6036d4a26674f86946fd7c8c895e85c
  experiment_tag: 7_activation=relu,height=54.6085,steps=100,width=1.2361
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 16.216283787762706
  neg_mean_loss: -16.216283787762706
  node_ip: 127.0.0.1
  pid: 46323
  time_since_restore: 10.67109203338623
  time_this_iter_s: 0.1058969497680664
  time_total_s: 10.67109203338623
  timestamp: 1658499825
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: dd4b4e94
  warmup_time: 0.003142118453979492
  
Result for objective_dd487b56:
  date: 2022-07-22_15-23-45
  done: true
  experiment_id: 75e118a5733e454fab54f4a20c036852
  experiment_tag: 6_activation=relu,height=-25.9818,steps=100,width=0.7585
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 8.577028987245935
  neg_mean_loss: -8.577028987245935
  node_ip: 127.0.0.1
  pid: 46322
  time_since_restore: 10.691627979278564
  time_this_iter_s: 0.10821914672851562
  time_total_s: 10.691627979278564
  timestamp: 1658499825
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: dd487b56
  warmup_time: 0.0028679370880126953
  
Result for objective_dd5a8bca:
  date: 2022-07-22_15-23-45
  done: true
  experiment_id: 775683e6d95a4705846a37e809b474d7
  experiment_tag: 8_activation=tanh,height=5.2213,steps=100,width=17.0952
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 0.5808712318418436
  neg_mean_loss: -0.5808712318418436
  node_ip: 127.0.0.1
  pid: 46328
  time_since_restore: 10.701741218566895
  time_this_iter_s: 0.11321616172790527
  time_total_s: 10.701741218566895
  timestamp: 1658499825
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: dd5a8bca
  warmup_time: 0.002931833267211914
  

Result for objective_e532d6e0:
  date: 2022-07-22_15-23-48
  done: false
  experiment_id: 34d05fd425474b82a450c22c2a063843
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  mean_loss: 3.30012197692201
  neg_mean_loss: -3.30012197692201
  node_ip: 127.0.0.1
  pid: 46353
  time_since_restore: 0.10400509834289551
  time_this_iter_s: 0.10400509834289551
  time_total_s: 0.10400509834289551
  timestamp: 1658499828
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: e532d6e0
  warmup_time: 0.0028901100158691406
  
Result for objective_e3482178:
  date: 2022-07-22_15-23-50
  done: false
  experiment_id: 6709b686be234eb996457210f3efbe08
  hostname: Kais-MacBook-Pro.local
  iterations: 29
  iterations_since_restore: 30
  mean_loss: 8.10192352359083
  neg_mean_loss: -8.10192352359083
  node_ip: 127.0.0.1
  pid: 46341
  time_since_restore: 5.183844089508057
  time_this_iter_s: 0.10574007034301758
  time_total_s: 5.183844089508057
  timestamp: 1658499830
  timesteps_since_restore: 0
  training_iteration: 30
  trial_id: e3482178
  warmup_time: 0.002877950668334961
  
Result for objective_e532d6e0:
  date: 2022-07-22_15-23-53
  done: false
  experiment_id: 34d05fd425474b82a450c22c2a063843
  hostname: Kais-MacBook-Pro.local
  iterations: 48
  iterations_since_restore: 49
  mean_loss: -6.416785967274913
  neg_mean_loss: 6.416785967274913
  node_ip: 127.0.0.1
  pid: 46353
  time_since_restore: 5.212116003036499
  time_this_iter_s: 0.10714387893676758
  time_total_s: 5.212116003036499
  timestamp: 1658499833
  timesteps_since_restore: 0
  training_iteration: 49
  trial_id: e532d6e0
  warmup_time: 0.0028901100158691406
  
Result for objective_e3482178:
  date: 2022-07-22_15-23-55
  done: false
  experiment_id: 6709b686be234eb996457210f3efbe08
  hostname: Kais-MacBook-Pro.local
  iterations: 76
  iterations_since_restore: 77
  mean_loss: 7.968536513457016
  neg_mean_loss: -7.968536513457016
  node_ip: 127.0.0.1
  pid: 46341
  time_since_restore: 10.190052032470703
  time_this_iter_s: 0.10843491554260254
  time_total_s: 10.190052032470703
  timestamp: 1658499835
  timesteps_since_restore: 0
  training_iteration: 77
  trial_id: e3482178
  warmup_time: 0.002877950668334961
  
Result for objective_e3482178:
  date: 2022-07-22_15-23-57
  done: true
  experiment_id: 6709b686be234eb996457210f3efbe08
  experiment_tag: 9_activation=tanh,height=78.8440,steps=100,width=15.5079
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: 7.9491170267942515
  neg_mean_loss: -7.9491170267942515
  node_ip: 127.0.0.1
  pid: 46341
  time_since_restore: 12.64172887802124
  time_this_iter_s: 0.10761594772338867
  time_total_s: 12.64172887802124
  timestamp: 1658499837
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: e3482178
  warmup_time: 0.002877950668334961
  
Result for objective_e532d6e0:
  date: 2022-07-22_15-23-58
  done: false
  experiment_id: 34d05fd425474b82a450c22c2a063843
  hostname: Kais-MacBook-Pro.local
  iterations: 95
  iterations_since_restore: 96
  mean_loss: -6.554810275125985
  neg_mean_loss: 6.554810275125985
  node_ip: 127.0.0.1
  pid: 46353
  time_since_restore: 10.305040121078491
  time_this_iter_s: 0.10702300071716309
  time_total_s: 10.305040121078491
  timestamp: 1658499838
  timesteps_since_restore: 0
  training_iteration: 96
  trial_id: e532d6e0
  warmup_time: 0.0028901100158691406
  
Result for objective_e532d6e0:
  date: 2022-07-22_15-23-58
  done: true
  experiment_id: 34d05fd425474b82a450c22c2a063843
  experiment_tag: 10_activation=tanh,height=-66.9988,steps=100,width=7.1509
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  mean_loss: -6.560589957038967
  neg_mean_loss: 6.560589957038967
  node_ip: 127.0.0.1
  pid: 46353
  time_since_restore: 10.74857211112976
  time_this_iter_s: 0.1075899600982666
  time_total_s: 10.74857211112976
  timestamp: 1658499838
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: e532d6e0
  warmup_time: 0.0028901100158691406
  

We take a look again at the optimal hyperparameters.

print("Best hyperparameters for loss found were: ", results.get_best_result("mean_loss", "min").config)

Best hyperparameters for loss found were:  {'activation': 'tanh', 'width': 17.73584230792165, 'height': -76.43866128895999, 'steps': 100}

Multi-objective optimization#

Finally, let’s take a look at the multi-objective case.

def multi_objective(config):
    # Hyperparameters
    width, height = config["width"], config["height"]

    for step in range(config["steps"]):
        # Iterative training function - can be any arbitrary training procedure
        intermediate_score = evaluate(step, config["width"], config["height"], config["activation"])
        # Feed the score back back to Tune.
        train.report({
           "iterations": step, "loss": intermediate_score, "gain": intermediate_score * width
        })

We define the OptunaSearch object this time with metric and mode as list arguments.

searcher = OptunaSearch(metric=["loss", "gain"], mode=["min", "max"])
algo = ConcurrencyLimiter(searcher, max_concurrent=4)

tuner = tune.Tuner(
    multi_objective,
    tune_config=tune.TuneConfig(
        search_alg=algo,
        num_samples=num_samples,
    ),
    param_space=search_space
)
results = tuner.fit()

[I 2022-07-22 15:26:50,680] A new study created in memory with name: optuna

== Status ==
Current time: 2022-07-22 15:27:34 (running for 00:00:43.55)
Memory usage on this node: 8.2/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects
Result logdir: /Users/kai/ray_results/multi_objective_2022-07-22_15-26-50
Number of trials: 10/10 (10 TERMINATED)

Trial name	status	loc	activation	height	width	iter	total time (s)	iterations	loss	gain
multi_objective_5378f1e8	TERMINATED	127.0.0.1:46621	tanh	-47.5236	10.9256	100	10.7064	99	-4.66076	-50.9216
multi_objective_550962f4	TERMINATED	127.0.0.1:46631	relu	-40.6828	4.34953	100	11.3948	99	6.15869	26.7874
multi_objective_550ad9a4	TERMINATED	127.0.0.1:46632	tanh	-42.0563	2.02124	100	11.2755	99	-3.72967	-7.53857
multi_objective_550c4faa	TERMINATED	127.0.0.1:46633	relu	-56.2838	5.65237	100	11.407	99	4.54719	25.7024
multi_objective_5b6c4ba2	TERMINATED	127.0.0.1:46679	relu	54.9133	16.9039	100	10.6698	99	15.5507	262.867
multi_objective_5d51f2d2	TERMINATED	127.0.0.1:46686	tanh	16.5793	5.16709	100	10.7451	99	1.84967	9.55744
multi_objective_5d636760	TERMINATED	127.0.0.1:46689	tanh	-13.8895	13.6043	100	10.6897	99	-1.31525	-17.8931
multi_objective_5d66093e	TERMINATED	127.0.0.1:46690	relu	-51.6714	19.8211	100	10.7246	99	4.88357	96.7974
multi_objective_634c9444	TERMINATED	127.0.0.1:46710	relu	-12.3384	15.1097	100	13.1986	99	8.83256	133.457
multi_objective_65469a2e	TERMINATED	127.0.0.1:46719	tanh	19.6774	13.6172	100	10.7279	99	2.04137	27.7978

Result for multi_objective_5378f1e8:
  date: 2022-07-22_15-26-53
  done: false
  experiment_id: 4ce03cab420a449a9f4827c3b6b76c38
  gain: 57.33359096413655
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  loss: 5.247638036988612
  node_ip: 127.0.0.1
  pid: 46621
  time_since_restore: 0.10418081283569336
  time_this_iter_s: 0.10418081283569336
  time_total_s: 0.10418081283569336
  timestamp: 1658500013
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 5378f1e8
  warmup_time: 0.002785921096801758
  
Result for multi_objective_550ad9a4:
  date: 2022-07-22_15-26-56
  done: false
  experiment_id: 102a4b06449b428b9870a75d02812be6
  gain: 11.711822825618256
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  loss: 5.794371166191576
  node_ip: 127.0.0.1
  pid: 46632
  time_since_restore: 0.10288000106811523
  time_this_iter_s: 0.10288000106811523
  time_total_s: 0.10288000106811523
  timestamp: 1658500016
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 550ad9a4
  warmup_time: 0.0030989646911621094
  
Result for multi_objective_550c4faa:
  date: 2022-07-22_15-26-56
  done: false
  experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d
  gain: 81.23378320141829
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  loss: 14.371624912716872
  node_ip: 127.0.0.1
  pid: 46633
  time_since_restore: 0.10215306282043457
  time_this_iter_s: 0.10215306282043457
  time_total_s: 0.10215306282043457
  timestamp: 1658500016
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 550c4faa
  warmup_time: 0.0036542415618896484
  
Result for multi_objective_550962f4:
  date: 2022-07-22_15-26-56
  done: false
  experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64
  gain: 69.29543163212045
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  loss: 15.93172358423837
  node_ip: 127.0.0.1
  pid: 46631
  time_since_restore: 0.10244321823120117
  time_this_iter_s: 0.10244321823120117
  time_total_s: 0.10244321823120117
  timestamp: 1658500016
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 550962f4
  warmup_time: 0.003384828567504883
  
Result for multi_objective_5378f1e8:
  date: 2022-07-22_15-26-58
  done: false
  experiment_id: 4ce03cab420a449a9f4827c3b6b76c38
  gain: -49.83538614085625
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  loss: -4.561341152769549
  node_ip: 127.0.0.1
  pid: 46621
  time_since_restore: 5.1368408203125
  time_this_iter_s: 0.1180417537689209
  time_total_s: 5.1368408203125
  timestamp: 1658500018
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 5378f1e8
  warmup_time: 0.002785921096801758
  
Result for multi_objective_550ad9a4:
  date: 2022-07-22_15-27-01
  done: false
  experiment_id: 102a4b06449b428b9870a75d02812be6
  gain: -6.575568930702184
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  loss: -3.2532328725144097
  node_ip: 127.0.0.1
  pid: 46632
  time_since_restore: 5.146106719970703
  time_this_iter_s: 0.10509181022644043
  time_total_s: 5.146106719970703
  timestamp: 1658500021
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 550ad9a4
  warmup_time: 0.0030989646911621094
  
Result for multi_objective_550c4faa:
  date: 2022-07-22_15-27-01
  done: false
  experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d
  gain: 26.760529995257745
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  loss: 4.73438862995905
  node_ip: 127.0.0.1
  pid: 46633
  time_since_restore: 5.1661481857299805
  time_this_iter_s: 0.10656309127807617
  time_total_s: 5.1661481857299805
  timestamp: 1658500021
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 550c4faa
  warmup_time: 0.0036542415618896484
  
Result for multi_objective_550962f4:
  date: 2022-07-22_15-27-01
  done: false
  experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64
  gain: 27.828615153405433
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  loss: 6.398081286367129
  node_ip: 127.0.0.1
  pid: 46631
  time_since_restore: 5.169018983840942
  time_this_iter_s: 0.10616683959960938
  time_total_s: 5.169018983840942
  timestamp: 1658500021
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 550962f4
  warmup_time: 0.003384828567504883
  
Result for multi_objective_5378f1e8:
  date: 2022-07-22_15-27-03
  done: false
  experiment_id: 4ce03cab420a449a9f4827c3b6b76c38
  gain: -50.86883186480195
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  loss: -4.65593053743818
  node_ip: 127.0.0.1
  pid: 46621
  time_since_restore: 10.17048692703247
  time_this_iter_s: 0.10630917549133301
  time_total_s: 10.17048692703247
  timestamp: 1658500023
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: 5378f1e8
  warmup_time: 0.002785921096801758
  
Result for multi_objective_5378f1e8:
  date: 2022-07-22_15-27-04
  done: true
  experiment_id: 4ce03cab420a449a9f4827c3b6b76c38
  experiment_tag: 1_activation=tanh,height=-47.5236,steps=100,width=10.9256
  gain: -50.92155508732405
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  loss: -4.66075619693878
  node_ip: 127.0.0.1
  pid: 46621
  time_since_restore: 10.706397771835327
  time_this_iter_s: 0.10694766044616699
  time_total_s: 10.706397771835327
  timestamp: 1658500024
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 5378f1e8
  warmup_time: 0.002785921096801758
  
Result for multi_objective_550ad9a4:
  date: 2022-07-22_15-27-05
  done: false
  experiment_id: 102a4b06449b428b9870a75d02812be6
  gain: -7.479635489825483
  hostname: Kais-MacBook-Pro.local
  iterations: 93
  iterations_since_restore: 94
  loss: -3.700515697784228
  node_ip: 127.0.0.1
  pid: 46632
  time_since_restore: 10.056212902069092
  time_this_iter_s: 0.10589313507080078
  time_total_s: 10.056212902069092
  timestamp: 1658500025
  timesteps_since_restore: 0
  training_iteration: 94
  trial_id: 550ad9a4
  warmup_time: 0.0030989646911621094
  
Result for multi_objective_550c4faa:
  date: 2022-07-22_15-27-05
  done: false
  experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d
  gain: 25.776502806907246
  hostname: Kais-MacBook-Pro.local
  iterations: 92
  iterations_since_restore: 93
  loss: 4.5602976409942295
  node_ip: 127.0.0.1
  pid: 46633
  time_since_restore: 9.99274206161499
  time_this_iter_s: 0.10583209991455078
  time_total_s: 9.99274206161499
  timestamp: 1658500025
  timesteps_since_restore: 0
  training_iteration: 93
  trial_id: 550c4faa
  warmup_time: 0.0036542415618896484
  
Result for multi_objective_550962f4:
  date: 2022-07-22_15-27-05
  done: false
  experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64
  gain: 26.860636112316993
  hostname: Kais-MacBook-Pro.local
  iterations: 92
  iterations_since_restore: 93
  loss: 6.175533072801938
  node_ip: 127.0.0.1
  pid: 46631
  time_since_restore: 9.971143007278442
  time_this_iter_s: 0.1054527759552002
  time_total_s: 9.971143007278442
  timestamp: 1658500025
  timesteps_since_restore: 0
  training_iteration: 93
  trial_id: 550962f4
  warmup_time: 0.003384828567504883
  
Result for multi_objective_5b6c4ba2:
  date: 2022-07-22_15-27-06
  done: false
  experiment_id: f143c125a0734166ac832270ac08dbb6
  gain: 430.90195437776714
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  loss: 25.491329340230966
  node_ip: 127.0.0.1
  pid: 46679
  time_since_restore: 0.10215902328491211
  time_this_iter_s: 0.10215902328491211
  time_total_s: 0.10215902328491211
  timestamp: 1658500026
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 5b6c4ba2
  warmup_time: 0.002663135528564453
  
Result for multi_objective_550ad9a4:
  date: 2022-07-22_15-27-07
  done: true
  experiment_id: 102a4b06449b428b9870a75d02812be6
  experiment_tag: 3_activation=tanh,height=-42.0563,steps=100,width=2.0212
  gain: -7.538566547680232
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  loss: -3.729671571085163
  node_ip: 127.0.0.1
  pid: 46632
  time_since_restore: 11.27550482749939
  time_this_iter_s: 0.10642290115356445
  time_total_s: 11.27550482749939
  timestamp: 1658500027
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 550ad9a4
  warmup_time: 0.0030989646911621094
  
Result for multi_objective_550962f4:
  date: 2022-07-22_15-27-07
  done: true
  experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64
  experiment_tag: 2_activation=relu,height=-40.6828,steps=100,width=4.3495
  gain: 26.787356208375382
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  loss: 6.15868527856241
  node_ip: 127.0.0.1
  pid: 46631
  time_since_restore: 11.394789934158325
  time_this_iter_s: 0.10491585731506348
  time_total_s: 11.394789934158325
  timestamp: 1658500027
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 550962f4
  warmup_time: 0.003384828567504883
  
Result for multi_objective_550c4faa:
  date: 2022-07-22_15-27-07
  done: true
  experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d
  experiment_tag: 4_activation=relu,height=-56.2838,steps=100,width=5.6524
  gain: 25.702421216002982
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  loss: 4.547191359402326
  node_ip: 127.0.0.1
  pid: 46633
  time_since_restore: 11.406961917877197
  time_this_iter_s: 0.12177181243896484
  time_total_s: 11.406961917877197
  timestamp: 1658500027
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 550c4faa
  warmup_time: 0.0036542415618896484
  

Result for multi_objective_5d51f2d2:
  date: 2022-07-22_15-27-09
  done: false
  experiment_id: 1433194530b14db4b77ce40a69d65407
  gain: 60.23765295445784
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  loss: 11.657934184433229
  node_ip: 127.0.0.1
  pid: 46686
  time_since_restore: 0.10274410247802734
  time_this_iter_s: 0.10274410247802734
  time_total_s: 0.10274410247802734
  timestamp: 1658500029
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 5d51f2d2
  warmup_time: 0.0028769969940185547
  
Result for multi_objective_5d636760:
  date: 2022-07-22_15-27-10
  done: false
  experiment_id: b8dccac5d8914569b1c2ca061ad47a56
  gain: 117.14725547845826
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  loss: 8.611046741256526
  node_ip: 127.0.0.1
  pid: 46689
  time_since_restore: 0.10460686683654785
  time_this_iter_s: 0.10460686683654785
  time_total_s: 0.10460686683654785
  timestamp: 1658500030
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 5d636760
  warmup_time: 0.0028090476989746094
  
Result for multi_objective_5d66093e:
  date: 2022-07-22_15-27-10
  done: false
  experiment_id: e7259e986278458899b46b485e069ec9
  gain: 294.00300634729234
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  loss: 14.832864636325777
  node_ip: 127.0.0.1
  pid: 46690
  time_since_restore: 0.10294318199157715
  time_this_iter_s: 0.10294318199157715
  time_total_s: 0.10294318199157715
  timestamp: 1658500030
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 5d66093e
  warmup_time: 0.0027511119842529297
  
Result for multi_objective_5b6c4ba2:
  date: 2022-07-22_15-27-11
  done: false
  experiment_id: f143c125a0734166ac832270ac08dbb6
  gain: 263.92128797150474
  hostname: Kais-MacBook-Pro.local
  iterations: 48
  iterations_since_restore: 49
  loss: 15.6130748612977
  node_ip: 127.0.0.1
  pid: 46679
  time_since_restore: 5.2078468799591064
  time_this_iter_s: 0.10803008079528809
  time_total_s: 5.2078468799591064
  timestamp: 1658500031
  timesteps_since_restore: 0
  training_iteration: 49
  trial_id: 5b6c4ba2
  warmup_time: 0.002663135528564453
  
Result for multi_objective_5d51f2d2:
  date: 2022-07-22_15-27-14
  done: false
  experiment_id: 1433194530b14db4b77ce40a69d65407
  gain: 10.610216987384797
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  loss: 2.053420165872076
  node_ip: 127.0.0.1
  pid: 46686
  time_since_restore: 5.178762912750244
  time_this_iter_s: 0.10710978507995605
  time_total_s: 5.178762912750244
  timestamp: 1658500034
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 5d51f2d2
  warmup_time: 0.0028769969940185547
  
Result for multi_objective_5d636760:
  date: 2022-07-22_15-27-15
  done: false
  experiment_id: b8dccac5d8914569b1c2ca061ad47a56
  gain: -16.800839364465826
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  loss: -1.234965449842409
  node_ip: 127.0.0.1
  pid: 46689
  time_since_restore: 5.146665811538696
  time_this_iter_s: 0.10526585578918457
  time_total_s: 5.146665811538696
  timestamp: 1658500035
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 5d636760
  warmup_time: 0.0028090476989746094
  
Result for multi_objective_5d66093e:
  date: 2022-07-22_15-27-15
  done: false
  experiment_id: e7259e986278458899b46b485e069ec9
  gain: 97.89753254610552
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  loss: 4.939068026982622
  node_ip: 127.0.0.1
  pid: 46690
  time_since_restore: 5.1532862186431885
  time_this_iter_s: 0.10752320289611816
  time_total_s: 5.1532862186431885
  timestamp: 1658500035
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 5d66093e
  warmup_time: 0.0027511119842529297
  
Result for multi_objective_5b6c4ba2:
  date: 2022-07-22_15-27-16
  done: false
  experiment_id: f143c125a0734166ac832270ac08dbb6
  gain: 262.90943552209455
  hostname: Kais-MacBook-Pro.local
  iterations: 95
  iterations_since_restore: 96
  loss: 15.553215620072208
  node_ip: 127.0.0.1
  pid: 46679
  time_since_restore: 10.240519762039185
  time_this_iter_s: 0.10443782806396484
  time_total_s: 10.240519762039185
  timestamp: 1658500036
  timesteps_since_restore: 0
  training_iteration: 96
  trial_id: 5b6c4ba2
  warmup_time: 0.002663135528564453
  
Result for multi_objective_5b6c4ba2:
  date: 2022-07-22_15-27-17
  done: true
  experiment_id: f143c125a0734166ac832270ac08dbb6
  experiment_tag: 5_activation=relu,height=54.9133,steps=100,width=16.9039
  gain: 262.86741923011425
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  loss: 15.550730017197484
  node_ip: 127.0.0.1
  pid: 46679
  time_since_restore: 10.669770956039429
  time_this_iter_s: 0.10598993301391602
  time_total_s: 10.669770956039429
  timestamp: 1658500037
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 5b6c4ba2
  warmup_time: 0.002663135528564453
  
Result for multi_objective_634c9444:
  date: 2022-07-22_15-27-19
  done: false
  experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee
  gain: 283.55076351521427
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  loss: 18.766157678691147
  node_ip: 127.0.0.1
  pid: 46710
  time_since_restore: 0.10465264320373535
  time_this_iter_s: 0.10465264320373535
  time_total_s: 0.10465264320373535
  timestamp: 1658500039
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 634c9444
  warmup_time: 0.004335880279541016
  
Result for multi_objective_5d51f2d2:
  date: 2022-07-22_15-27-20
  done: false
  experiment_id: 1433194530b14db4b77ce40a69d65407
  gain: 9.60907229177291
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  loss: 1.8596662860626851
  node_ip: 127.0.0.1
  pid: 46686
  time_since_restore: 10.218497037887573
  time_this_iter_s: 0.10553288459777832
  time_total_s: 10.218497037887573
  timestamp: 1658500040
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: 5d51f2d2
  warmup_time: 0.0028769969940185547
  
Result for multi_objective_5d636760:
  date: 2022-07-22_15-27-20
  done: false
  experiment_id: b8dccac5d8914569b1c2ca061ad47a56
  gain: -17.840160182810624
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  loss: -1.311361948500235
  node_ip: 127.0.0.1
  pid: 46689
  time_since_restore: 10.16199278831482
  time_this_iter_s: 0.10700488090515137
  time_total_s: 10.16199278831482
  timestamp: 1658500040
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: 5d636760
  warmup_time: 0.0028090476989746094
  
Result for multi_objective_5d66093e:
  date: 2022-07-22_15-27-20
  done: false
  experiment_id: e7259e986278458899b46b485e069ec9
  gain: 96.85061995053432
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  loss: 4.886249816008858
  node_ip: 127.0.0.1
  pid: 46690
  time_since_restore: 10.197196006774902
  time_this_iter_s: 0.10463690757751465
  time_total_s: 10.197196006774902
  timestamp: 1658500040
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: 5d66093e
  warmup_time: 0.0027511119842529297
  
Result for multi_objective_5d51f2d2:
  date: 2022-07-22_15-27-20
  done: true
  experiment_id: 1433194530b14db4b77ce40a69d65407
  experiment_tag: 6_activation=tanh,height=16.5793,steps=100,width=5.1671
  gain: 9.557436804323405
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  loss: 1.8496731491334866
  node_ip: 127.0.0.1
  pid: 46686
  time_since_restore: 10.745142936706543
  time_this_iter_s: 0.10509181022644043
  time_total_s: 10.745142936706543
  timestamp: 1658500040
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 5d51f2d2
  warmup_time: 0.0028769969940185547
  
Result for multi_objective_5d636760:
  date: 2022-07-22_15-27-20
  done: true
  experiment_id: b8dccac5d8914569b1c2ca061ad47a56
  experiment_tag: 7_activation=tanh,height=-13.8895,steps=100,width=13.6043
  gain: -17.893079155032332
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  loss: -1.3152518197690004
  node_ip: 127.0.0.1
  pid: 46689
  time_since_restore: 10.689707040786743
  time_this_iter_s: 0.10322403907775879
  time_total_s: 10.689707040786743
  timestamp: 1658500040
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 5d636760
  warmup_time: 0.0028090476989746094
  
Result for multi_objective_5d66093e:
  date: 2022-07-22_15-27-20
  done: true
  experiment_id: e7259e986278458899b46b485e069ec9
  experiment_tag: 8_activation=relu,height=-51.6714,steps=100,width=19.8211
  gain: 96.79744897015512
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  loss: 4.8835672653630535
  node_ip: 127.0.0.1
  pid: 46690
  time_since_restore: 10.72464108467102
  time_this_iter_s: 0.10328292846679688
  time_total_s: 10.72464108467102
  timestamp: 1658500040
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 5d66093e
  warmup_time: 0.0027511119842529297
  

Result for multi_objective_65469a2e:
  date: 2022-07-22_15-27-23
  done: false
  experiment_id: 015ee29ddb8b4c239d85a022a23d4093
  gain: 162.96737941207124
  hostname: Kais-MacBook-Pro.local
  iterations: 0
  iterations_since_restore: 1
  loss: 11.96774070054979
  node_ip: 127.0.0.1
  pid: 46719
  time_since_restore: 0.10403180122375488
  time_this_iter_s: 0.10403180122375488
  time_total_s: 0.10403180122375488
  timestamp: 1658500043
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 65469a2e
  warmup_time: 0.0033850669860839844
  
Result for multi_objective_634c9444:
  date: 2022-07-22_15-27-24
  done: false
  experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee
  gain: 136.50874589912112
  hostname: Kais-MacBook-Pro.local
  iterations: 24
  iterations_since_restore: 25
  loss: 9.034518610724309
  node_ip: 127.0.0.1
  pid: 46710
  time_since_restore: 5.125189781188965
  time_this_iter_s: 0.11187601089477539
  time_total_s: 5.125189781188965
  timestamp: 1658500044
  timesteps_since_restore: 0
  training_iteration: 25
  trial_id: 634c9444
  warmup_time: 0.004335880279541016
  
Result for multi_objective_65469a2e:
  date: 2022-07-22_15-27-28
  done: false
  experiment_id: 015ee29ddb8b4c239d85a022a23d4093
  gain: 28.890088364068774
  hostname: Kais-MacBook-Pro.local
  iterations: 47
  iterations_since_restore: 48
  loss: 2.1215846238952016
  node_ip: 127.0.0.1
  pid: 46719
  time_since_restore: 5.174077749252319
  time_this_iter_s: 0.10734677314758301
  time_total_s: 5.174077749252319
  timestamp: 1658500048
  timesteps_since_restore: 0
  training_iteration: 48
  trial_id: 65469a2e
  warmup_time: 0.0033850669860839844
  
Result for multi_objective_634c9444:
  date: 2022-07-22_15-27-30
  done: false
  experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee
  gain: 133.84933935751317
  hostname: Kais-MacBook-Pro.local
  iterations: 71
  iterations_since_restore: 72
  loss: 8.85851188137237
  node_ip: 127.0.0.1
  pid: 46710
  time_since_restore: 10.204095840454102
  time_this_iter_s: 0.10942888259887695
  time_total_s: 10.204095840454102
  timestamp: 1658500050
  timesteps_since_restore: 0
  training_iteration: 72
  trial_id: 634c9444
  warmup_time: 0.004335880279541016
  
Result for multi_objective_634c9444:
  date: 2022-07-22_15-27-33
  done: true
  experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee
  experiment_tag: 9_activation=relu,height=-12.3384,steps=100,width=15.1097
  gain: 133.45728949146132
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  loss: 8.83256495915982
  node_ip: 127.0.0.1
  pid: 46710
  time_since_restore: 13.198630809783936
  time_this_iter_s: 0.1067969799041748
  time_total_s: 13.198630809783936
  timestamp: 1658500053
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 634c9444
  warmup_time: 0.004335880279541016
  
Result for multi_objective_65469a2e:
  date: 2022-07-22_15-27-33
  done: false
  experiment_id: 015ee29ddb8b4c239d85a022a23d4093
  gain: 27.850744704436202
  hostname: Kais-MacBook-Pro.local
  iterations: 94
  iterations_since_restore: 95
  loss: 2.0452589477867855
  node_ip: 127.0.0.1
  pid: 46719
  time_since_restore: 10.19201397895813
  time_this_iter_s: 0.10735702514648438
  time_total_s: 10.19201397895813
  timestamp: 1658500053
  timesteps_since_restore: 0
  training_iteration: 95
  trial_id: 65469a2e
  warmup_time: 0.0033850669860839844
  
Result for multi_objective_65469a2e:
  date: 2022-07-22_15-27-34
  done: true
  experiment_id: 015ee29ddb8b4c239d85a022a23d4093
  experiment_tag: 10_activation=tanh,height=19.6774,steps=100,width=13.6172
  gain: 27.797824972416166
  hostname: Kais-MacBook-Pro.local
  iterations: 99
  iterations_since_restore: 100
  loss: 2.041372712191397
  node_ip: 127.0.0.1
  pid: 46719
  time_since_restore: 10.727944135665894
  time_this_iter_s: 0.10885024070739746
  time_total_s: 10.727944135665894
  timestamp: 1658500054
  timesteps_since_restore: 0
  training_iteration: 100
  trial_id: 65469a2e
  warmup_time: 0.0033850669860839844
  

Now there are two hyperparameter sets for the two objectives.

print("Best hyperparameters for loss found were: ", results.get_best_result("loss", "min").config)
print("Best hyperparameters for gain found were: ", results.get_best_result("gain", "max").config)

Best hyperparameters for loss found were:  {'steps': 100, 'width': 10.925599395387751, 'height': -47.52361963011387, 'activation': 'tanh'}
Best hyperparameters for gain found were:  {'steps': 100, 'width': 16.90386360893735, 'height': 54.91329340230965, 'activation': 'relu'}

We can mix-and-match the use of initial hyperparameter evaluations, conditional search spaces via define-by-run functions, and multi-objective tasks. This is also true of scheduler usage, with the exception of multi-objective optimization– schedulers typically rely on a single scalar score, rather than the two scores we use here: loss, gain.

Ray 3.0.0.dev0

Running Tune experiments with Optuna#

Providing an initial set of hyperparameters#

Conditional search spaces#

Multi-objective optimization#