Running Tune experiments with Optuna
Contents
Running Tune experiments with Optuna#
In this tutorial we introduce Optuna, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with Optuna and, as a result, allow you to seamlessly scale up a Optuna optimization process - without sacrificing performance.
Similar to Ray Tune, Optuna is an automatic hyperparameter optimization software framework, particularly designed for machine learning. It features an imperative (“how” over “what” emphasis), define-by-run style user API. With Optuna, a user has the ability to dynamically construct the search spaces for the hyperparameters. Optuna falls in the domain of “derivative-free optimization” and “black-box optimization”.
In this example we minimize a simple objective to briefly demonstrate the usage of Optuna with Ray Tune via OptunaSearch, including examples of conditional search spaces (string together relationships between hyperparameters), and the multi-objective problem (measure trade-offs among all important metrics). It’s useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume optuna==2.9.1 library is installed. To learn more, please refer to Optuna website.
Please note that sophisticated schedulers, such as AsyncHyperBandScheduler, may not work correctly with multi-objective optimization, since they typically expect a scalar score to compare fitness among trials.
Click below to see all the imports we need for this example. You can also launch directly into a Binder instance to run this notebook yourself. Just click on the rocket symbol at the top of the navigation.
import time
from typing import Dict, Optional, Any
import ray
from ray import tune
from ray.air import session
from ray.tune.search import ConcurrencyLimiter
from ray.tune.search.optuna import OptunaSearch
Let’s start by defining a simple evaluation function.
An explicit math formula is queried here for demonstration, yet in practice this is typically a black-box function– e.g. the performance results after training an ML model.
We artificially sleep for a bit (0.1 seconds) to simulate a long-running ML experiment.
This setup assumes that we’re running multiple steps of an experiment while tuning three hyperparameters,
namely width, height, and activation.
def evaluate(step, width, height, activation):
time.sleep(0.1)
activation_boost = 10 if activation=="relu" else 0
return (0.1 + width * step / 100) ** (-1) + height * 0.1 + activation_boost
Next, our objective function to be optimized takes a Tune config, evaluates the score of your experiment in a training loop,
and uses session.report to report the score back to Tune.
def objective(config):
for step in range(config["steps"]):
score = evaluate(step, config["width"], config["height"], config["activation"])
session.report({"iterations": step, "mean_loss": score})
Next we define a search space. The critical assumption is that the optimal hyperparamters live within this space. Yet, if the space is very large, then those hyperparamters may be difficult to find in a short amount of time.
The simplest case is a search space with independent dimensions. In this case, a config dictionary will suffice.
search_space = {
"steps": 100,
"width": tune.uniform(0, 20),
"height": tune.uniform(-100, 100),
"activation": tune.choice(["relu", "tanh"]),
}
Here we define the Optuna search algorithm:
algo = OptunaSearch()
We also constrain the number of concurrent trials to 4 with a ConcurrencyLimiter.
algo = ConcurrencyLimiter(algo, max_concurrent=4)
The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to 1000 samples.
(you can decrease this if it takes too long on your machine).
num_samples = 1000
Finally, we run the experiment to "min"imize the “mean_loss” of the objective by searching search_space via algo, num_samples times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute tuner.fit().
tuner = tune.Tuner(
objective,
tune_config=tune.TuneConfig(
metric="mean_loss",
mode="min",
search_alg=algo,
num_samples=num_samples,
),
param_space=search_space,
)
results = tuner.fit()
Function checkpointing is disabled. This may result in unexpected behavior when using checkpointing features or certain schedulers. To enable, set the train function arguments to be `func(config, checkpoint_dir=None)`.
[I 2022-07-22 15:21:47,769] A new study created in memory with name: optuna
== Status ==
Current time: 2022-07-22 15:22:32 (running for 00:00:43.89)
Memory usage on this node: 10.1/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects
Current best trial: b1bad2fe with mean_loss=-1.5166144695235813 and parameters={'steps': 100, 'width': 7.688465886501621, 'height': -16.46289560854555, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/optuna_example
Number of trials: 10/10 (10 TERMINATED)
Current time: 2022-07-22 15:22:32 (running for 00:00:43.89)
Memory usage on this node: 10.1/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects
Current best trial: b1bad2fe with mean_loss=-1.5166144695235813 and parameters={'steps': 100, 'width': 7.688465886501621, 'height': -16.46289560854555, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/optuna_example
Number of trials: 10/10 (10 TERMINATED)
| Trial name | status | loc | activation | height | width | loss | iter | total time (s) | iterations | neg_mean_loss |
|---|---|---|---|---|---|---|---|---|---|---|
| objective_9f689668 | TERMINATED | 127.0.0.1:46141 | tanh | 32.9806 | 16.0406 | 3.36064 | 100 | 11.0409 | 99 | -3.36064 |
| objective_a11d2104 | TERMINATED | 127.0.0.1:46149 | relu | 72.2627 | 0.763881 | 18.3942 | 100 | 11.7018 | 99 | -18.3942 |
| objective_a11eef8e | TERMINATED | 127.0.0.1:46150 | relu | -32.8474 | 13.1144 | 6.79169 | 100 | 11.8071 | 99 | -6.79169 |
| objective_a1209e56 | TERMINATED | 127.0.0.1:46151 | tanh | 75.6408 | 10.4415 | 7.65989 | 100 | 11.6987 | 99 | -7.65989 |
| objective_a7b383e6 | TERMINATED | 127.0.0.1:46174 | relu | 50.1501 | 3.4612 | 15.2986 | 100 | 10.7015 | 99 | -15.2986 |
| objective_a9c844c8 | TERMINATED | 127.0.0.1:46183 | relu | -40.3931 | 14.0525 | 6.03205 | 100 | 10.6687 | 99 | -6.03205 |
| objective_a9cb308e | TERMINATED | 127.0.0.1:46184 | tanh | 48.4802 | 12.7746 | 4.92647 | 100 | 10.7122 | 99 | -4.92647 |
| objective_a9d8332e | TERMINATED | 127.0.0.1:46189 | tanh | -1.26682 | 10.9788 | -0.0355157 | 100 | 10.7089 | 99 | 0.0355157 |
| objective_af948a1a | TERMINATED | 127.0.0.1:46204 | tanh | 61.6381 | 11.0105 | 6.25472 | 100 | 10.744 | 99 | -6.25472 |
| objective_b1bad2fe | TERMINATED | 127.0.0.1:46209 | tanh | -16.4629 | 7.68847 | -1.51661 | 100 | 10.7274 | 99 | 1.51661 |
Result for objective_9f689668:
date: 2022-07-22_15-21-51
done: false
experiment_id: 7d29e5c95aa44d77becc24b23a326b9b
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 13.298063134872326
neg_mean_loss: -13.298063134872326
node_ip: 127.0.0.1
pid: 46141
time_since_restore: 0.10468196868896484
time_this_iter_s: 0.10468196868896484
time_total_s: 0.10468196868896484
timestamp: 1658499711
timesteps_since_restore: 0
training_iteration: 1
trial_id: 9f689668
warmup_time: 0.003930091857910156
Result for objective_a1209e56:
date: 2022-07-22_15-21-54
done: false
experiment_id: 70d9788c327c43fb94450e5084416b81
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 17.564083106485064
neg_mean_loss: -17.564083106485064
node_ip: 127.0.0.1
pid: 46151
time_since_restore: 0.10416483879089355
time_this_iter_s: 0.10416483879089355
time_total_s: 0.10416483879089355
timestamp: 1658499714
timesteps_since_restore: 0
training_iteration: 1
trial_id: a1209e56
warmup_time: 0.0026938915252685547
Result for objective_a11d2104:
date: 2022-07-22_15-21-54
done: false
experiment_id: af2598364a004cd28f37ede19afd1ffb
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 27.2262748487915
neg_mean_loss: -27.2262748487915
node_ip: 127.0.0.1
pid: 46149
time_since_restore: 0.10507392883300781
time_this_iter_s: 0.10507392883300781
time_total_s: 0.10507392883300781
timestamp: 1658499714
timesteps_since_restore: 0
training_iteration: 1
trial_id: a11d2104
warmup_time: 0.0031232833862304688
Result for objective_a11eef8e:
date: 2022-07-22_15-21-54
done: false
experiment_id: e4a5922603d44b459c23607a42d5e574
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 16.71525618736072
neg_mean_loss: -16.71525618736072
node_ip: 127.0.0.1
pid: 46150
time_since_restore: 0.10363006591796875
time_this_iter_s: 0.10363006591796875
time_total_s: 0.10363006591796875
timestamp: 1658499714
timesteps_since_restore: 0
training_iteration: 1
trial_id: a11eef8e
warmup_time: 0.0028891563415527344
Result for objective_9f689668:
date: 2022-07-22_15-21-56
done: false
experiment_id: 7d29e5c95aa44d77becc24b23a326b9b
hostname: Kais-MacBook-Pro.local
iterations: 45
iterations_since_restore: 46
mean_loss: 3.434707064844201
neg_mean_loss: -3.434707064844201
node_ip: 127.0.0.1
pid: 46141
time_since_restore: 5.202661752700806
time_this_iter_s: 0.10527586936950684
time_total_s: 5.202661752700806
timestamp: 1658499716
timesteps_since_restore: 0
training_iteration: 46
trial_id: 9f689668
warmup_time: 0.003930091857910156
Result for objective_a1209e56:
date: 2022-07-22_15-21-59
done: false
experiment_id: 70d9788c327c43fb94450e5084416b81
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 7.763782433167051
neg_mean_loss: -7.763782433167051
node_ip: 127.0.0.1
pid: 46151
time_since_restore: 5.146381139755249
time_this_iter_s: 0.10691022872924805
time_total_s: 5.146381139755249
timestamp: 1658499719
timesteps_since_restore: 0
training_iteration: 48
trial_id: a1209e56
warmup_time: 0.0026938915252685547
Result for objective_a11eef8e:
date: 2022-07-22_15-21-59
done: false
experiment_id: e4a5922603d44b459c23607a42d5e574
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 6.874905099790281
neg_mean_loss: -6.874905099790281
node_ip: 127.0.0.1
pid: 46150
time_since_restore: 5.1618921756744385
time_this_iter_s: 0.10776615142822266
time_total_s: 5.1618921756744385
timestamp: 1658499719
timesteps_since_restore: 0
training_iteration: 48
trial_id: a11eef8e
warmup_time: 0.0028891563415527344
Result for objective_a11d2104:
date: 2022-07-22_15-21-59
done: false
experiment_id: af2598364a004cd28f37ede19afd1ffb
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 19.4048102011043
neg_mean_loss: -19.4048102011043
node_ip: 127.0.0.1
pid: 46149
time_since_restore: 5.170748233795166
time_this_iter_s: 0.11556220054626465
time_total_s: 5.170748233795166
timestamp: 1658499719
timesteps_since_restore: 0
training_iteration: 48
trial_id: a11d2104
warmup_time: 0.0031232833862304688
Result for objective_9f689668:
date: 2022-07-22_15-22-01
done: false
experiment_id: 7d29e5c95aa44d77becc24b23a326b9b
hostname: Kais-MacBook-Pro.local
iterations: 92
iterations_since_restore: 93
mean_loss: 3.3653696881211697
neg_mean_loss: -3.3653696881211697
node_ip: 127.0.0.1
pid: 46141
time_since_restore: 10.248206853866577
time_this_iter_s: 0.10745501518249512
time_total_s: 10.248206853866577
timestamp: 1658499721
timesteps_since_restore: 0
training_iteration: 93
trial_id: 9f689668
warmup_time: 0.003930091857910156
Result for objective_9f689668:
date: 2022-07-22_15-22-02
done: true
experiment_id: 7d29e5c95aa44d77becc24b23a326b9b
experiment_tag: 1_activation=tanh,height=32.9806,steps=100,width=16.0406
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 3.3606404197394255
neg_mean_loss: -3.3606404197394255
node_ip: 127.0.0.1
pid: 46141
time_since_restore: 11.04089903831482
time_this_iter_s: 0.10824704170227051
time_total_s: 11.04089903831482
timestamp: 1658499722
timesteps_since_restore: 0
training_iteration: 100
trial_id: 9f689668
warmup_time: 0.003930091857910156
Result for objective_a1209e56:
date: 2022-07-22_15-22-04
done: false
experiment_id: 70d9788c327c43fb94450e5084416b81
hostname: Kais-MacBook-Pro.local
iterations: 90
iterations_since_restore: 91
mean_loss: 7.669375143981996
neg_mean_loss: -7.669375143981996
node_ip: 127.0.0.1
pid: 46151
time_since_restore: 9.724387168884277
time_this_iter_s: 0.10670638084411621
time_total_s: 9.724387168884277
timestamp: 1658499724
timesteps_since_restore: 0
training_iteration: 91
trial_id: a1209e56
warmup_time: 0.0026938915252685547
Result for objective_a11d2104:
date: 2022-07-22_15-22-04
done: false
experiment_id: af2598364a004cd28f37ede19afd1ffb
hostname: Kais-MacBook-Pro.local
iterations: 90
iterations_since_restore: 91
mean_loss: 18.49612779999574
neg_mean_loss: -18.49612779999574
node_ip: 127.0.0.1
pid: 46149
time_since_restore: 9.818917274475098
time_this_iter_s: 0.10725140571594238
time_total_s: 9.818917274475098
timestamp: 1658499724
timesteps_since_restore: 0
training_iteration: 91
trial_id: a11d2104
warmup_time: 0.0031232833862304688
Result for objective_a11eef8e:
date: 2022-07-22_15-22-03
done: false
experiment_id: e4a5922603d44b459c23607a42d5e574
hostname: Kais-MacBook-Pro.local
iterations: 89
iterations_since_restore: 90
mean_loss: 6.800205168670805
neg_mean_loss: -6.800205168670805
node_ip: 127.0.0.1
pid: 46150
time_since_restore: 9.714852094650269
time_this_iter_s: 0.10286402702331543
time_total_s: 9.714852094650269
timestamp: 1658499723
timesteps_since_restore: 0
training_iteration: 90
trial_id: a11eef8e
warmup_time: 0.0028891563415527344
Result for objective_a7b383e6:
date: 2022-07-22_15-22-05
done: false
experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 25.015010793208496
neg_mean_loss: -25.015010793208496
node_ip: 127.0.0.1
pid: 46174
time_since_restore: 0.10384607315063477
time_this_iter_s: 0.10384607315063477
time_total_s: 0.10384607315063477
timestamp: 1658499725
timesteps_since_restore: 0
training_iteration: 1
trial_id: a7b383e6
warmup_time: 0.002817869186401367
Result for objective_a1209e56:
date: 2022-07-22_15-22-05
done: true
experiment_id: 70d9788c327c43fb94450e5084416b81
experiment_tag: 4_activation=tanh,height=75.6408,steps=100,width=10.4415
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 7.659894851608276
neg_mean_loss: -7.659894851608276
node_ip: 127.0.0.1
pid: 46151
time_since_restore: 11.698678970336914
time_this_iter_s: 0.10874581336975098
time_total_s: 11.698678970336914
timestamp: 1658499725
timesteps_since_restore: 0
training_iteration: 100
trial_id: a1209e56
warmup_time: 0.0026938915252685547
Result for objective_a11d2104:
date: 2022-07-22_15-22-05
done: true
experiment_id: af2598364a004cd28f37ede19afd1ffb
experiment_tag: 2_activation=relu,height=72.2627,steps=100,width=0.7639
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 18.394168929385685
neg_mean_loss: -18.394168929385685
node_ip: 127.0.0.1
pid: 46149
time_since_restore: 11.701792001724243
time_this_iter_s: 0.10734391212463379
time_total_s: 11.701792001724243
timestamp: 1658499725
timesteps_since_restore: 0
training_iteration: 100
trial_id: a11d2104
warmup_time: 0.0031232833862304688
Result for objective_a11eef8e:
date: 2022-07-22_15-22-06
done: true
experiment_id: e4a5922603d44b459c23607a42d5e574
experiment_tag: 3_activation=relu,height=-32.8474,steps=100,width=13.1144
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 6.791690049131352
neg_mean_loss: -6.791690049131352
node_ip: 127.0.0.1
pid: 46150
time_since_restore: 11.807135105133057
time_this_iter_s: 0.10672497749328613
time_total_s: 11.807135105133057
timestamp: 1658499726
timesteps_since_restore: 0
training_iteration: 100
trial_id: a11eef8e
warmup_time: 0.0028891563415527344
Result for objective_a9cb308e:
date: 2022-07-22_15-22-08
done: false
experiment_id: fcc8b8f4880c471c8e8545228d7e591a
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 14.84802392036395
neg_mean_loss: -14.84802392036395
node_ip: 127.0.0.1
pid: 46184
time_since_restore: 0.1039888858795166
time_this_iter_s: 0.1039888858795166
time_total_s: 0.1039888858795166
timestamp: 1658499728
timesteps_since_restore: 0
training_iteration: 1
trial_id: a9cb308e
warmup_time: 0.0027310848236083984
Result for objective_a9c844c8:
date: 2022-07-22_15-22-08
done: false
experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 15.960686077942514
neg_mean_loss: -15.960686077942514
node_ip: 127.0.0.1
pid: 46183
time_since_restore: 0.1004641056060791
time_this_iter_s: 0.1004641056060791
time_total_s: 0.1004641056060791
timestamp: 1658499728
timesteps_since_restore: 0
training_iteration: 1
trial_id: a9c844c8
warmup_time: 0.002599954605102539
Result for objective_a9d8332e:
date: 2022-07-22_15-22-08
done: false
experiment_id: 9c2f8aa7e4ed45c8867025673b146195
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 9.873318211243287
neg_mean_loss: -9.873318211243287
node_ip: 127.0.0.1
pid: 46189
time_since_restore: 0.10463905334472656
time_this_iter_s: 0.10463905334472656
time_total_s: 0.10463905334472656
timestamp: 1658499728
timesteps_since_restore: 0
training_iteration: 1
trial_id: a9d8332e
warmup_time: 0.002753019332885742
Result for objective_a7b383e6:
date: 2022-07-22_15-22-10
done: false
experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 15.594128340598196
neg_mean_loss: -15.594128340598196
node_ip: 127.0.0.1
pid: 46174
time_since_restore: 5.12034797668457
time_this_iter_s: 0.10577988624572754
time_total_s: 5.12034797668457
timestamp: 1658499730
timesteps_since_restore: 0
training_iteration: 48
trial_id: a7b383e6
warmup_time: 0.002817869186401367
Result for objective_a9cb308e:
date: 2022-07-22_15-22-13
done: false
experiment_id: fcc8b8f4880c471c8e8545228d7e591a
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 5.011849709111575
neg_mean_loss: -5.011849709111575
node_ip: 127.0.0.1
pid: 46184
time_since_restore: 5.150436162948608
time_this_iter_s: 0.10827016830444336
time_total_s: 5.150436162948608
timestamp: 1658499733
timesteps_since_restore: 0
training_iteration: 48
trial_id: a9cb308e
warmup_time: 0.0027310848236083984
Result for objective_a9c844c8:
date: 2022-07-22_15-22-13
done: false
experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 6.109835222238134
neg_mean_loss: -6.109835222238134
node_ip: 127.0.0.1
pid: 46183
time_since_restore: 5.136260032653809
time_this_iter_s: 0.10730218887329102
time_total_s: 5.136260032653809
timestamp: 1658499733
timesteps_since_restore: 0
training_iteration: 48
trial_id: a9c844c8
warmup_time: 0.002599954605102539
Result for objective_a9d8332e:
date: 2022-07-22_15-22-13
done: false
experiment_id: 9c2f8aa7e4ed45c8867025673b146195
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 0.06343129971586822
neg_mean_loss: -0.06343129971586822
node_ip: 127.0.0.1
pid: 46189
time_since_restore: 5.141700983047485
time_this_iter_s: 0.1074991226196289
time_total_s: 5.141700983047485
timestamp: 1658499733
timesteps_since_restore: 0
training_iteration: 48
trial_id: a9d8332e
warmup_time: 0.002753019332885742
Result for objective_a7b383e6:
date: 2022-07-22_15-22-15
done: false
experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 15.313204013214245
neg_mean_loss: -15.313204013214245
node_ip: 127.0.0.1
pid: 46174
time_since_restore: 10.163710832595825
time_this_iter_s: 0.10754895210266113
time_total_s: 10.163710832595825
timestamp: 1658499735
timesteps_since_restore: 0
training_iteration: 95
trial_id: a7b383e6
warmup_time: 0.002817869186401367
Result for objective_a7b383e6:
date: 2022-07-22_15-22-15
done: true
experiment_id: d2b2d4c7590142b3bdd660dcf6c947aa
experiment_tag: 5_activation=relu,height=50.1501,steps=100,width=3.4612
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 15.298570798421121
neg_mean_loss: -15.298570798421121
node_ip: 127.0.0.1
pid: 46174
time_since_restore: 10.701509952545166
time_this_iter_s: 0.10684585571289062
time_total_s: 10.701509952545166
timestamp: 1658499735
timesteps_since_restore: 0
training_iteration: 100
trial_id: a7b383e6
warmup_time: 0.002817869186401367
Result for objective_af948a1a:
date: 2022-07-22_15-22-18
done: false
experiment_id: d27f58d6527a4d47ae4c12496f5275cf
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 16.16381214127525
neg_mean_loss: -16.16381214127525
node_ip: 127.0.0.1
pid: 46204
time_since_restore: 0.10407209396362305
time_this_iter_s: 0.10407209396362305
time_total_s: 0.10407209396362305
timestamp: 1658499738
timesteps_since_restore: 0
training_iteration: 1
trial_id: af948a1a
warmup_time: 0.002669095993041992
Result for objective_a9cb308e:
date: 2022-07-22_15-22-18
done: false
experiment_id: fcc8b8f4880c471c8e8545228d7e591a
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 4.930613328484136
neg_mean_loss: -4.930613328484136
node_ip: 127.0.0.1
pid: 46184
time_since_restore: 10.175731897354126
time_this_iter_s: 0.10618400573730469
time_total_s: 10.175731897354126
timestamp: 1658499738
timesteps_since_restore: 0
training_iteration: 95
trial_id: a9cb308e
warmup_time: 0.0027310848236083984
Result for objective_a9d8332e:
date: 2022-07-22_15-22-18
done: false
experiment_id: 9c2f8aa7e4ed45c8867025673b146195
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: -0.030712998363490673
neg_mean_loss: 0.030712998363490673
node_ip: 127.0.0.1
pid: 46189
time_since_restore: 10.171820163726807
time_this_iter_s: 0.10807609558105469
time_total_s: 10.171820163726807
timestamp: 1658499738
timesteps_since_restore: 0
training_iteration: 95
trial_id: a9d8332e
warmup_time: 0.002753019332885742
Result for objective_a9c844c8:
date: 2022-07-22_15-22-18
done: false
experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e
hostname: Kais-MacBook-Pro.local
iterations: 95
iterations_since_restore: 96
mean_loss: 6.035035952030598
neg_mean_loss: -6.035035952030598
node_ip: 127.0.0.1
pid: 46183
time_since_restore: 10.23954701423645
time_this_iter_s: 0.10863399505615234
time_total_s: 10.23954701423645
timestamp: 1658499738
timesteps_since_restore: 0
training_iteration: 96
trial_id: a9c844c8
warmup_time: 0.002599954605102539
Result for objective_a9cb308e:
date: 2022-07-22_15-22-19
done: true
experiment_id: fcc8b8f4880c471c8e8545228d7e591a
experiment_tag: 7_activation=tanh,height=48.4802,steps=100,width=12.7746
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 4.926474869576359
neg_mean_loss: -4.926474869576359
node_ip: 127.0.0.1
pid: 46184
time_since_restore: 10.712197065353394
time_this_iter_s: 0.10726809501647949
time_total_s: 10.712197065353394
timestamp: 1658499739
timesteps_since_restore: 0
training_iteration: 100
trial_id: a9cb308e
warmup_time: 0.0027310848236083984
Result for objective_a9c844c8:
date: 2022-07-22_15-22-19
done: true
experiment_id: 3ef3f95b10fb4a65a21c80130f69ed3e
experiment_tag: 6_activation=relu,height=-40.3931,steps=100,width=14.0525
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 6.03205335569619
neg_mean_loss: -6.03205335569619
node_ip: 127.0.0.1
pid: 46183
time_since_restore: 10.668684959411621
time_this_iter_s: 0.1074681282043457
time_total_s: 10.668684959411621
timestamp: 1658499739
timesteps_since_restore: 0
training_iteration: 100
trial_id: a9c844c8
warmup_time: 0.002599954605102539
Result for objective_a9d8332e:
date: 2022-07-22_15-22-19
done: true
experiment_id: 9c2f8aa7e4ed45c8867025673b146195
experiment_tag: 8_activation=tanh,height=-1.2668,steps=100,width=10.9788
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -0.035515719608698634
neg_mean_loss: 0.035515719608698634
node_ip: 127.0.0.1
pid: 46189
time_since_restore: 10.708936929702759
time_this_iter_s: 0.10948991775512695
time_total_s: 10.708936929702759
timestamp: 1658499739
timesteps_since_restore: 0
training_iteration: 100
trial_id: a9d8332e
warmup_time: 0.002753019332885742
Result for objective_b1bad2fe:
date: 2022-07-22_15-22-21
done: false
experiment_id: f58c8e48e2324dd69c28f526bf96497c
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 8.353710439145445
neg_mean_loss: -8.353710439145445
node_ip: 127.0.0.1
pid: 46209
time_since_restore: 0.10369992256164551
time_this_iter_s: 0.10369992256164551
time_total_s: 0.10369992256164551
timestamp: 1658499741
timesteps_since_restore: 0
training_iteration: 1
trial_id: b1bad2fe
warmup_time: 0.0026597976684570312
Result for objective_af948a1a:
date: 2022-07-22_15-22-23
done: false
experiment_id: d27f58d6527a4d47ae4c12496f5275cf
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 6.353387644558537
neg_mean_loss: -6.353387644558537
node_ip: 127.0.0.1
pid: 46204
time_since_restore: 5.161527872085571
time_this_iter_s: 0.10784792900085449
time_total_s: 5.161527872085571
timestamp: 1658499743
timesteps_since_restore: 0
training_iteration: 48
trial_id: af948a1a
warmup_time: 0.002669095993041992
Result for objective_b1bad2fe:
date: 2022-07-22_15-22-26
done: false
experiment_id: f58c8e48e2324dd69c28f526bf96497c
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -1.3770075531258914
neg_mean_loss: 1.3770075531258914
node_ip: 127.0.0.1
pid: 46209
time_since_restore: 5.158898115158081
time_this_iter_s: 0.10701417922973633
time_total_s: 5.158898115158081
timestamp: 1658499746
timesteps_since_restore: 0
training_iteration: 48
trial_id: b1bad2fe
warmup_time: 0.0026597976684570312
Result for objective_af948a1a:
date: 2022-07-22_15-22-28
done: false
experiment_id: d27f58d6527a4d47ae4c12496f5275cf
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 6.259506962612728
neg_mean_loss: -6.259506962612728
node_ip: 127.0.0.1
pid: 46204
time_since_restore: 10.20361876487732
time_this_iter_s: 0.10824179649353027
time_total_s: 10.20361876487732
timestamp: 1658499748
timesteps_since_restore: 0
training_iteration: 95
trial_id: af948a1a
warmup_time: 0.002669095993041992
Result for objective_af948a1a:
date: 2022-07-22_15-22-28
done: true
experiment_id: d27f58d6527a4d47ae4c12496f5275cf
experiment_tag: 9_activation=tanh,height=61.6381,steps=100,width=11.0105
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 6.254717826198094
neg_mean_loss: -6.254717826198094
node_ip: 127.0.0.1
pid: 46204
time_since_restore: 10.744001865386963
time_this_iter_s: 0.10536718368530273
time_total_s: 10.744001865386963
timestamp: 1658499748
timesteps_since_restore: 0
training_iteration: 100
trial_id: af948a1a
warmup_time: 0.002669095993041992
Result for objective_b1bad2fe:
date: 2022-07-22_15-22-31
done: false
experiment_id: f58c8e48e2324dd69c28f526bf96497c
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: -1.5098109958909571
neg_mean_loss: 1.5098109958909571
node_ip: 127.0.0.1
pid: 46209
time_since_restore: 10.190011024475098
time_this_iter_s: 0.10639405250549316
time_total_s: 10.190011024475098
timestamp: 1658499751
timesteps_since_restore: 0
training_iteration: 95
trial_id: b1bad2fe
warmup_time: 0.0026597976684570312
Result for objective_b1bad2fe:
date: 2022-07-22_15-22-32
done: true
experiment_id: f58c8e48e2324dd69c28f526bf96497c
experiment_tag: 10_activation=tanh,height=-16.4629,steps=100,width=7.6885
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -1.5166144695235813
neg_mean_loss: 1.5166144695235813
node_ip: 127.0.0.1
pid: 46209
time_since_restore: 10.727396011352539
time_this_iter_s: 0.10742807388305664
time_total_s: 10.727396011352539
timestamp: 1658499752
timesteps_since_restore: 0
training_iteration: 100
trial_id: b1bad2fe
warmup_time: 0.0026597976684570312
And now we have the hyperparameters found to minimize the mean loss.
print("Best hyperparameters found were: ", results.get_best_result().config)
Best hyperparameters found were: {'steps': 100, 'width': 7.688465886501621, 'height': -16.46289560854555, 'activation': 'tanh'}
Providing an initial set of hyperparameters#
While defining the search algorithm, we may choose to provide an initial set of hyperparameters that we believe are especially promising or informative, and
pass this information as a helpful starting point for the OptunaSearch object.
initial_params = [
{"width": 1, "height": 2, "activation": "relu"},
{"width": 4, "height": 2, "activation": "relu"},
]
Now the search_alg built using OptunaSearch takes points_to_evaluate.
searcher = OptunaSearch(points_to_evaluate=initial_params)
algo = ConcurrencyLimiter(searcher, max_concurrent=4)
And run the experiment with initial hyperparameter evaluations:
tuner = tune.Tuner(
objective,
tune_config=tune.TuneConfig(
metric="mean_loss",
mode="min",
search_alg=algo,
num_samples=num_samples,
),
param_space=search_space,
)
results = tuner.fit()
[I 2022-07-22 15:22:32,644] A new study created in memory with name: optuna
/Users/kai/coding/ray/python/ray/tune/search/optuna/optuna_search.py:389: ExperimentalWarning: enqueue_trial is experimental (supported from v1.2.0). The interface can change in the future.
self._ot_study.enqueue_trial(point)
/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/optuna/study/study.py:857: ExperimentalWarning: create_trial is experimental (supported from v2.0.0). The interface can change in the future.
create_trial(state=TrialState.WAITING, system_attrs={"fixed_params": params})
/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/optuna/study/study.py:857: ExperimentalWarning: add_trial is experimental (supported from v2.0.0). The interface can change in the future.
create_trial(state=TrialState.WAITING, system_attrs={"fixed_params": params})
== Status ==
Current time: 2022-07-22 15:23:15 (running for 00:00:42.97)
Memory usage on this node: 10.1/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects
Current best trial: c3762426 with mean_loss=-8.621481572769046 and parameters={'steps': 100, 'width': 0.7236705053153614, 'height': -98.46320622400306, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/optuna_example_2
Number of trials: 10/10 (10 TERMINATED)
Current time: 2022-07-22 15:23:15 (running for 00:00:42.97)
Memory usage on this node: 10.1/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects
Current best trial: c3762426 with mean_loss=-8.621481572769046 and parameters={'steps': 100, 'width': 0.7236705053153614, 'height': -98.46320622400306, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/optuna_example_2
Number of trials: 10/10 (10 TERMINATED)
| Trial name | status | loc | activation | height | width | loss | iter | total time (s) | iterations | neg_mean_loss |
|---|---|---|---|---|---|---|---|---|---|---|
| objective_b9acff32 | TERMINATED | 127.0.0.1:46220 | relu | 2 | 1 | 11.1174 | 100 | 10.6837 | 99 | -11.1174 |
| objective_bb3b2702 | TERMINATED | 127.0.0.1:46227 | relu | 2 | 4 | 10.4463 | 100 | 11.2816 | 99 | -10.4463 |
| objective_bb3c8714 | TERMINATED | 127.0.0.1:46228 | relu | -90.7016 | 0.566597 | 2.44285 | 100 | 11.2766 | 99 | -2.44285 |
| objective_bb3e0210 | TERMINATED | 127.0.0.1:46229 | tanh | -81.4501 | 14.783 | -8.07715 | 100 | 11.2196 | 99 | 8.07715 |
| objective_c19b9dde | TERMINATED | 127.0.0.1:46248 | relu | 86.466 | 0.308659 | 21.1122 | 100 | 10.7226 | 99 | -21.1122 |
| objective_c3762426 | TERMINATED | 127.0.0.1:46253 | tanh | -98.4632 | 0.723671 | -8.62148 | 100 | 10.7271 | 99 | 8.62148 |
| objective_c385528e | TERMINATED | 127.0.0.1:46256 | relu | -54.8368 | 11.0765 | 4.60669 | 100 | 10.7498 | 99 | -4.60669 |
| objective_c387a7c8 | TERMINATED | 127.0.0.1:46257 | tanh | 42.9466 | 8.01708 | 4.41909 | 100 | 10.7214 | 99 | -4.41909 |
| objective_c97678f8 | TERMINATED | 127.0.0.1:46272 | tanh | 9.25214 | 15.5288 | 0.989841 | 100 | 12.4591 | 99 | -0.989841 |
| objective_cb6180fe | TERMINATED | 127.0.0.1:46277 | relu | 41.6647 | 8.0585 | 14.2903 | 100 | 10.752 | 99 | -14.2903 |
Result for objective_b9acff32:
date: 2022-07-22_15-22-35
done: false
experiment_id: ec3c260f0ad348fa9b188f9c93c01eba
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 20.2
neg_mean_loss: -20.2
node_ip: 127.0.0.1
pid: 46220
time_since_restore: 0.10049605369567871
time_this_iter_s: 0.10049605369567871
time_total_s: 0.10049605369567871
timestamp: 1658499755
timesteps_since_restore: 0
training_iteration: 1
trial_id: b9acff32
warmup_time: 0.0029239654541015625
Result for objective_bb3e0210:
date: 2022-07-22_15-22-37
done: false
experiment_id: 08a7bc47c140405296e58a049cd2f6b4
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 1.8549886153795079
neg_mean_loss: -1.8549886153795079
node_ip: 127.0.0.1
pid: 46229
time_since_restore: 0.10361814498901367
time_this_iter_s: 0.10361814498901367
time_total_s: 0.10361814498901367
timestamp: 1658499757
timesteps_since_restore: 0
training_iteration: 1
trial_id: bb3e0210
warmup_time: 0.002833843231201172
Result for objective_bb3b2702:
date: 2022-07-22_15-22-37
done: false
experiment_id: 2f984120624b4e9aa3392fcc1b16b811
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 20.2
neg_mean_loss: -20.2
node_ip: 127.0.0.1
pid: 46227
time_since_restore: 0.10420107841491699
time_this_iter_s: 0.10420107841491699
time_total_s: 0.10420107841491699
timestamp: 1658499757
timesteps_since_restore: 0
training_iteration: 1
trial_id: bb3b2702
warmup_time: 0.002672910690307617
Result for objective_bb3c8714:
date: 2022-07-22_15-22-37
done: false
experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 10.929835533284928
neg_mean_loss: -10.929835533284928
node_ip: 127.0.0.1
pid: 46228
time_since_restore: 0.1048588752746582
time_this_iter_s: 0.1048588752746582
time_total_s: 0.1048588752746582
timestamp: 1658499757
timesteps_since_restore: 0
training_iteration: 1
trial_id: bb3c8714
warmup_time: 0.0027511119842529297
Result for objective_b9acff32:
date: 2022-07-22_15-22-40
done: false
experiment_id: ec3c260f0ad348fa9b188f9c93c01eba
hostname: Kais-MacBook-Pro.local
iterations: 48
iterations_since_restore: 49
mean_loss: 11.924137931034483
neg_mean_loss: -11.924137931034483
node_ip: 127.0.0.1
pid: 46220
time_since_restore: 5.207761287689209
time_this_iter_s: 0.10753417015075684
time_total_s: 5.207761287689209
timestamp: 1658499760
timesteps_since_restore: 0
training_iteration: 49
trial_id: b9acff32
warmup_time: 0.0029239654541015625
Result for objective_bb3e0210:
date: 2022-07-22_15-22-43
done: false
experiment_id: 08a7bc47c140405296e58a049cd2f6b4
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -8.003127819283387
neg_mean_loss: 8.003127819283387
node_ip: 127.0.0.1
pid: 46229
time_since_restore: 5.188760042190552
time_this_iter_s: 0.10853385925292969
time_total_s: 5.188760042190552
timestamp: 1658499763
timesteps_since_restore: 0
training_iteration: 48
trial_id: bb3e0210
warmup_time: 0.002833843231201172
Result for objective_bb3b2702:
date: 2022-07-22_15-22-43
done: false
experiment_id: 2f984120624b4e9aa3392fcc1b16b811
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 10.705050505050504
neg_mean_loss: -10.705050505050504
node_ip: 127.0.0.1
pid: 46227
time_since_restore: 5.1574530601501465
time_this_iter_s: 0.10719585418701172
time_total_s: 5.1574530601501465
timestamp: 1658499763
timesteps_since_restore: 0
training_iteration: 48
trial_id: bb3b2702
warmup_time: 0.002672910690307617
Result for objective_bb3c8714:
date: 2022-07-22_15-22-43
done: false
experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 3.6598337799021596
neg_mean_loss: -3.6598337799021596
node_ip: 127.0.0.1
pid: 46228
time_since_restore: 5.163677930831909
time_this_iter_s: 0.10699272155761719
time_total_s: 5.163677930831909
timestamp: 1658499763
timesteps_since_restore: 0
training_iteration: 48
trial_id: bb3c8714
warmup_time: 0.0027511119842529297
Result for objective_b9acff32:
date: 2022-07-22_15-22-45
done: false
experiment_id: ec3c260f0ad348fa9b188f9c93c01eba
hostname: Kais-MacBook-Pro.local
iterations: 95
iterations_since_restore: 96
mean_loss: 11.152380952380952
neg_mean_loss: -11.152380952380952
node_ip: 127.0.0.1
pid: 46220
time_since_restore: 10.250982284545898
time_this_iter_s: 0.1073312759399414
time_total_s: 10.250982284545898
timestamp: 1658499765
timesteps_since_restore: 0
training_iteration: 96
trial_id: b9acff32
warmup_time: 0.0029239654541015625
Result for objective_b9acff32:
date: 2022-07-22_15-22-45
done: true
experiment_id: ec3c260f0ad348fa9b188f9c93c01eba
experiment_tag: 1_activation=relu,height=2,steps=100,width=1
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 11.11743119266055
neg_mean_loss: -11.11743119266055
node_ip: 127.0.0.1
pid: 46220
time_since_restore: 10.68371295928955
time_this_iter_s: 0.10785365104675293
time_total_s: 10.68371295928955
timestamp: 1658499765
timesteps_since_restore: 0
training_iteration: 100
trial_id: b9acff32
warmup_time: 0.0029239654541015625
Result for objective_bb3e0210:
date: 2022-07-22_15-22-47
done: false
experiment_id: 08a7bc47c140405296e58a049cd2f6b4
hostname: Kais-MacBook-Pro.local
iterations: 93
iterations_since_restore: 94
mean_loss: -8.072800015147221
neg_mean_loss: 8.072800015147221
node_ip: 127.0.0.1
pid: 46229
time_since_restore: 10.099772214889526
time_this_iter_s: 0.10250687599182129
time_total_s: 10.099772214889526
timestamp: 1658499767
timesteps_since_restore: 0
training_iteration: 94
trial_id: bb3e0210
warmup_time: 0.002833843231201172
Result for objective_bb3c8714:
date: 2022-07-22_15-22-47
done: false
experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf
hostname: Kais-MacBook-Pro.local
iterations: 92
iterations_since_restore: 93
mean_loss: 2.53944357306219
neg_mean_loss: -2.53944357306219
node_ip: 127.0.0.1
pid: 46228
time_since_restore: 9.980368852615356
time_this_iter_s: 0.10658979415893555
time_total_s: 9.980368852615356
timestamp: 1658499767
timesteps_since_restore: 0
training_iteration: 93
trial_id: bb3c8714
warmup_time: 0.0027511119842529297
Result for objective_bb3b2702:
date: 2022-07-22_15-22-47
done: false
experiment_id: 2f984120624b4e9aa3392fcc1b16b811
hostname: Kais-MacBook-Pro.local
iterations: 92
iterations_since_restore: 93
mean_loss: 10.464550264550265
neg_mean_loss: -10.464550264550265
node_ip: 127.0.0.1
pid: 46227
time_since_restore: 9.964121103286743
time_this_iter_s: 0.10547995567321777
time_total_s: 9.964121103286743
timestamp: 1658499767
timesteps_since_restore: 0
training_iteration: 93
trial_id: bb3b2702
warmup_time: 0.002672910690307617
Result for objective_c19b9dde:
date: 2022-07-22_15-22-48
done: false
experiment_id: fab35a2f68c347c4bb815c268e9500a6
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 28.646596578422436
neg_mean_loss: -28.646596578422436
node_ip: 127.0.0.1
pid: 46248
time_since_restore: 0.10230708122253418
time_this_iter_s: 0.10230708122253418
time_total_s: 0.10230708122253418
timestamp: 1658499768
timesteps_since_restore: 0
training_iteration: 1
trial_id: c19b9dde
warmup_time: 0.0026540756225585938
Result for objective_bb3e0210:
date: 2022-07-22_15-22-49
done: true
experiment_id: 08a7bc47c140405296e58a049cd2f6b4
experiment_tag: 4_activation=tanh,height=-81.4501,steps=100,width=14.7830
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -8.077146761191683
neg_mean_loss: 8.077146761191683
node_ip: 127.0.0.1
pid: 46229
time_since_restore: 11.219618082046509
time_this_iter_s: 0.1108238697052002
time_total_s: 11.219618082046509
timestamp: 1658499769
timesteps_since_restore: 0
training_iteration: 100
trial_id: bb3e0210
warmup_time: 0.002833843231201172
Result for objective_bb3c8714:
date: 2022-07-22_15-22-49
done: true
experiment_id: 5fdcb12c4e234f1d9904a09b8705c0bf
experiment_tag: 3_activation=relu,height=-90.7016,steps=100,width=0.5666
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 2.442852652712544
neg_mean_loss: -2.442852652712544
node_ip: 127.0.0.1
pid: 46228
time_since_restore: 11.276630163192749
time_this_iter_s: 0.10824012756347656
time_total_s: 11.276630163192749
timestamp: 1658499769
timesteps_since_restore: 0
training_iteration: 100
trial_id: bb3c8714
warmup_time: 0.0027511119842529297
Result for objective_bb3b2702:
date: 2022-07-22_15-22-49
done: true
experiment_id: 2f984120624b4e9aa3392fcc1b16b811
experiment_tag: 2_activation=relu,height=2,steps=100,width=4
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 10.446305418719211
neg_mean_loss: -10.446305418719211
node_ip: 127.0.0.1
pid: 46227
time_since_restore: 11.281576871871948
time_this_iter_s: 0.1068716049194336
time_total_s: 11.281576871871948
timestamp: 1658499769
timesteps_since_restore: 0
training_iteration: 100
trial_id: bb3b2702
warmup_time: 0.002672910690307617
Result for objective_c3762426:
date: 2022-07-22_15-22-51
done: false
experiment_id: 034596724b2a4c9f86726e9a15fb386b
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 0.15367937759969408
neg_mean_loss: -0.15367937759969408
node_ip: 127.0.0.1
pid: 46253
time_since_restore: 0.1050560474395752
time_this_iter_s: 0.1050560474395752
time_total_s: 0.1050560474395752
timestamp: 1658499771
timesteps_since_restore: 0
training_iteration: 1
trial_id: c3762426
warmup_time: 0.0028579235076904297
Result for objective_c385528e:
date: 2022-07-22_15-22-51
done: false
experiment_id: 3044e8f1500f4b16aba04b72105c67be
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 14.516317633001044
neg_mean_loss: -14.516317633001044
node_ip: 127.0.0.1
pid: 46256
time_since_restore: 0.10402488708496094
time_this_iter_s: 0.10402488708496094
time_total_s: 0.10402488708496094
timestamp: 1658499771
timesteps_since_restore: 0
training_iteration: 1
trial_id: c385528e
warmup_time: 0.0027010440826416016
Result for objective_c387a7c8:
date: 2022-07-22_15-22-51
done: false
experiment_id: fc7f3d46b136437cab526c2fa6ea2944
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 14.294664350689382
neg_mean_loss: -14.294664350689382
node_ip: 127.0.0.1
pid: 46257
time_since_restore: 0.10461091995239258
time_this_iter_s: 0.10461091995239258
time_total_s: 0.10461091995239258
timestamp: 1658499771
timesteps_since_restore: 0
training_iteration: 1
trial_id: c387a7c8
warmup_time: 0.0025899410247802734
Result for objective_c19b9dde:
date: 2022-07-22_15-22-53
done: false
experiment_id: fab35a2f68c347c4bb815c268e9500a6
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 22.72706638821166
neg_mean_loss: -22.72706638821166
node_ip: 127.0.0.1
pid: 46248
time_since_restore: 5.118642330169678
time_this_iter_s: 0.10826611518859863
time_total_s: 5.118642330169678
timestamp: 1658499773
timesteps_since_restore: 0
training_iteration: 48
trial_id: c19b9dde
warmup_time: 0.0026540756225585938
Result for objective_c3762426:
date: 2022-07-22_15-22-56
done: false
experiment_id: 034596724b2a4c9f86726e9a15fb386b
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -7.5742395372701825
neg_mean_loss: 7.5742395372701825
node_ip: 127.0.0.1
pid: 46253
time_since_restore: 5.14322304725647
time_this_iter_s: 0.10359907150268555
time_total_s: 5.14322304725647
timestamp: 1658499776
timesteps_since_restore: 0
training_iteration: 48
trial_id: c3762426
warmup_time: 0.0028579235076904297
Result for objective_c385528e:
date: 2022-07-22_15-22-56
done: false
experiment_id: 3044e8f1500f4b16aba04b72105c67be
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 4.704784733826337
neg_mean_loss: -4.704784733826337
node_ip: 127.0.0.1
pid: 46256
time_since_restore: 5.16935396194458
time_this_iter_s: 0.10719513893127441
time_total_s: 5.16935396194458
timestamp: 1658499776
timesteps_since_restore: 0
training_iteration: 48
trial_id: c385528e
warmup_time: 0.0027010440826416016
Result for objective_c387a7c8:
date: 2022-07-22_15-22-56
done: false
experiment_id: fc7f3d46b136437cab526c2fa6ea2944
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 4.553194134463865
neg_mean_loss: -4.553194134463865
node_ip: 127.0.0.1
pid: 46257
time_since_restore: 5.1458728313446045
time_this_iter_s: 0.10374999046325684
time_total_s: 5.1458728313446045
timestamp: 1658499776
timesteps_since_restore: 0
training_iteration: 48
trial_id: c387a7c8
warmup_time: 0.0025899410247802734
Result for objective_c19b9dde:
date: 2022-07-22_15-22-58
done: false
experiment_id: fab35a2f68c347c4bb815c268e9500a6
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 21.20978139671461
neg_mean_loss: -21.20978139671461
node_ip: 127.0.0.1
pid: 46248
time_since_restore: 10.183105230331421
time_this_iter_s: 0.10718917846679688
time_total_s: 10.183105230331421
timestamp: 1658499778
timesteps_since_restore: 0
training_iteration: 95
trial_id: c19b9dde
warmup_time: 0.0026540756225585938
Result for objective_c19b9dde:
date: 2022-07-22_15-22-59
done: true
experiment_id: fab35a2f68c347c4bb815c268e9500a6
experiment_tag: 5_activation=relu,height=86.4660,steps=100,width=0.3087
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 21.112246386473707
neg_mean_loss: -21.112246386473707
node_ip: 127.0.0.1
pid: 46248
time_since_restore: 10.722583055496216
time_this_iter_s: 0.10788774490356445
time_total_s: 10.722583055496216
timestamp: 1658499779
timesteps_since_restore: 0
training_iteration: 100
trial_id: c19b9dde
warmup_time: 0.0026540756225585938
Result for objective_c97678f8:
date: 2022-07-22_15-23-01
done: false
experiment_id: 7c4187f99b9246bdbae534ce20f33bfc
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 10.925214276212655
neg_mean_loss: -10.925214276212655
node_ip: 127.0.0.1
pid: 46272
time_since_restore: 0.1039581298828125
time_this_iter_s: 0.1039581298828125
time_total_s: 0.1039581298828125
timestamp: 1658499781
timesteps_since_restore: 0
training_iteration: 1
trial_id: c97678f8
warmup_time: 0.002711057662963867
Result for objective_c3762426:
date: 2022-07-22_15-23-01
done: false
experiment_id: 034596724b2a4c9f86726e9a15fb386b
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: -8.564680574270156
neg_mean_loss: 8.564680574270156
node_ip: 127.0.0.1
pid: 46253
time_since_restore: 10.190189123153687
time_this_iter_s: 0.10848307609558105
time_total_s: 10.190189123153687
timestamp: 1658499781
timesteps_since_restore: 0
training_iteration: 95
trial_id: c3762426
warmup_time: 0.0028579235076904297
Result for objective_c387a7c8:
date: 2022-07-22_15-23-01
done: false
experiment_id: fc7f3d46b136437cab526c2fa6ea2944
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 4.425622066068654
neg_mean_loss: -4.425622066068654
node_ip: 127.0.0.1
pid: 46257
time_since_restore: 10.181554794311523
time_this_iter_s: 0.10594582557678223
time_total_s: 10.181554794311523
timestamp: 1658499781
timesteps_since_restore: 0
training_iteration: 95
trial_id: c387a7c8
warmup_time: 0.0025899410247802734
Result for objective_c385528e:
date: 2022-07-22_15-23-01
done: false
experiment_id: 3044e8f1500f4b16aba04b72105c67be
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 4.611447627123378
neg_mean_loss: -4.611447627123378
node_ip: 127.0.0.1
pid: 46256
time_since_restore: 10.214791059494019
time_this_iter_s: 0.10866808891296387
time_total_s: 10.214791059494019
timestamp: 1658499781
timesteps_since_restore: 0
training_iteration: 95
trial_id: c385528e
warmup_time: 0.0027010440826416016
Result for objective_c3762426:
date: 2022-07-22_15-23-02
done: true
experiment_id: 034596724b2a4c9f86726e9a15fb386b
experiment_tag: 6_activation=tanh,height=-98.4632,steps=100,width=0.7237
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -8.621481572769046
neg_mean_loss: 8.621481572769046
node_ip: 127.0.0.1
pid: 46253
time_since_restore: 10.72708010673523
time_this_iter_s: 0.10719490051269531
time_total_s: 10.72708010673523
timestamp: 1658499782
timesteps_since_restore: 0
training_iteration: 100
trial_id: c3762426
warmup_time: 0.0028579235076904297
Result for objective_c387a7c8:
date: 2022-07-22_15-23-02
done: true
experiment_id: fc7f3d46b136437cab526c2fa6ea2944
experiment_tag: 8_activation=tanh,height=42.9466,steps=100,width=8.0171
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 4.419090335709762
neg_mean_loss: -4.419090335709762
node_ip: 127.0.0.1
pid: 46257
time_since_restore: 10.721372842788696
time_this_iter_s: 0.10873699188232422
time_total_s: 10.721372842788696
timestamp: 1658499782
timesteps_since_restore: 0
training_iteration: 100
trial_id: c387a7c8
warmup_time: 0.0025899410247802734
Result for objective_c385528e:
date: 2022-07-22_15-23-02
done: true
experiment_id: 3044e8f1500f4b16aba04b72105c67be
experiment_tag: 7_activation=relu,height=-54.8368,steps=100,width=11.0765
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 4.606686500095965
neg_mean_loss: -4.606686500095965
node_ip: 127.0.0.1
pid: 46256
time_since_restore: 10.749844074249268
time_this_iter_s: 0.10710501670837402
time_total_s: 10.749844074249268
timestamp: 1658499782
timesteps_since_restore: 0
training_iteration: 100
trial_id: c385528e
warmup_time: 0.0027010440826416016
Result for objective_cb6180fe:
date: 2022-07-22_15-23-04
done: false
experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 24.166473702046588
neg_mean_loss: -24.166473702046588
node_ip: 127.0.0.1
pid: 46277
time_since_restore: 0.10389995574951172
time_this_iter_s: 0.10389995574951172
time_total_s: 0.10389995574951172
timestamp: 1658499784
timesteps_since_restore: 0
training_iteration: 1
trial_id: cb6180fe
warmup_time: 0.0027260780334472656
Result for objective_c97678f8:
date: 2022-07-22_15-23-06
done: false
experiment_id: 7c4187f99b9246bdbae534ce20f33bfc
hostname: Kais-MacBook-Pro.local
iterations: 31
iterations_since_restore: 32
mean_loss: 1.1287174007116527
neg_mean_loss: -1.1287174007116527
node_ip: 127.0.0.1
pid: 46272
time_since_restore: 5.140314340591431
time_this_iter_s: 0.10663819313049316
time_total_s: 5.140314340591431
timestamp: 1658499786
timesteps_since_restore: 0
training_iteration: 32
trial_id: c97678f8
warmup_time: 0.002711057662963867
Result for objective_cb6180fe:
date: 2022-07-22_15-23-09
done: false
experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 14.42370878193022
neg_mean_loss: -14.42370878193022
node_ip: 127.0.0.1
pid: 46277
time_since_restore: 5.139087915420532
time_this_iter_s: 0.10686802864074707
time_total_s: 5.139087915420532
timestamp: 1658499789
timesteps_since_restore: 0
training_iteration: 48
trial_id: cb6180fe
warmup_time: 0.0027260780334472656
Result for objective_c97678f8:
date: 2022-07-22_15-23-11
done: false
experiment_id: 7c4187f99b9246bdbae534ce20f33bfc
hostname: Kais-MacBook-Pro.local
iterations: 78
iterations_since_restore: 79
mean_loss: 1.0070978091228147
neg_mean_loss: -1.0070978091228147
node_ip: 127.0.0.1
pid: 46272
time_since_restore: 10.206932067871094
time_this_iter_s: 0.10774993896484375
time_total_s: 10.206932067871094
timestamp: 1658499791
timesteps_since_restore: 0
training_iteration: 79
trial_id: c97678f8
warmup_time: 0.002711057662963867
Result for objective_c97678f8:
date: 2022-07-22_15-23-14
done: true
experiment_id: 7c4187f99b9246bdbae534ce20f33bfc
experiment_tag: 9_activation=tanh,height=9.2521,steps=100,width=15.5288
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 0.9898408262377837
neg_mean_loss: -0.9898408262377837
node_ip: 127.0.0.1
pid: 46272
time_since_restore: 12.459095239639282
time_this_iter_s: 0.10785508155822754
time_total_s: 12.459095239639282
timestamp: 1658499794
timesteps_since_restore: 0
training_iteration: 100
trial_id: c97678f8
warmup_time: 0.002711057662963867
Result for objective_cb6180fe:
date: 2022-07-22_15-23-15
done: false
experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 14.296767042885353
neg_mean_loss: -14.296767042885353
node_ip: 127.0.0.1
pid: 46277
time_since_restore: 10.220713138580322
time_this_iter_s: 0.10750126838684082
time_total_s: 10.220713138580322
timestamp: 1658499795
timesteps_since_restore: 0
training_iteration: 95
trial_id: cb6180fe
warmup_time: 0.0027260780334472656
Result for objective_cb6180fe:
date: 2022-07-22_15-23-15
done: true
experiment_id: 0cfd73afdca24f8f894b2a3e7ebb8cea
experiment_tag: 10_activation=relu,height=41.6647,steps=100,width=8.0585
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 14.29026803363612
neg_mean_loss: -14.29026803363612
node_ip: 127.0.0.1
pid: 46277
time_since_restore: 10.752035856246948
time_this_iter_s: 0.1066899299621582
time_total_s: 10.752035856246948
timestamp: 1658499795
timesteps_since_restore: 0
training_iteration: 100
trial_id: cb6180fe
warmup_time: 0.0027260780334472656
We take another look at the optimal hyperparamters.
print("Best hyperparameters found were: ", results.get_best_result().config)
Best hyperparameters found were: {'steps': 100, 'width': 0.7236705053153614, 'height': -98.46320622400306, 'activation': 'tanh'}
Conditional search spaces#
Sometimes we may want to build a more complicated search space that has conditional dependencies on other hyperparameters. In this case, we pass a define-by-run function to the search_alg argument in ray.tune().
def define_by_run_func(trial) -> Optional[Dict[str, Any]]:
"""Define-by-run function to create the search space.
Ensure no actual computation takes place here. That should go into
the trainable passed to ``Tuner()`` (in this example, that's
``objective``).
For more information, see https://optuna.readthedocs.io/en/stable\
/tutorial/10_key_features/002_configurations.html
This function should either return None or a dict with constant values.
"""
activation = trial.suggest_categorical("activation", ["relu", "tanh"])
# Define-by-run allows for conditional search spaces.
if activation == "relu":
trial.suggest_float("width", 0, 20)
trial.suggest_float("height", -100, 100)
else:
trial.suggest_float("width", -1, 21)
trial.suggest_float("height", -101, 101)
# Return all constants in a dictionary.
return {"steps": 100}
As before, we create the search_alg from OptunaSearch and ConcurrencyLimiter, this time we define the scope of search via the space argument and provide no initialization. We also must specific metric and mode when using space.
searcher = OptunaSearch(space=define_by_run_func, metric="mean_loss", mode="min")
algo = ConcurrencyLimiter(searcher, max_concurrent=4)
[I 2022-07-22 15:23:15,784] A new study created in memory with name: optuna
Running the experiment with a define-by-run search space:
tuner = tune.Tuner(
objective,
tune_config=tune.TuneConfig(
search_alg=algo,
num_samples=num_samples,
),
)
results = tuner.fit()
== Status ==
Current time: 2022-07-22 15:23:58 (running for 00:00:43.15)
Memory usage on this node: 10.4/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects
Result logdir: /Users/kai/ray_results/optuna_example_3
Number of trials: 10/10 (10 TERMINATED)
Current time: 2022-07-22 15:23:58 (running for 00:00:43.15)
Memory usage on this node: 10.4/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects
Result logdir: /Users/kai/ray_results/optuna_example_3
Number of trials: 10/10 (10 TERMINATED)
| Trial name | status | loc | activation | height | steps | width | loss | iter | total time (s) | iterations | neg_mean_loss |
|---|---|---|---|---|---|---|---|---|---|---|---|
| objective_d363eed6 | TERMINATED | 127.0.0.1:46290 | relu | 96.9804 | 100 | 6.47293 | 19.8517 | 100 | 10.735 | 99 | -19.8517 |
| objective_d4f3a700 | TERMINATED | 127.0.0.1:46298 | tanh | -76.4387 | 100 | 17.7358 | -7.58724 | 100 | 11.396 | 99 | 7.58724 |
| objective_d4f50ce4 | TERMINATED | 127.0.0.1:46299 | relu | -89.8609 | 100 | 6.28321 | 1.17212 | 100 | 11.4182 | 99 | -1.17212 |
| objective_d4f67908 | TERMINATED | 127.0.0.1:46300 | relu | 67.9266 | 100 | 14.0599 | 16.864 | 100 | 11.341 | 99 | -16.864 |
| objective_db5c0402 | TERMINATED | 127.0.0.1:46317 | tanh | -54.13 | 100 | 11.4712 | -5.32571 | 100 | 10.705 | 99 | 5.32571 |
| objective_dd487b56 | TERMINATED | 127.0.0.1:46322 | relu | -25.9818 | 100 | 0.758497 | 8.57703 | 100 | 10.6916 | 99 | -8.57703 |
| objective_dd4b4e94 | TERMINATED | 127.0.0.1:46323 | relu | 54.6085 | 100 | 1.2361 | 16.2163 | 100 | 10.6711 | 99 | -16.2163 |
| objective_dd5a8bca | TERMINATED | 127.0.0.1:46328 | tanh | 5.22131 | 100 | 17.0952 | 0.580871 | 100 | 10.7017 | 99 | -0.580871 |
| objective_e3482178 | TERMINATED | 127.0.0.1:46341 | tanh | 78.844 | 100 | 15.5079 | 7.94912 | 100 | 12.6417 | 99 | -7.94912 |
| objective_e532d6e0 | TERMINATED | 127.0.0.1:46353 | tanh | -66.9988 | 100 | 7.15087 | -6.56059 | 100 | 10.7486 | 99 | 6.56059 |
Result for objective_d363eed6:
date: 2022-07-22_15-23-18
done: false
experiment_id: af8f846e60254bc794827e76909df4f0
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 29.698044266420816
neg_mean_loss: -29.698044266420816
node_ip: 127.0.0.1
pid: 46290
time_since_restore: 0.10497498512268066
time_this_iter_s: 0.10497498512268066
time_total_s: 0.10497498512268066
timestamp: 1658499798
timesteps_since_restore: 0
training_iteration: 1
trial_id: d363eed6
warmup_time: 0.0027971267700195312
Result for objective_d4f3a700:
date: 2022-07-22_15-23-21
done: false
experiment_id: 52e5f2a557cf4466939b6c80bb9ff905
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 2.3561338711040003
neg_mean_loss: -2.3561338711040003
node_ip: 127.0.0.1
pid: 46298
time_since_restore: 0.10145115852355957
time_this_iter_s: 0.10145115852355957
time_total_s: 0.10145115852355957
timestamp: 1658499801
timesteps_since_restore: 0
training_iteration: 1
trial_id: d4f3a700
warmup_time: 0.0033049583435058594
Result for objective_d4f67908:
date: 2022-07-22_15-23-21
done: false
experiment_id: 230c6da41a834386be865bb19780b3a5
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 26.792658178391687
neg_mean_loss: -26.792658178391687
node_ip: 127.0.0.1
pid: 46300
time_since_restore: 0.10428690910339355
time_this_iter_s: 0.10428690910339355
time_total_s: 0.10428690910339355
timestamp: 1658499801
timesteps_since_restore: 0
training_iteration: 1
trial_id: d4f67908
warmup_time: 0.0030798912048339844
Result for objective_d4f50ce4:
date: 2022-07-22_15-23-21
done: false
experiment_id: f2396d75f50b417f9fae16d1c9ddcda4
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 11.013906266700227
neg_mean_loss: -11.013906266700227
node_ip: 127.0.0.1
pid: 46299
time_since_restore: 0.1020050048828125
time_this_iter_s: 0.1020050048828125
time_total_s: 0.1020050048828125
timestamp: 1658499801
timesteps_since_restore: 0
training_iteration: 1
trial_id: d4f50ce4
warmup_time: 0.0027348995208740234
Result for objective_d363eed6:
date: 2022-07-22_15-23-23
done: false
experiment_id: af8f846e60254bc794827e76909df4f0
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 20.016284953435374
neg_mean_loss: -20.016284953435374
node_ip: 127.0.0.1
pid: 46290
time_since_restore: 5.138504981994629
time_this_iter_s: 0.10720086097717285
time_total_s: 5.138504981994629
timestamp: 1658499803
timesteps_since_restore: 0
training_iteration: 48
trial_id: d363eed6
warmup_time: 0.0027971267700195312
Result for objective_d4f3a700:
date: 2022-07-22_15-23-26
done: false
experiment_id: 52e5f2a557cf4466939b6c80bb9ff905
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -7.525324371061028
neg_mean_loss: 7.525324371061028
node_ip: 127.0.0.1
pid: 46298
time_since_restore: 5.153075218200684
time_this_iter_s: 0.10515189170837402
time_total_s: 5.153075218200684
timestamp: 1658499806
timesteps_since_restore: 0
training_iteration: 48
trial_id: d4f3a700
warmup_time: 0.0033049583435058594
Result for objective_d4f67908:
date: 2022-07-22_15-23-26
done: false
experiment_id: 230c6da41a834386be865bb19780b3a5
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 16.94173032408916
neg_mean_loss: -16.94173032408916
node_ip: 127.0.0.1
pid: 46300
time_since_restore: 5.153704881668091
time_this_iter_s: 0.10537075996398926
time_total_s: 5.153704881668091
timestamp: 1658499806
timesteps_since_restore: 0
training_iteration: 48
trial_id: d4f67908
warmup_time: 0.0030798912048339844
Result for objective_d4f50ce4:
date: 2022-07-22_15-23-26
done: false
experiment_id: f2396d75f50b417f9fae16d1c9ddcda4
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 1.3414410369095506
neg_mean_loss: -1.3414410369095506
node_ip: 127.0.0.1
pid: 46299
time_since_restore: 5.1622138023376465
time_this_iter_s: 0.10337996482849121
time_total_s: 5.1622138023376465
timestamp: 1658499806
timesteps_since_restore: 0
training_iteration: 48
trial_id: d4f50ce4
warmup_time: 0.0027348995208740234
Result for objective_d363eed6:
date: 2022-07-22_15-23-28
done: false
experiment_id: af8f846e60254bc794827e76909df4f0
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 19.85973747786284
neg_mean_loss: -19.85973747786284
node_ip: 127.0.0.1
pid: 46290
time_since_restore: 10.1943039894104
time_this_iter_s: 0.12109899520874023
time_total_s: 10.1943039894104
timestamp: 1658499808
timesteps_since_restore: 0
training_iteration: 95
trial_id: d363eed6
warmup_time: 0.0027971267700195312
Result for objective_d363eed6:
date: 2022-07-22_15-23-29
done: true
experiment_id: af8f846e60254bc794827e76909df4f0
experiment_tag: 1_activation=relu,height=96.9804,steps=100,width=6.4729
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 19.851696631549423
neg_mean_loss: -19.851696631549423
node_ip: 127.0.0.1
pid: 46290
time_since_restore: 10.734967947006226
time_this_iter_s: 0.10534882545471191
time_total_s: 10.734967947006226
timestamp: 1658499809
timesteps_since_restore: 0
training_iteration: 100
trial_id: d363eed6
warmup_time: 0.0027971267700195312
Result for objective_d4f67908:
date: 2022-07-22_15-23-31
done: false
experiment_id: 230c6da41a834386be865bb19780b3a5
hostname: Kais-MacBook-Pro.local
iterations: 93
iterations_since_restore: 94
mean_loss: 16.86855533914186
neg_mean_loss: -16.86855533914186
node_ip: 127.0.0.1
pid: 46300
time_since_restore: 10.096334218978882
time_this_iter_s: 0.10534310340881348
time_total_s: 10.096334218978882
timestamp: 1658499811
timesteps_since_restore: 0
training_iteration: 94
trial_id: d4f67908
warmup_time: 0.0030798912048339844
Result for objective_d4f50ce4:
date: 2022-07-22_15-23-31
done: false
experiment_id: f2396d75f50b417f9fae16d1c9ddcda4
hostname: Kais-MacBook-Pro.local
iterations: 92
iterations_since_restore: 93
mean_loss: 1.183958166642718
neg_mean_loss: -1.183958166642718
node_ip: 127.0.0.1
pid: 46299
time_since_restore: 9.976859092712402
time_this_iter_s: 0.1057121753692627
time_total_s: 9.976859092712402
timestamp: 1658499811
timesteps_since_restore: 0
training_iteration: 93
trial_id: d4f50ce4
warmup_time: 0.0027348995208740234
Result for objective_d4f3a700:
date: 2022-07-22_15-23-31
done: false
experiment_id: 52e5f2a557cf4466939b6c80bb9ff905
hostname: Kais-MacBook-Pro.local
iterations: 93
iterations_since_restore: 94
mean_loss: -7.583604593881161
neg_mean_loss: 7.583604593881161
node_ip: 127.0.0.1
pid: 46298
time_since_restore: 10.091833114624023
time_this_iter_s: 0.10643887519836426
time_total_s: 10.091833114624023
timestamp: 1658499811
timesteps_since_restore: 0
training_iteration: 94
trial_id: d4f3a700
warmup_time: 0.0033049583435058594
Result for objective_db5c0402:
date: 2022-07-22_15-23-31
done: false
experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 4.587004175501654
neg_mean_loss: -4.587004175501654
node_ip: 127.0.0.1
pid: 46317
time_since_restore: 0.10449004173278809
time_this_iter_s: 0.10449004173278809
time_total_s: 0.10449004173278809
timestamp: 1658499811
timesteps_since_restore: 0
training_iteration: 1
trial_id: db5c0402
warmup_time: 0.003406047821044922
Result for objective_d4f67908:
date: 2022-07-22_15-23-32
done: true
experiment_id: 230c6da41a834386be865bb19780b3a5
experiment_tag: 4_activation=relu,height=67.9266,steps=100,width=14.0599
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 16.86398832185618
neg_mean_loss: -16.86398832185618
node_ip: 127.0.0.1
pid: 46300
time_since_restore: 11.341041088104248
time_this_iter_s: 0.10513019561767578
time_total_s: 11.341041088104248
timestamp: 1658499812
timesteps_since_restore: 0
training_iteration: 100
trial_id: d4f67908
warmup_time: 0.0030798912048339844
Result for objective_d4f3a700:
date: 2022-07-22_15-23-32
done: true
experiment_id: 52e5f2a557cf4466939b6c80bb9ff905
experiment_tag: 2_activation=tanh,height=-76.4387,steps=100,width=17.7358
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -7.587236125634034
neg_mean_loss: 7.587236125634034
node_ip: 127.0.0.1
pid: 46298
time_since_restore: 11.396040916442871
time_this_iter_s: 0.10666394233703613
time_total_s: 11.396040916442871
timestamp: 1658499812
timesteps_since_restore: 0
training_iteration: 100
trial_id: d4f3a700
warmup_time: 0.0033049583435058594
Result for objective_d4f50ce4:
date: 2022-07-22_15-23-32
done: true
experiment_id: f2396d75f50b417f9fae16d1c9ddcda4
experiment_tag: 3_activation=relu,height=-89.8609,steps=100,width=6.2832
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 1.1721245345451354
neg_mean_loss: -1.1721245345451354
node_ip: 127.0.0.1
pid: 46299
time_since_restore: 11.418162822723389
time_this_iter_s: 0.11828303337097168
time_total_s: 11.418162822723389
timestamp: 1658499812
timesteps_since_restore: 0
training_iteration: 100
trial_id: d4f50ce4
warmup_time: 0.0027348995208740234
Result for objective_dd4b4e94:
date: 2022-07-22_15-23-35
done: false
experiment_id: c6036d4a26674f86946fd7c8c895e85c
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 25.460850093028082
neg_mean_loss: -25.460850093028082
node_ip: 127.0.0.1
pid: 46323
time_since_restore: 0.10411691665649414
time_this_iter_s: 0.10411691665649414
time_total_s: 0.10411691665649414
timestamp: 1658499815
timesteps_since_restore: 0
training_iteration: 1
trial_id: dd4b4e94
warmup_time: 0.003142118453979492
Result for objective_dd487b56:
date: 2022-07-22_15-23-35
done: false
experiment_id: 75e118a5733e454fab54f4a20c036852
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 17.401819651158622
neg_mean_loss: -17.401819651158622
node_ip: 127.0.0.1
pid: 46322
time_since_restore: 0.1048891544342041
time_this_iter_s: 0.1048891544342041
time_total_s: 0.1048891544342041
timestamp: 1658499815
timesteps_since_restore: 0
training_iteration: 1
trial_id: dd487b56
warmup_time: 0.0028679370880126953
Result for objective_dd5a8bca:
date: 2022-07-22_15-23-35
done: false
experiment_id: 775683e6d95a4705846a37e809b474d7
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 10.522131438315931
neg_mean_loss: -10.522131438315931
node_ip: 127.0.0.1
pid: 46328
time_since_restore: 0.10494732856750488
time_this_iter_s: 0.10494732856750488
time_total_s: 0.10494732856750488
timestamp: 1658499815
timesteps_since_restore: 0
training_iteration: 1
trial_id: dd5a8bca
warmup_time: 0.002931833267211914
Result for objective_db5c0402:
date: 2022-07-22_15-23-36
done: false
experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -5.230895562358142
neg_mean_loss: 5.230895562358142
node_ip: 127.0.0.1
pid: 46317
time_since_restore: 5.120880842208862
time_this_iter_s: 0.1076059341430664
time_total_s: 5.120880842208862
timestamp: 1658499816
timesteps_since_restore: 0
training_iteration: 48
trial_id: db5c0402
warmup_time: 0.003406047821044922
Result for objective_dd4b4e94:
date: 2022-07-22_15-23-40
done: false
experiment_id: c6036d4a26674f86946fd7c8c895e85c
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 16.929346028466988
neg_mean_loss: -16.929346028466988
node_ip: 127.0.0.1
pid: 46323
time_since_restore: 5.137623071670532
time_this_iter_s: 0.10670709609985352
time_total_s: 5.137623071670532
timestamp: 1658499820
timesteps_since_restore: 0
training_iteration: 48
trial_id: dd4b4e94
warmup_time: 0.003142118453979492
Result for objective_dd487b56:
date: 2022-07-22_15-23-40
done: false
experiment_id: 75e118a5733e454fab54f4a20c036852
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 9.592430453048937
neg_mean_loss: -9.592430453048937
node_ip: 127.0.0.1
pid: 46322
time_since_restore: 5.140692949295044
time_this_iter_s: 0.1086587905883789
time_total_s: 5.140692949295044
timestamp: 1658499820
timesteps_since_restore: 0
training_iteration: 48
trial_id: dd487b56
warmup_time: 0.0028679370880126953
Result for objective_dd5a8bca:
date: 2022-07-22_15-23-40
done: false
experiment_id: 775683e6d95a4705846a37e809b474d7
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 0.6450610360030851
neg_mean_loss: -0.6450610360030851
node_ip: 127.0.0.1
pid: 46328
time_since_restore: 5.138496160507202
time_this_iter_s: 0.10809993743896484
time_total_s: 5.138496160507202
timestamp: 1658499820
timesteps_since_restore: 0
training_iteration: 48
trial_id: dd5a8bca
warmup_time: 0.002931833267211914
Result for objective_db5c0402:
date: 2022-07-22_15-23-41
done: false
experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: -5.321109063262834
neg_mean_loss: 5.321109063262834
node_ip: 127.0.0.1
pid: 46317
time_since_restore: 10.162945985794067
time_this_iter_s: 0.10671615600585938
time_total_s: 10.162945985794067
timestamp: 1658499821
timesteps_since_restore: 0
training_iteration: 95
trial_id: db5c0402
warmup_time: 0.003406047821044922
Result for objective_db5c0402:
date: 2022-07-22_15-23-42
done: true
experiment_id: 69a34f42e51f4b97a5237fa5ae2be8d8
experiment_tag: 5_activation=tanh,height=-54.1300,steps=100,width=11.4712
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -5.325709301325668
neg_mean_loss: 5.325709301325668
node_ip: 127.0.0.1
pid: 46317
time_since_restore: 10.704953908920288
time_this_iter_s: 0.1082158088684082
time_total_s: 10.704953908920288
timestamp: 1658499822
timesteps_since_restore: 0
training_iteration: 100
trial_id: db5c0402
warmup_time: 0.003406047821044922
Result for objective_dd4b4e94:
date: 2022-07-22_15-23-45
done: false
experiment_id: c6036d4a26674f86946fd7c8c895e85c
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 16.253282229309562
neg_mean_loss: -16.253282229309562
node_ip: 127.0.0.1
pid: 46323
time_since_restore: 10.140441179275513
time_this_iter_s: 0.10501408576965332
time_total_s: 10.140441179275513
timestamp: 1658499825
timesteps_since_restore: 0
training_iteration: 95
trial_id: dd4b4e94
warmup_time: 0.003142118453979492
Result for objective_dd487b56:
date: 2022-07-22_15-23-45
done: false
experiment_id: 75e118a5733e454fab54f4a20c036852
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 8.631851056235014
neg_mean_loss: -8.631851056235014
node_ip: 127.0.0.1
pid: 46322
time_since_restore: 10.156053066253662
time_this_iter_s: 0.10682201385498047
time_total_s: 10.156053066253662
timestamp: 1658499825
timesteps_since_restore: 0
training_iteration: 95
trial_id: dd487b56
warmup_time: 0.0028679370880126953
Result for objective_e3482178:
date: 2022-07-22_15-23-45
done: false
experiment_id: 6709b686be234eb996457210f3efbe08
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 17.884403838898738
neg_mean_loss: -17.884403838898738
node_ip: 127.0.0.1
pid: 46341
time_since_restore: 0.10499191284179688
time_this_iter_s: 0.10499191284179688
time_total_s: 0.10499191284179688
timestamp: 1658499825
timesteps_since_restore: 0
training_iteration: 1
trial_id: e3482178
warmup_time: 0.002877950668334961
Result for objective_dd5a8bca:
date: 2022-07-22_15-23-45
done: false
experiment_id: 775683e6d95a4705846a37e809b474d7
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 0.583976365762005
neg_mean_loss: -0.583976365762005
node_ip: 127.0.0.1
pid: 46328
time_since_restore: 10.165286302566528
time_this_iter_s: 0.10989904403686523
time_total_s: 10.165286302566528
timestamp: 1658499825
timesteps_since_restore: 0
training_iteration: 95
trial_id: dd5a8bca
warmup_time: 0.002931833267211914
Result for objective_dd4b4e94:
date: 2022-07-22_15-23-45
done: true
experiment_id: c6036d4a26674f86946fd7c8c895e85c
experiment_tag: 7_activation=relu,height=54.6085,steps=100,width=1.2361
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 16.216283787762706
neg_mean_loss: -16.216283787762706
node_ip: 127.0.0.1
pid: 46323
time_since_restore: 10.67109203338623
time_this_iter_s: 0.1058969497680664
time_total_s: 10.67109203338623
timestamp: 1658499825
timesteps_since_restore: 0
training_iteration: 100
trial_id: dd4b4e94
warmup_time: 0.003142118453979492
Result for objective_dd487b56:
date: 2022-07-22_15-23-45
done: true
experiment_id: 75e118a5733e454fab54f4a20c036852
experiment_tag: 6_activation=relu,height=-25.9818,steps=100,width=0.7585
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 8.577028987245935
neg_mean_loss: -8.577028987245935
node_ip: 127.0.0.1
pid: 46322
time_since_restore: 10.691627979278564
time_this_iter_s: 0.10821914672851562
time_total_s: 10.691627979278564
timestamp: 1658499825
timesteps_since_restore: 0
training_iteration: 100
trial_id: dd487b56
warmup_time: 0.0028679370880126953
Result for objective_dd5a8bca:
date: 2022-07-22_15-23-45
done: true
experiment_id: 775683e6d95a4705846a37e809b474d7
experiment_tag: 8_activation=tanh,height=5.2213,steps=100,width=17.0952
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 0.5808712318418436
neg_mean_loss: -0.5808712318418436
node_ip: 127.0.0.1
pid: 46328
time_since_restore: 10.701741218566895
time_this_iter_s: 0.11321616172790527
time_total_s: 10.701741218566895
timestamp: 1658499825
timesteps_since_restore: 0
training_iteration: 100
trial_id: dd5a8bca
warmup_time: 0.002931833267211914
Result for objective_e532d6e0:
date: 2022-07-22_15-23-48
done: false
experiment_id: 34d05fd425474b82a450c22c2a063843
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 3.30012197692201
neg_mean_loss: -3.30012197692201
node_ip: 127.0.0.1
pid: 46353
time_since_restore: 0.10400509834289551
time_this_iter_s: 0.10400509834289551
time_total_s: 0.10400509834289551
timestamp: 1658499828
timesteps_since_restore: 0
training_iteration: 1
trial_id: e532d6e0
warmup_time: 0.0028901100158691406
Result for objective_e3482178:
date: 2022-07-22_15-23-50
done: false
experiment_id: 6709b686be234eb996457210f3efbe08
hostname: Kais-MacBook-Pro.local
iterations: 29
iterations_since_restore: 30
mean_loss: 8.10192352359083
neg_mean_loss: -8.10192352359083
node_ip: 127.0.0.1
pid: 46341
time_since_restore: 5.183844089508057
time_this_iter_s: 0.10574007034301758
time_total_s: 5.183844089508057
timestamp: 1658499830
timesteps_since_restore: 0
training_iteration: 30
trial_id: e3482178
warmup_time: 0.002877950668334961
Result for objective_e532d6e0:
date: 2022-07-22_15-23-53
done: false
experiment_id: 34d05fd425474b82a450c22c2a063843
hostname: Kais-MacBook-Pro.local
iterations: 48
iterations_since_restore: 49
mean_loss: -6.416785967274913
neg_mean_loss: 6.416785967274913
node_ip: 127.0.0.1
pid: 46353
time_since_restore: 5.212116003036499
time_this_iter_s: 0.10714387893676758
time_total_s: 5.212116003036499
timestamp: 1658499833
timesteps_since_restore: 0
training_iteration: 49
trial_id: e532d6e0
warmup_time: 0.0028901100158691406
Result for objective_e3482178:
date: 2022-07-22_15-23-55
done: false
experiment_id: 6709b686be234eb996457210f3efbe08
hostname: Kais-MacBook-Pro.local
iterations: 76
iterations_since_restore: 77
mean_loss: 7.968536513457016
neg_mean_loss: -7.968536513457016
node_ip: 127.0.0.1
pid: 46341
time_since_restore: 10.190052032470703
time_this_iter_s: 0.10843491554260254
time_total_s: 10.190052032470703
timestamp: 1658499835
timesteps_since_restore: 0
training_iteration: 77
trial_id: e3482178
warmup_time: 0.002877950668334961
Result for objective_e3482178:
date: 2022-07-22_15-23-57
done: true
experiment_id: 6709b686be234eb996457210f3efbe08
experiment_tag: 9_activation=tanh,height=78.8440,steps=100,width=15.5079
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 7.9491170267942515
neg_mean_loss: -7.9491170267942515
node_ip: 127.0.0.1
pid: 46341
time_since_restore: 12.64172887802124
time_this_iter_s: 0.10761594772338867
time_total_s: 12.64172887802124
timestamp: 1658499837
timesteps_since_restore: 0
training_iteration: 100
trial_id: e3482178
warmup_time: 0.002877950668334961
Result for objective_e532d6e0:
date: 2022-07-22_15-23-58
done: false
experiment_id: 34d05fd425474b82a450c22c2a063843
hostname: Kais-MacBook-Pro.local
iterations: 95
iterations_since_restore: 96
mean_loss: -6.554810275125985
neg_mean_loss: 6.554810275125985
node_ip: 127.0.0.1
pid: 46353
time_since_restore: 10.305040121078491
time_this_iter_s: 0.10702300071716309
time_total_s: 10.305040121078491
timestamp: 1658499838
timesteps_since_restore: 0
training_iteration: 96
trial_id: e532d6e0
warmup_time: 0.0028901100158691406
Result for objective_e532d6e0:
date: 2022-07-22_15-23-58
done: true
experiment_id: 34d05fd425474b82a450c22c2a063843
experiment_tag: 10_activation=tanh,height=-66.9988,steps=100,width=7.1509
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -6.560589957038967
neg_mean_loss: 6.560589957038967
node_ip: 127.0.0.1
pid: 46353
time_since_restore: 10.74857211112976
time_this_iter_s: 0.1075899600982666
time_total_s: 10.74857211112976
timestamp: 1658499838
timesteps_since_restore: 0
training_iteration: 100
trial_id: e532d6e0
warmup_time: 0.0028901100158691406
We take a look again at the optimal hyperparameters.
print("Best hyperparameters for loss found were: ", results.get_best_result("mean_loss", "min").config)
Best hyperparameters for loss found were: {'activation': 'tanh', 'width': 17.73584230792165, 'height': -76.43866128895999, 'steps': 100}
Multi-objective optimization#
Finally, let’s take a look at the multi-objective case.
def multi_objective(config):
# Hyperparameters
width, height = config["width"], config["height"]
for step in range(config["steps"]):
# Iterative training function - can be any arbitrary training procedure
intermediate_score = evaluate(step, config["width"], config["height"], config["activation"])
# Feed the score back back to Tune.
session.report({
"iterations": step, "loss": intermediate_score, "gain": intermediate_score * width
})
We define the OptunaSearch object this time with metric and mode as list arguments.
searcher = OptunaSearch(metric=["loss", "gain"], mode=["min", "max"])
algo = ConcurrencyLimiter(searcher, max_concurrent=4)
tuner = tune.Tuner(
multi_objective,
tune_config=tune.TuneConfig(
search_alg=algo,
num_samples=num_samples,
),
param_space=search_space
)
results = tuner.fit()
[I 2022-07-22 15:26:50,680] A new study created in memory with name: optuna
== Status ==
Current time: 2022-07-22 15:27:34 (running for 00:00:43.55)
Memory usage on this node: 8.2/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects
Result logdir: /Users/kai/ray_results/multi_objective_2022-07-22_15-26-50
Number of trials: 10/10 (10 TERMINATED)
Current time: 2022-07-22 15:27:34 (running for 00:00:43.55)
Memory usage on this node: 8.2/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.07 GiB heap, 0.0/2.0 GiB objects
Result logdir: /Users/kai/ray_results/multi_objective_2022-07-22_15-26-50
Number of trials: 10/10 (10 TERMINATED)
| Trial name | status | loc | activation | height | width | iter | total time (s) | iterations | loss | gain |
|---|---|---|---|---|---|---|---|---|---|---|
| multi_objective_5378f1e8 | TERMINATED | 127.0.0.1:46621 | tanh | -47.5236 | 10.9256 | 100 | 10.7064 | 99 | -4.66076 | -50.9216 |
| multi_objective_550962f4 | TERMINATED | 127.0.0.1:46631 | relu | -40.6828 | 4.34953 | 100 | 11.3948 | 99 | 6.15869 | 26.7874 |
| multi_objective_550ad9a4 | TERMINATED | 127.0.0.1:46632 | tanh | -42.0563 | 2.02124 | 100 | 11.2755 | 99 | -3.72967 | -7.53857 |
| multi_objective_550c4faa | TERMINATED | 127.0.0.1:46633 | relu | -56.2838 | 5.65237 | 100 | 11.407 | 99 | 4.54719 | 25.7024 |
| multi_objective_5b6c4ba2 | TERMINATED | 127.0.0.1:46679 | relu | 54.9133 | 16.9039 | 100 | 10.6698 | 99 | 15.5507 | 262.867 |
| multi_objective_5d51f2d2 | TERMINATED | 127.0.0.1:46686 | tanh | 16.5793 | 5.16709 | 100 | 10.7451 | 99 | 1.84967 | 9.55744 |
| multi_objective_5d636760 | TERMINATED | 127.0.0.1:46689 | tanh | -13.8895 | 13.6043 | 100 | 10.6897 | 99 | -1.31525 | -17.8931 |
| multi_objective_5d66093e | TERMINATED | 127.0.0.1:46690 | relu | -51.6714 | 19.8211 | 100 | 10.7246 | 99 | 4.88357 | 96.7974 |
| multi_objective_634c9444 | TERMINATED | 127.0.0.1:46710 | relu | -12.3384 | 15.1097 | 100 | 13.1986 | 99 | 8.83256 | 133.457 |
| multi_objective_65469a2e | TERMINATED | 127.0.0.1:46719 | tanh | 19.6774 | 13.6172 | 100 | 10.7279 | 99 | 2.04137 | 27.7978 |
Result for multi_objective_5378f1e8:
date: 2022-07-22_15-26-53
done: false
experiment_id: 4ce03cab420a449a9f4827c3b6b76c38
gain: 57.33359096413655
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
loss: 5.247638036988612
node_ip: 127.0.0.1
pid: 46621
time_since_restore: 0.10418081283569336
time_this_iter_s: 0.10418081283569336
time_total_s: 0.10418081283569336
timestamp: 1658500013
timesteps_since_restore: 0
training_iteration: 1
trial_id: 5378f1e8
warmup_time: 0.002785921096801758
Result for multi_objective_550ad9a4:
date: 2022-07-22_15-26-56
done: false
experiment_id: 102a4b06449b428b9870a75d02812be6
gain: 11.711822825618256
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
loss: 5.794371166191576
node_ip: 127.0.0.1
pid: 46632
time_since_restore: 0.10288000106811523
time_this_iter_s: 0.10288000106811523
time_total_s: 0.10288000106811523
timestamp: 1658500016
timesteps_since_restore: 0
training_iteration: 1
trial_id: 550ad9a4
warmup_time: 0.0030989646911621094
Result for multi_objective_550c4faa:
date: 2022-07-22_15-26-56
done: false
experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d
gain: 81.23378320141829
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
loss: 14.371624912716872
node_ip: 127.0.0.1
pid: 46633
time_since_restore: 0.10215306282043457
time_this_iter_s: 0.10215306282043457
time_total_s: 0.10215306282043457
timestamp: 1658500016
timesteps_since_restore: 0
training_iteration: 1
trial_id: 550c4faa
warmup_time: 0.0036542415618896484
Result for multi_objective_550962f4:
date: 2022-07-22_15-26-56
done: false
experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64
gain: 69.29543163212045
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
loss: 15.93172358423837
node_ip: 127.0.0.1
pid: 46631
time_since_restore: 0.10244321823120117
time_this_iter_s: 0.10244321823120117
time_total_s: 0.10244321823120117
timestamp: 1658500016
timesteps_since_restore: 0
training_iteration: 1
trial_id: 550962f4
warmup_time: 0.003384828567504883
Result for multi_objective_5378f1e8:
date: 2022-07-22_15-26-58
done: false
experiment_id: 4ce03cab420a449a9f4827c3b6b76c38
gain: -49.83538614085625
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
loss: -4.561341152769549
node_ip: 127.0.0.1
pid: 46621
time_since_restore: 5.1368408203125
time_this_iter_s: 0.1180417537689209
time_total_s: 5.1368408203125
timestamp: 1658500018
timesteps_since_restore: 0
training_iteration: 48
trial_id: 5378f1e8
warmup_time: 0.002785921096801758
Result for multi_objective_550ad9a4:
date: 2022-07-22_15-27-01
done: false
experiment_id: 102a4b06449b428b9870a75d02812be6
gain: -6.575568930702184
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
loss: -3.2532328725144097
node_ip: 127.0.0.1
pid: 46632
time_since_restore: 5.146106719970703
time_this_iter_s: 0.10509181022644043
time_total_s: 5.146106719970703
timestamp: 1658500021
timesteps_since_restore: 0
training_iteration: 48
trial_id: 550ad9a4
warmup_time: 0.0030989646911621094
Result for multi_objective_550c4faa:
date: 2022-07-22_15-27-01
done: false
experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d
gain: 26.760529995257745
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
loss: 4.73438862995905
node_ip: 127.0.0.1
pid: 46633
time_since_restore: 5.1661481857299805
time_this_iter_s: 0.10656309127807617
time_total_s: 5.1661481857299805
timestamp: 1658500021
timesteps_since_restore: 0
training_iteration: 48
trial_id: 550c4faa
warmup_time: 0.0036542415618896484
Result for multi_objective_550962f4:
date: 2022-07-22_15-27-01
done: false
experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64
gain: 27.828615153405433
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
loss: 6.398081286367129
node_ip: 127.0.0.1
pid: 46631
time_since_restore: 5.169018983840942
time_this_iter_s: 0.10616683959960938
time_total_s: 5.169018983840942
timestamp: 1658500021
timesteps_since_restore: 0
training_iteration: 48
trial_id: 550962f4
warmup_time: 0.003384828567504883
Result for multi_objective_5378f1e8:
date: 2022-07-22_15-27-03
done: false
experiment_id: 4ce03cab420a449a9f4827c3b6b76c38
gain: -50.86883186480195
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
loss: -4.65593053743818
node_ip: 127.0.0.1
pid: 46621
time_since_restore: 10.17048692703247
time_this_iter_s: 0.10630917549133301
time_total_s: 10.17048692703247
timestamp: 1658500023
timesteps_since_restore: 0
training_iteration: 95
trial_id: 5378f1e8
warmup_time: 0.002785921096801758
Result for multi_objective_5378f1e8:
date: 2022-07-22_15-27-04
done: true
experiment_id: 4ce03cab420a449a9f4827c3b6b76c38
experiment_tag: 1_activation=tanh,height=-47.5236,steps=100,width=10.9256
gain: -50.92155508732405
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
loss: -4.66075619693878
node_ip: 127.0.0.1
pid: 46621
time_since_restore: 10.706397771835327
time_this_iter_s: 0.10694766044616699
time_total_s: 10.706397771835327
timestamp: 1658500024
timesteps_since_restore: 0
training_iteration: 100
trial_id: 5378f1e8
warmup_time: 0.002785921096801758
Result for multi_objective_550ad9a4:
date: 2022-07-22_15-27-05
done: false
experiment_id: 102a4b06449b428b9870a75d02812be6
gain: -7.479635489825483
hostname: Kais-MacBook-Pro.local
iterations: 93
iterations_since_restore: 94
loss: -3.700515697784228
node_ip: 127.0.0.1
pid: 46632
time_since_restore: 10.056212902069092
time_this_iter_s: 0.10589313507080078
time_total_s: 10.056212902069092
timestamp: 1658500025
timesteps_since_restore: 0
training_iteration: 94
trial_id: 550ad9a4
warmup_time: 0.0030989646911621094
Result for multi_objective_550c4faa:
date: 2022-07-22_15-27-05
done: false
experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d
gain: 25.776502806907246
hostname: Kais-MacBook-Pro.local
iterations: 92
iterations_since_restore: 93
loss: 4.5602976409942295
node_ip: 127.0.0.1
pid: 46633
time_since_restore: 9.99274206161499
time_this_iter_s: 0.10583209991455078
time_total_s: 9.99274206161499
timestamp: 1658500025
timesteps_since_restore: 0
training_iteration: 93
trial_id: 550c4faa
warmup_time: 0.0036542415618896484
Result for multi_objective_550962f4:
date: 2022-07-22_15-27-05
done: false
experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64
gain: 26.860636112316993
hostname: Kais-MacBook-Pro.local
iterations: 92
iterations_since_restore: 93
loss: 6.175533072801938
node_ip: 127.0.0.1
pid: 46631
time_since_restore: 9.971143007278442
time_this_iter_s: 0.1054527759552002
time_total_s: 9.971143007278442
timestamp: 1658500025
timesteps_since_restore: 0
training_iteration: 93
trial_id: 550962f4
warmup_time: 0.003384828567504883
Result for multi_objective_5b6c4ba2:
date: 2022-07-22_15-27-06
done: false
experiment_id: f143c125a0734166ac832270ac08dbb6
gain: 430.90195437776714
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
loss: 25.491329340230966
node_ip: 127.0.0.1
pid: 46679
time_since_restore: 0.10215902328491211
time_this_iter_s: 0.10215902328491211
time_total_s: 0.10215902328491211
timestamp: 1658500026
timesteps_since_restore: 0
training_iteration: 1
trial_id: 5b6c4ba2
warmup_time: 0.002663135528564453
Result for multi_objective_550ad9a4:
date: 2022-07-22_15-27-07
done: true
experiment_id: 102a4b06449b428b9870a75d02812be6
experiment_tag: 3_activation=tanh,height=-42.0563,steps=100,width=2.0212
gain: -7.538566547680232
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
loss: -3.729671571085163
node_ip: 127.0.0.1
pid: 46632
time_since_restore: 11.27550482749939
time_this_iter_s: 0.10642290115356445
time_total_s: 11.27550482749939
timestamp: 1658500027
timesteps_since_restore: 0
training_iteration: 100
trial_id: 550ad9a4
warmup_time: 0.0030989646911621094
Result for multi_objective_550962f4:
date: 2022-07-22_15-27-07
done: true
experiment_id: b916abdd07bd4e7cb4e9a1451cd76d64
experiment_tag: 2_activation=relu,height=-40.6828,steps=100,width=4.3495
gain: 26.787356208375382
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
loss: 6.15868527856241
node_ip: 127.0.0.1
pid: 46631
time_since_restore: 11.394789934158325
time_this_iter_s: 0.10491585731506348
time_total_s: 11.394789934158325
timestamp: 1658500027
timesteps_since_restore: 0
training_iteration: 100
trial_id: 550962f4
warmup_time: 0.003384828567504883
Result for multi_objective_550c4faa:
date: 2022-07-22_15-27-07
done: true
experiment_id: 538d9621144b4ecea8c91ec7b70c8a9d
experiment_tag: 4_activation=relu,height=-56.2838,steps=100,width=5.6524
gain: 25.702421216002982
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
loss: 4.547191359402326
node_ip: 127.0.0.1
pid: 46633
time_since_restore: 11.406961917877197
time_this_iter_s: 0.12177181243896484
time_total_s: 11.406961917877197
timestamp: 1658500027
timesteps_since_restore: 0
training_iteration: 100
trial_id: 550c4faa
warmup_time: 0.0036542415618896484
Result for multi_objective_5d51f2d2:
date: 2022-07-22_15-27-09
done: false
experiment_id: 1433194530b14db4b77ce40a69d65407
gain: 60.23765295445784
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
loss: 11.657934184433229
node_ip: 127.0.0.1
pid: 46686
time_since_restore: 0.10274410247802734
time_this_iter_s: 0.10274410247802734
time_total_s: 0.10274410247802734
timestamp: 1658500029
timesteps_since_restore: 0
training_iteration: 1
trial_id: 5d51f2d2
warmup_time: 0.0028769969940185547
Result for multi_objective_5d636760:
date: 2022-07-22_15-27-10
done: false
experiment_id: b8dccac5d8914569b1c2ca061ad47a56
gain: 117.14725547845826
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
loss: 8.611046741256526
node_ip: 127.0.0.1
pid: 46689
time_since_restore: 0.10460686683654785
time_this_iter_s: 0.10460686683654785
time_total_s: 0.10460686683654785
timestamp: 1658500030
timesteps_since_restore: 0
training_iteration: 1
trial_id: 5d636760
warmup_time: 0.0028090476989746094
Result for multi_objective_5d66093e:
date: 2022-07-22_15-27-10
done: false
experiment_id: e7259e986278458899b46b485e069ec9
gain: 294.00300634729234
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
loss: 14.832864636325777
node_ip: 127.0.0.1
pid: 46690
time_since_restore: 0.10294318199157715
time_this_iter_s: 0.10294318199157715
time_total_s: 0.10294318199157715
timestamp: 1658500030
timesteps_since_restore: 0
training_iteration: 1
trial_id: 5d66093e
warmup_time: 0.0027511119842529297
Result for multi_objective_5b6c4ba2:
date: 2022-07-22_15-27-11
done: false
experiment_id: f143c125a0734166ac832270ac08dbb6
gain: 263.92128797150474
hostname: Kais-MacBook-Pro.local
iterations: 48
iterations_since_restore: 49
loss: 15.6130748612977
node_ip: 127.0.0.1
pid: 46679
time_since_restore: 5.2078468799591064
time_this_iter_s: 0.10803008079528809
time_total_s: 5.2078468799591064
timestamp: 1658500031
timesteps_since_restore: 0
training_iteration: 49
trial_id: 5b6c4ba2
warmup_time: 0.002663135528564453
Result for multi_objective_5d51f2d2:
date: 2022-07-22_15-27-14
done: false
experiment_id: 1433194530b14db4b77ce40a69d65407
gain: 10.610216987384797
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
loss: 2.053420165872076
node_ip: 127.0.0.1
pid: 46686
time_since_restore: 5.178762912750244
time_this_iter_s: 0.10710978507995605
time_total_s: 5.178762912750244
timestamp: 1658500034
timesteps_since_restore: 0
training_iteration: 48
trial_id: 5d51f2d2
warmup_time: 0.0028769969940185547
Result for multi_objective_5d636760:
date: 2022-07-22_15-27-15
done: false
experiment_id: b8dccac5d8914569b1c2ca061ad47a56
gain: -16.800839364465826
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
loss: -1.234965449842409
node_ip: 127.0.0.1
pid: 46689
time_since_restore: 5.146665811538696
time_this_iter_s: 0.10526585578918457
time_total_s: 5.146665811538696
timestamp: 1658500035
timesteps_since_restore: 0
training_iteration: 48
trial_id: 5d636760
warmup_time: 0.0028090476989746094
Result for multi_objective_5d66093e:
date: 2022-07-22_15-27-15
done: false
experiment_id: e7259e986278458899b46b485e069ec9
gain: 97.89753254610552
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
loss: 4.939068026982622
node_ip: 127.0.0.1
pid: 46690
time_since_restore: 5.1532862186431885
time_this_iter_s: 0.10752320289611816
time_total_s: 5.1532862186431885
timestamp: 1658500035
timesteps_since_restore: 0
training_iteration: 48
trial_id: 5d66093e
warmup_time: 0.0027511119842529297
Result for multi_objective_5b6c4ba2:
date: 2022-07-22_15-27-16
done: false
experiment_id: f143c125a0734166ac832270ac08dbb6
gain: 262.90943552209455
hostname: Kais-MacBook-Pro.local
iterations: 95
iterations_since_restore: 96
loss: 15.553215620072208
node_ip: 127.0.0.1
pid: 46679
time_since_restore: 10.240519762039185
time_this_iter_s: 0.10443782806396484
time_total_s: 10.240519762039185
timestamp: 1658500036
timesteps_since_restore: 0
training_iteration: 96
trial_id: 5b6c4ba2
warmup_time: 0.002663135528564453
Result for multi_objective_5b6c4ba2:
date: 2022-07-22_15-27-17
done: true
experiment_id: f143c125a0734166ac832270ac08dbb6
experiment_tag: 5_activation=relu,height=54.9133,steps=100,width=16.9039
gain: 262.86741923011425
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
loss: 15.550730017197484
node_ip: 127.0.0.1
pid: 46679
time_since_restore: 10.669770956039429
time_this_iter_s: 0.10598993301391602
time_total_s: 10.669770956039429
timestamp: 1658500037
timesteps_since_restore: 0
training_iteration: 100
trial_id: 5b6c4ba2
warmup_time: 0.002663135528564453
Result for multi_objective_634c9444:
date: 2022-07-22_15-27-19
done: false
experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee
gain: 283.55076351521427
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
loss: 18.766157678691147
node_ip: 127.0.0.1
pid: 46710
time_since_restore: 0.10465264320373535
time_this_iter_s: 0.10465264320373535
time_total_s: 0.10465264320373535
timestamp: 1658500039
timesteps_since_restore: 0
training_iteration: 1
trial_id: 634c9444
warmup_time: 0.004335880279541016
Result for multi_objective_5d51f2d2:
date: 2022-07-22_15-27-20
done: false
experiment_id: 1433194530b14db4b77ce40a69d65407
gain: 9.60907229177291
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
loss: 1.8596662860626851
node_ip: 127.0.0.1
pid: 46686
time_since_restore: 10.218497037887573
time_this_iter_s: 0.10553288459777832
time_total_s: 10.218497037887573
timestamp: 1658500040
timesteps_since_restore: 0
training_iteration: 95
trial_id: 5d51f2d2
warmup_time: 0.0028769969940185547
Result for multi_objective_5d636760:
date: 2022-07-22_15-27-20
done: false
experiment_id: b8dccac5d8914569b1c2ca061ad47a56
gain: -17.840160182810624
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
loss: -1.311361948500235
node_ip: 127.0.0.1
pid: 46689
time_since_restore: 10.16199278831482
time_this_iter_s: 0.10700488090515137
time_total_s: 10.16199278831482
timestamp: 1658500040
timesteps_since_restore: 0
training_iteration: 95
trial_id: 5d636760
warmup_time: 0.0028090476989746094
Result for multi_objective_5d66093e:
date: 2022-07-22_15-27-20
done: false
experiment_id: e7259e986278458899b46b485e069ec9
gain: 96.85061995053432
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
loss: 4.886249816008858
node_ip: 127.0.0.1
pid: 46690
time_since_restore: 10.197196006774902
time_this_iter_s: 0.10463690757751465
time_total_s: 10.197196006774902
timestamp: 1658500040
timesteps_since_restore: 0
training_iteration: 95
trial_id: 5d66093e
warmup_time: 0.0027511119842529297
Result for multi_objective_5d51f2d2:
date: 2022-07-22_15-27-20
done: true
experiment_id: 1433194530b14db4b77ce40a69d65407
experiment_tag: 6_activation=tanh,height=16.5793,steps=100,width=5.1671
gain: 9.557436804323405
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
loss: 1.8496731491334866
node_ip: 127.0.0.1
pid: 46686
time_since_restore: 10.745142936706543
time_this_iter_s: 0.10509181022644043
time_total_s: 10.745142936706543
timestamp: 1658500040
timesteps_since_restore: 0
training_iteration: 100
trial_id: 5d51f2d2
warmup_time: 0.0028769969940185547
Result for multi_objective_5d636760:
date: 2022-07-22_15-27-20
done: true
experiment_id: b8dccac5d8914569b1c2ca061ad47a56
experiment_tag: 7_activation=tanh,height=-13.8895,steps=100,width=13.6043
gain: -17.893079155032332
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
loss: -1.3152518197690004
node_ip: 127.0.0.1
pid: 46689
time_since_restore: 10.689707040786743
time_this_iter_s: 0.10322403907775879
time_total_s: 10.689707040786743
timestamp: 1658500040
timesteps_since_restore: 0
training_iteration: 100
trial_id: 5d636760
warmup_time: 0.0028090476989746094
Result for multi_objective_5d66093e:
date: 2022-07-22_15-27-20
done: true
experiment_id: e7259e986278458899b46b485e069ec9
experiment_tag: 8_activation=relu,height=-51.6714,steps=100,width=19.8211
gain: 96.79744897015512
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
loss: 4.8835672653630535
node_ip: 127.0.0.1
pid: 46690
time_since_restore: 10.72464108467102
time_this_iter_s: 0.10328292846679688
time_total_s: 10.72464108467102
timestamp: 1658500040
timesteps_since_restore: 0
training_iteration: 100
trial_id: 5d66093e
warmup_time: 0.0027511119842529297
Result for multi_objective_65469a2e:
date: 2022-07-22_15-27-23
done: false
experiment_id: 015ee29ddb8b4c239d85a022a23d4093
gain: 162.96737941207124
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
loss: 11.96774070054979
node_ip: 127.0.0.1
pid: 46719
time_since_restore: 0.10403180122375488
time_this_iter_s: 0.10403180122375488
time_total_s: 0.10403180122375488
timestamp: 1658500043
timesteps_since_restore: 0
training_iteration: 1
trial_id: 65469a2e
warmup_time: 0.0033850669860839844
Result for multi_objective_634c9444:
date: 2022-07-22_15-27-24
done: false
experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee
gain: 136.50874589912112
hostname: Kais-MacBook-Pro.local
iterations: 24
iterations_since_restore: 25
loss: 9.034518610724309
node_ip: 127.0.0.1
pid: 46710
time_since_restore: 5.125189781188965
time_this_iter_s: 0.11187601089477539
time_total_s: 5.125189781188965
timestamp: 1658500044
timesteps_since_restore: 0
training_iteration: 25
trial_id: 634c9444
warmup_time: 0.004335880279541016
Result for multi_objective_65469a2e:
date: 2022-07-22_15-27-28
done: false
experiment_id: 015ee29ddb8b4c239d85a022a23d4093
gain: 28.890088364068774
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
loss: 2.1215846238952016
node_ip: 127.0.0.1
pid: 46719
time_since_restore: 5.174077749252319
time_this_iter_s: 0.10734677314758301
time_total_s: 5.174077749252319
timestamp: 1658500048
timesteps_since_restore: 0
training_iteration: 48
trial_id: 65469a2e
warmup_time: 0.0033850669860839844
Result for multi_objective_634c9444:
date: 2022-07-22_15-27-30
done: false
experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee
gain: 133.84933935751317
hostname: Kais-MacBook-Pro.local
iterations: 71
iterations_since_restore: 72
loss: 8.85851188137237
node_ip: 127.0.0.1
pid: 46710
time_since_restore: 10.204095840454102
time_this_iter_s: 0.10942888259887695
time_total_s: 10.204095840454102
timestamp: 1658500050
timesteps_since_restore: 0
training_iteration: 72
trial_id: 634c9444
warmup_time: 0.004335880279541016
Result for multi_objective_634c9444:
date: 2022-07-22_15-27-33
done: true
experiment_id: 1c6a5d9e38394f1ca770da6a92b6a0ee
experiment_tag: 9_activation=relu,height=-12.3384,steps=100,width=15.1097
gain: 133.45728949146132
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
loss: 8.83256495915982
node_ip: 127.0.0.1
pid: 46710
time_since_restore: 13.198630809783936
time_this_iter_s: 0.1067969799041748
time_total_s: 13.198630809783936
timestamp: 1658500053
timesteps_since_restore: 0
training_iteration: 100
trial_id: 634c9444
warmup_time: 0.004335880279541016
Result for multi_objective_65469a2e:
date: 2022-07-22_15-27-33
done: false
experiment_id: 015ee29ddb8b4c239d85a022a23d4093
gain: 27.850744704436202
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
loss: 2.0452589477867855
node_ip: 127.0.0.1
pid: 46719
time_since_restore: 10.19201397895813
time_this_iter_s: 0.10735702514648438
time_total_s: 10.19201397895813
timestamp: 1658500053
timesteps_since_restore: 0
training_iteration: 95
trial_id: 65469a2e
warmup_time: 0.0033850669860839844
Result for multi_objective_65469a2e:
date: 2022-07-22_15-27-34
done: true
experiment_id: 015ee29ddb8b4c239d85a022a23d4093
experiment_tag: 10_activation=tanh,height=19.6774,steps=100,width=13.6172
gain: 27.797824972416166
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
loss: 2.041372712191397
node_ip: 127.0.0.1
pid: 46719
time_since_restore: 10.727944135665894
time_this_iter_s: 0.10885024070739746
time_total_s: 10.727944135665894
timestamp: 1658500054
timesteps_since_restore: 0
training_iteration: 100
trial_id: 65469a2e
warmup_time: 0.0033850669860839844
Now there are two hyperparameter sets for the two objectives.
print("Best hyperparameters for loss found were: ", results.get_best_result("loss", "min").config)
print("Best hyperparameters for gain found were: ", results.get_best_result("gain", "max").config)
Best hyperparameters for loss found were: {'steps': 100, 'width': 10.925599395387751, 'height': -47.52361963011387, 'activation': 'tanh'}
Best hyperparameters for gain found were: {'steps': 100, 'width': 16.90386360893735, 'height': 54.91329340230965, 'activation': 'relu'}
We can mix-and-match the use of initial hyperparameter evaluations, conditional search spaces via define-by-run functions, and multi-objective tasks. This is also true of scheduler usage, with the exception of multi-objective optimization– schedulers typically rely on a single scalar score, rather than the two scores we use here: loss, gain.