Running Tune experiments with BlendSearch and CFO
Contents
Running Tune experiments with BlendSearch and CFO#
In this tutorial we introduce BlendSearch and CFO, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with FLAML and, as a result, allow you to seamlessly scale up a BlendSearch and CFO optimization process - without sacrificing performance.
Fast Library for Automated Machine Learning & Tuning (FLAML) does not rely on the gradient of the objective function, but instead, learns from samples of the search space. It is suitable for optimizing functions that are non-differentiable, with many local minima, or even unknown but only testable. Therefore, it is necessarily belongs to the domain of “derivative-free optimization” and “black-box optimization”.
FLAML has two primary algorithms: (1) Frugal Optimization for Cost-related Hyperparameters (CFO) begins with a low-cost initial point and gradually moves to a high-cost region as needed. It is a local search method that leverages randomized direct search method with an adaptive step-size and random restarts. As a local search method, it has an appealing provable convergence rate and bounded cost but may get trapped in suboptimal local minima. (2) Economical Hyperparameter Optimization With Blended Search Strategy (BlendSearch) combines CFO’s local search with global search, making it less suspectable to local minima traps. It leverages the frugality of CFO and the space exploration ability of global search methods such as Bayesian optimization.
In this example we minimize a simple objective to briefly demonstrate the usage of
FLAML with Ray Tune via BlendSearch and CFO. It’s useful to keep in mind that
despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit
or explicit objective. Here we assume flaml==1.1.1 and optuna==2.8.0 libraries
are installed. To learn more, please refer to
the FLAML website.
Click below to see all the imports we need for this example. You can also launch directly into a Binder instance to run this notebook yourself. Just click on the rocket symbol at the top of the navigation.
import time
import ray
from ray import tune
from ray.air import session
from ray.tune.search import ConcurrencyLimiter
from ray.tune.search.flaml import BlendSearch, CFO
/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/flaml/searcher/blendsearch.py:14: DeprecationWarning: The module `ray.tune.suggest` has been moved to `ray.tune.search` and the old location will be deprecated soon. Please adjust your imports to point to the new location. Example: Do a global search and replace `ray.tune.suggest` with `ray.tune.search`.
from ray.tune.suggest import Searcher
/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/flaml/searcher/blendsearch.py:15: DeprecationWarning: The module `ray.tune.suggest.optuna` has been moved to `ray.tune.search.optuna` and the old location will be deprecated soon. Please adjust your imports to point to the new location. Example: Do a global search and replace `ray.tune.suggest.optuna` with `ray.tune.search.optuna`.
from ray.tune.suggest.optuna import OptunaSearch as GlobalSearch
/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/flaml/tune/sample.py:39: DeprecationWarning: The module `ray.tune.sample` has been moved to `ray.tune.search.sample` and the old location will be deprecated soon. Please adjust your imports to point to the new location. Example: Do a global search and replace `ray.tune.sample` with `ray.tune.search.sample`.
from ray.tune.sample import _BackwardsCompatibleNumpyRng
/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/flaml/tune/space.py:6: DeprecationWarning: The module `ray.tune.suggest.variant_generator` has been moved to `ray.tune.search.variant_generator` and the old location will be deprecated soon. Please adjust your imports to point to the new location. Example: Do a global search and replace `ray.tune.suggest.variant_generator` with `ray.tune.search.variant_generator`.
from ray.tune.suggest.variant_generator import generate_variants
Let’s start by defining a simple evaluation function.
We artificially sleep for a bit (0.1 seconds) to simulate a long-running ML experiment.
This setup assumes that we’re running multiple steps of an experiment and try to
tune three hyperparameters, namely width and height, and activation.
def evaluate(step, width, height, activation):
time.sleep(0.1)
activation_boost = 10 if activation=="relu" else 1
return (0.1 + width * step / 100) ** (-1) + height * 0.1 + activation_boost
Next, our objective function takes a Tune config, evaluates the score of your
experiment in a training loop, and uses session.report to report the score back to Tune.
def objective(config):
for step in range(config["steps"]):
score = evaluate(step, config["width"], config["height"], config["activation"])
session.report({"iterations": step, "mean_loss": score})
Running Tune experiments with BlendSearch#
This example demonstrates the usage of Economical Hyperparameter Optimization With Blended Search Strategy (BlendSearch) with Ray Tune.
Now we define the search algorithm built from BlendSearch, constrained to a
maximum of 4 concurrent trials with a ConcurrencyLimiter.
algo = BlendSearch()
algo = ConcurrencyLimiter(algo, max_concurrent=4)
The number of samples this Tune run is set to 1000.
(you can decrease this if it takes too long on your machine).
num_samples = 1000
Next we define a search space. The critical assumption is that the optimal hyperparameters live within this space. Yet, if the space is very large, then those hyperparameters may be difficult to find in a short amount of time.
search_config = {
"steps": 100,
"width": tune.uniform(0, 20),
"height": tune.uniform(-100, 100),
"activation": tune.choice(["relu, tanh"])
}
Finally, we run the experiment to "min"imize the “mean_loss” of the objective by
searching search_config via algo, num_samples times. This previous sentence is
fully characterizes the search problem we aim to solve. With this in mind, observe
how efficient it is to execute tuner.fit().
tuner = tune.Tuner(
objective,
tune_config=tune.TuneConfig(
metric="mean_loss",
mode="min",
search_alg=algo,
num_samples=num_samples,
),
param_space=search_config,
)
results = tuner.fit()
Function checkpointing is disabled. This may result in unexpected behavior when using checkpointing features or certain schedulers. To enable, set the train function arguments to be `func(config, checkpoint_dir=None)`.
[I 2022-07-22 15:18:10,659] A new study created in memory with name: optuna
== Status ==
Current time: 2022-07-22 15:18:55 (running for 00:00:43.79)
Memory usage on this node: 10.0/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/4.69 GiB heap, 0.0/2.0 GiB objects
Current best trial: 1fc231f8 with mean_loss=-8.519908298508359 and parameters={'steps': 100, 'width': 15.42641286533492, 'height': -95.8496101281197, 'activation': 'relu, tanh'}
Result logdir: /Users/kai/ray_results/blendsearch_exp
Number of trials: 10/10 (10 TERMINATED)
Current time: 2022-07-22 15:18:55 (running for 00:00:43.79)
Memory usage on this node: 10.0/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/4.69 GiB heap, 0.0/2.0 GiB objects
Current best trial: 1fc231f8 with mean_loss=-8.519908298508359 and parameters={'steps': 100, 'width': 15.42641286533492, 'height': -95.8496101281197, 'activation': 'relu, tanh'}
Result logdir: /Users/kai/ray_results/blendsearch_exp
Number of trials: 10/10 (10 TERMINATED)
| Trial name | status | loc | activation | height | steps | width | loss | iter | total time (s) | iterations | neg_mean_loss |
|---|---|---|---|---|---|---|---|---|---|---|---|
| objective_1e0f814e | TERMINATED | 127.0.0.1:45801 | relu, tanh | 29.4532 | 100 | 1.94864 | 4.43814 | 100 | 11.061 | 99 | -4.43814 |
| objective_1fc231f8 | TERMINATED | 127.0.0.1:45809 | relu, tanh | -95.8496 | 100 | 15.4264 | -8.51991 | 100 | 11.6807 | 99 | 8.51991 |
| objective_1fc3b668 | TERMINATED | 127.0.0.1:45810 | relu, tanh | 49.7608 | 100 | 12.673 | 6.05515 | 100 | 11.7372 | 99 | -6.05515 |
| objective_1fc58b1e | TERMINATED | 127.0.0.1:45811 | relu, tanh | -55.0407 | 100 | 9.97014 | -4.40377 | 100 | 11.6335 | 99 | 4.40377 |
| objective_265c09ee | TERMINATED | 127.0.0.1:45837 | relu, tanh | 52.1061 | 100 | 3.96126 | 6.45927 | 100 | 10.7539 | 99 | -6.45927 |
| objective_28587494 | TERMINATED | 127.0.0.1:45842 | relu, tanh | -82.332 | 100 | 3.38222 | -6.94321 | 100 | 10.7116 | 99 | 6.94321 |
| objective_28682bfa | TERMINATED | 127.0.0.1:45845 | relu, tanh | -94.9771 | 100 | 17.681 | -8.4409 | 100 | 10.7471 | 99 | 8.4409 |
| objective_28788388 | TERMINATED | 127.0.0.1:45848 | relu, tanh | 90.6787 | 100 | 13.7072 | 10.141 | 100 | 10.7525 | 99 | -10.141 |
| objective_2e3cd63e | TERMINATED | 127.0.0.1:45864 | relu, tanh | 2.43845 | 100 | 0.0789653 | 6.85628 | 100 | 10.7006 | 99 | -6.85628 |
| objective_3046455a | TERMINATED | 127.0.0.1:45869 | relu, tanh | 22.5052 | 100 | 16.2524 | 3.31229 | 100 | 10.7176 | 99 | -3.31229 |
Result for objective_1e0f814e:
date: 2022-07-22_15-18-14
done: false
experiment_id: 623da6c1aa28400a9d85da19bdc5721d
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 13.945321626643842
neg_mean_loss: -13.945321626643842
node_ip: 127.0.0.1
pid: 45801
time_since_restore: 0.10402607917785645
time_this_iter_s: 0.10402607917785645
time_total_s: 0.10402607917785645
timestamp: 1658499494
timesteps_since_restore: 0
training_iteration: 1
trial_id: 1e0f814e
warmup_time: 0.003920078277587891
Result for objective_1fc58b1e:
date: 2022-07-22_15-18-17
done: false
experiment_id: da508e209a1946e89ec15a3f1a5ce5ef
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 5.495932910616953
neg_mean_loss: -5.495932910616953
node_ip: 127.0.0.1
pid: 45811
time_since_restore: 0.10269379615783691
time_this_iter_s: 0.10269379615783691
time_total_s: 0.10269379615783691
timestamp: 1658499497
timesteps_since_restore: 0
training_iteration: 1
trial_id: 1fc58b1e
warmup_time: 0.004099130630493164
Result for objective_1fc231f8:
date: 2022-07-22_15-18-17
done: false
experiment_id: 24c1fa443be246229e2c2ed9581452de
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 1.4150389871880282
neg_mean_loss: -1.4150389871880282
node_ip: 127.0.0.1
pid: 45809
time_since_restore: 0.10376811027526855
time_this_iter_s: 0.10376811027526855
time_total_s: 0.10376811027526855
timestamp: 1658499497
timesteps_since_restore: 0
training_iteration: 1
trial_id: 1fc231f8
warmup_time: 0.004148244857788086
Result for objective_1fc3b668:
date: 2022-07-22_15-18-17
done: false
experiment_id: 5cdd489afb0e43b7a44aa4484765dc92
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 15.976077650772236
neg_mean_loss: -15.976077650772236
node_ip: 127.0.0.1
pid: 45810
time_since_restore: 0.1012420654296875
time_this_iter_s: 0.1012420654296875
time_total_s: 0.1012420654296875
timestamp: 1658499497
timesteps_since_restore: 0
training_iteration: 1
trial_id: 1fc3b668
warmup_time: 0.002666950225830078
Result for objective_1e0f814e:
date: 2022-07-22_15-18-19
done: false
experiment_id: 623da6c1aa28400a9d85da19bdc5721d
hostname: Kais-MacBook-Pro.local
iterations: 44
iterations_since_restore: 45
mean_loss: 4.989814690087961
neg_mean_loss: -4.989814690087961
node_ip: 127.0.0.1
pid: 45801
time_since_restore: 5.1339499950408936
time_this_iter_s: 0.10842394828796387
time_total_s: 5.1339499950408936
timestamp: 1658499499
timesteps_since_restore: 0
training_iteration: 45
trial_id: 1e0f814e
warmup_time: 0.003920078277587891
Result for objective_1fc58b1e:
date: 2022-07-22_15-18-22
done: false
experiment_id: da508e209a1946e89ec15a3f1a5ce5ef
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -4.295122851662673
neg_mean_loss: 4.295122851662673
node_ip: 127.0.0.1
pid: 45811
time_since_restore: 5.167006015777588
time_this_iter_s: 0.10817217826843262
time_total_s: 5.167006015777588
timestamp: 1658499502
timesteps_since_restore: 0
training_iteration: 48
trial_id: 1fc58b1e
warmup_time: 0.004099130630493164
Result for objective_1fc231f8:
date: 2022-07-22_15-18-22
done: false
experiment_id: 24c1fa443be246229e2c2ed9581452de
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -8.44891425494968
neg_mean_loss: 8.44891425494968
node_ip: 127.0.0.1
pid: 45809
time_since_restore: 5.138737916946411
time_this_iter_s: 0.10868191719055176
time_total_s: 5.138737916946411
timestamp: 1658499502
timesteps_since_restore: 0
training_iteration: 48
trial_id: 1fc231f8
warmup_time: 0.004148244857788086
Result for objective_1fc3b668:
date: 2022-07-22_15-18-22
done: false
experiment_id: 5cdd489afb0e43b7a44aa4484765dc92
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 6.141195146339472
neg_mean_loss: -6.141195146339472
node_ip: 127.0.0.1
pid: 45810
time_since_restore: 5.181504011154175
time_this_iter_s: 0.10758399963378906
time_total_s: 5.181504011154175
timestamp: 1658499502
timesteps_since_restore: 0
training_iteration: 48
trial_id: 1fc3b668
warmup_time: 0.002666950225830078
Result for objective_1e0f814e:
date: 2022-07-22_15-18-24
done: false
experiment_id: 623da6c1aa28400a9d85da19bdc5721d
hostname: Kais-MacBook-Pro.local
iterations: 91
iterations_since_restore: 92
mean_loss: 4.479149291114477
neg_mean_loss: -4.479149291114477
node_ip: 127.0.0.1
pid: 45801
time_since_restore: 10.196602821350098
time_this_iter_s: 0.10757303237915039
time_total_s: 10.196602821350098
timestamp: 1658499504
timesteps_since_restore: 0
training_iteration: 92
trial_id: 1e0f814e
warmup_time: 0.003920078277587891
Result for objective_1e0f814e:
date: 2022-07-22_15-18-25
done: true
experiment_id: 623da6c1aa28400a9d85da19bdc5721d
experiment_tag: 1_activation=relu_tanh,height=29.4532,steps=100,width=1.9486
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 4.438137591322411
neg_mean_loss: -4.438137591322411
node_ip: 127.0.0.1
pid: 45801
time_since_restore: 11.060971975326538
time_this_iter_s: 0.10748600959777832
time_total_s: 11.060971975326538
timestamp: 1658499505
timesteps_since_restore: 0
training_iteration: 100
trial_id: 1e0f814e
warmup_time: 0.003920078277587891
Result for objective_1fc231f8:
date: 2022-07-22_15-18-26
done: false
experiment_id: 24c1fa443be246229e2c2ed9581452de
hostname: Kais-MacBook-Pro.local
iterations: 90
iterations_since_restore: 91
mean_loss: -8.513449547227397
neg_mean_loss: 8.513449547227397
node_ip: 127.0.0.1
pid: 45809
time_since_restore: 9.72878909111023
time_this_iter_s: 0.10644912719726562
time_total_s: 9.72878909111023
timestamp: 1658499506
timesteps_since_restore: 0
training_iteration: 91
trial_id: 1fc231f8
warmup_time: 0.004148244857788086
Result for objective_1fc3b668:
date: 2022-07-22_15-18-26
done: false
experiment_id: 5cdd489afb0e43b7a44aa4484765dc92
hostname: Kais-MacBook-Pro.local
iterations: 89
iterations_since_restore: 90
mean_loss: 6.063959309753095
neg_mean_loss: -6.063959309753095
node_ip: 127.0.0.1
pid: 45810
time_since_restore: 9.6911301612854
time_this_iter_s: 0.10597920417785645
time_total_s: 9.6911301612854
timestamp: 1658499506
timesteps_since_restore: 0
training_iteration: 90
trial_id: 1fc3b668
warmup_time: 0.002666950225830078
Result for objective_1fc58b1e:
date: 2022-07-22_15-18-27
done: false
experiment_id: da508e209a1946e89ec15a3f1a5ce5ef
hostname: Kais-MacBook-Pro.local
iterations: 91
iterations_since_restore: 92
mean_loss: -4.395049451529077
neg_mean_loss: 4.395049451529077
node_ip: 127.0.0.1
pid: 45811
time_since_restore: 9.873695850372314
time_this_iter_s: 0.10703802108764648
time_total_s: 9.873695850372314
timestamp: 1658499507
timesteps_since_restore: 0
training_iteration: 92
trial_id: 1fc58b1e
warmup_time: 0.004099130630493164
Result for objective_265c09ee:
date: 2022-07-22_15-18-28
done: false
experiment_id: a7543e8d696745edbfe36a6f4ebe4071
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 16.210614243979172
neg_mean_loss: -16.210614243979172
node_ip: 127.0.0.1
pid: 45837
time_since_restore: 0.1049339771270752
time_this_iter_s: 0.1049339771270752
time_total_s: 0.1049339771270752
timestamp: 1658499508
timesteps_since_restore: 0
training_iteration: 1
trial_id: 265c09ee
warmup_time: 0.0026078224182128906
Result for objective_1fc58b1e:
date: 2022-07-22_15-18-28
done: true
experiment_id: da508e209a1946e89ec15a3f1a5ce5ef
experiment_tag: 4_activation=relu_tanh,height=-55.0407,steps=100,width=9.9701
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -4.403770601366095
neg_mean_loss: 4.403770601366095
node_ip: 127.0.0.1
pid: 45811
time_since_restore: 11.633522987365723
time_this_iter_s: 0.10734701156616211
time_total_s: 11.633522987365723
timestamp: 1658499508
timesteps_since_restore: 0
training_iteration: 100
trial_id: 1fc58b1e
warmup_time: 0.004099130630493164
Result for objective_1fc231f8:
date: 2022-07-22_15-18-28
done: true
experiment_id: 24c1fa443be246229e2c2ed9581452de
experiment_tag: 2_activation=relu_tanh,height=-95.8496,steps=100,width=15.4264
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -8.519908298508359
neg_mean_loss: 8.519908298508359
node_ip: 127.0.0.1
pid: 45809
time_since_restore: 11.680676937103271
time_this_iter_s: 0.10814285278320312
time_total_s: 11.680676937103271
timestamp: 1658499508
timesteps_since_restore: 0
training_iteration: 100
trial_id: 1fc231f8
warmup_time: 0.004148244857788086
Result for objective_1fc3b668:
date: 2022-07-22_15-18-29
done: true
experiment_id: 5cdd489afb0e43b7a44aa4484765dc92
experiment_tag: 3_activation=relu_tanh,height=49.7608,steps=100,width=12.6730
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 6.055152568795227
neg_mean_loss: -6.055152568795227
node_ip: 127.0.0.1
pid: 45810
time_since_restore: 11.73720097541809
time_this_iter_s: 0.10816287994384766
time_total_s: 11.73720097541809
timestamp: 1658499509
timesteps_since_restore: 0
training_iteration: 100
trial_id: 1fc3b668
warmup_time: 0.002666950225830078
Result for objective_28587494:
date: 2022-07-22_15-18-31
done: false
experiment_id: 750bd707fda147f0bd615c32243d64e5
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 2.766796283480206
neg_mean_loss: -2.766796283480206
node_ip: 127.0.0.1
pid: 45842
time_since_restore: 0.10265803337097168
time_this_iter_s: 0.10265803337097168
time_total_s: 0.10265803337097168
timestamp: 1658499511
timesteps_since_restore: 0
training_iteration: 1
trial_id: '28587494'
warmup_time: 0.0028159618377685547
Result for objective_28682bfa:
date: 2022-07-22_15-18-31
done: false
experiment_id: 5487e6a1cf70423096cf6285197fb824
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 1.5022935169945555
neg_mean_loss: -1.5022935169945555
node_ip: 127.0.0.1
pid: 45845
time_since_restore: 0.10289216041564941
time_this_iter_s: 0.10289216041564941
time_total_s: 0.10289216041564941
timestamp: 1658499511
timesteps_since_restore: 0
training_iteration: 1
trial_id: 28682bfa
warmup_time: 0.003072023391723633
Result for objective_28788388:
date: 2022-07-22_15-18-31
done: false
experiment_id: e959dae493a240c489a3193352c46f3a
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 20.067866923898734
neg_mean_loss: -20.067866923898734
node_ip: 127.0.0.1
pid: 45848
time_since_restore: 0.1038503646850586
time_this_iter_s: 0.1038503646850586
time_total_s: 0.1038503646850586
timestamp: 1658499511
timesteps_since_restore: 0
training_iteration: 1
trial_id: '28788388'
warmup_time: 0.0027070045471191406
Result for objective_265c09ee:
date: 2022-07-22_15-18-33
done: false
experiment_id: a7543e8d696745edbfe36a6f4ebe4071
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 6.720352557756592
neg_mean_loss: -6.720352557756592
node_ip: 127.0.0.1
pid: 45837
time_since_restore: 5.148331165313721
time_this_iter_s: 0.10472607612609863
time_total_s: 5.148331165313721
timestamp: 1658499513
timesteps_since_restore: 0
training_iteration: 48
trial_id: 265c09ee
warmup_time: 0.0026078224182128906
Result for objective_28587494:
date: 2022-07-22_15-18-36
done: false
experiment_id: 750bd707fda147f0bd615c32243d64e5
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -6.641362320132167
neg_mean_loss: 6.641362320132167
node_ip: 127.0.0.1
pid: 45842
time_since_restore: 5.175055980682373
time_this_iter_s: 0.10797286033630371
time_total_s: 5.175055980682373
timestamp: 1658499516
timesteps_since_restore: 0
training_iteration: 48
trial_id: '28587494'
warmup_time: 0.0028159618377685547
Result for objective_28682bfa:
date: 2022-07-22_15-18-36
done: false
experiment_id: 5487e6a1cf70423096cf6285197fb824
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -8.378801379858526
neg_mean_loss: 8.378801379858526
node_ip: 127.0.0.1
pid: 45845
time_since_restore: 5.192234039306641
time_this_iter_s: 0.10880398750305176
time_total_s: 5.192234039306641
timestamp: 1658499516
timesteps_since_restore: 0
training_iteration: 48
trial_id: 28682bfa
warmup_time: 0.003072023391723633
Result for objective_28788388:
date: 2022-07-22_15-18-36
done: false
experiment_id: e959dae493a240c489a3193352c46f3a
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 10.22071644495505
neg_mean_loss: -10.22071644495505
node_ip: 127.0.0.1
pid: 45848
time_since_restore: 5.175441265106201
time_this_iter_s: 0.10689902305603027
time_total_s: 5.175441265106201
timestamp: 1658499516
timesteps_since_restore: 0
training_iteration: 48
trial_id: '28788388'
warmup_time: 0.0027070045471191406
Result for objective_265c09ee:
date: 2022-07-22_15-18-38
done: false
experiment_id: a7543e8d696745edbfe36a6f4ebe4071
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 6.472149118155721
neg_mean_loss: -6.472149118155721
node_ip: 127.0.0.1
pid: 45837
time_since_restore: 10.2187819480896
time_this_iter_s: 0.10822892189025879
time_total_s: 10.2187819480896
timestamp: 1658499518
timesteps_since_restore: 0
training_iteration: 95
trial_id: 265c09ee
warmup_time: 0.0026078224182128906
Result for objective_265c09ee:
date: 2022-07-22_15-18-38
done: true
experiment_id: a7543e8d696745edbfe36a6f4ebe4071
experiment_tag: 5_activation=relu_tanh,height=52.1061,steps=100,width=3.9613
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 6.459268729660066
neg_mean_loss: -6.459268729660066
node_ip: 127.0.0.1
pid: 45837
time_since_restore: 10.753926277160645
time_this_iter_s: 0.10714435577392578
time_total_s: 10.753926277160645
timestamp: 1658499518
timesteps_since_restore: 0
training_iteration: 100
trial_id: 265c09ee
warmup_time: 0.0026078224182128906
Result for objective_2e3cd63e:
date: 2022-07-22_15-18-41
done: false
experiment_id: a2f52d042c5d43bcbb59c6a16249fdb9
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 11.243845267715532
neg_mean_loss: -11.243845267715532
node_ip: 127.0.0.1
pid: 45864
time_since_restore: 0.10467910766601562
time_this_iter_s: 0.10467910766601562
time_total_s: 0.10467910766601562
timestamp: 1658499521
timesteps_since_restore: 0
training_iteration: 1
trial_id: 2e3cd63e
warmup_time: 0.002730131149291992
Result for objective_28587494:
date: 2022-07-22_15-18-41
done: false
experiment_id: 750bd707fda147f0bd615c32243d64e5
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: -6.928259075209273
neg_mean_loss: 6.928259075209273
node_ip: 127.0.0.1
pid: 45842
time_since_restore: 10.179664850234985
time_this_iter_s: 0.10827279090881348
time_total_s: 10.179664850234985
timestamp: 1658499521
timesteps_since_restore: 0
training_iteration: 95
trial_id: '28587494'
warmup_time: 0.0028159618377685547
Result for objective_28682bfa:
date: 2022-07-22_15-18-41
done: false
experiment_id: 5487e6a1cf70423096cf6285197fb824
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: -8.437898356861577
neg_mean_loss: 8.437898356861577
node_ip: 127.0.0.1
pid: 45845
time_since_restore: 10.208577156066895
time_this_iter_s: 0.10761713981628418
time_total_s: 10.208577156066895
timestamp: 1658499521
timesteps_since_restore: 0
training_iteration: 95
trial_id: 28682bfa
warmup_time: 0.003072023391723633
Result for objective_28788388:
date: 2022-07-22_15-18-41
done: false
experiment_id: e959dae493a240c489a3193352c46f3a
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 10.144880256980718
neg_mean_loss: -10.144880256980718
node_ip: 127.0.0.1
pid: 45848
time_since_restore: 10.215306997299194
time_this_iter_s: 0.10723686218261719
time_total_s: 10.215306997299194
timestamp: 1658499521
timesteps_since_restore: 0
training_iteration: 95
trial_id: '28788388'
warmup_time: 0.0027070045471191406
Result for objective_28587494:
date: 2022-07-22_15-18-42
done: true
experiment_id: 750bd707fda147f0bd615c32243d64e5
experiment_tag: 6_activation=relu_tanh,height=-82.3320,steps=100,width=3.3822
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -6.943213699003365
neg_mean_loss: 6.943213699003365
node_ip: 127.0.0.1
pid: 45842
time_since_restore: 10.711625099182129
time_this_iter_s: 0.10703706741333008
time_total_s: 10.711625099182129
timestamp: 1658499522
timesteps_since_restore: 0
training_iteration: 100
trial_id: '28587494'
warmup_time: 0.0028159618377685547
Result for objective_28682bfa:
date: 2022-07-22_15-18-42
done: true
experiment_id: 5487e6a1cf70423096cf6285197fb824
experiment_tag: 7_activation=relu_tanh,height=-94.9771,steps=100,width=17.6810
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -8.440901810803183
neg_mean_loss: 8.440901810803183
node_ip: 127.0.0.1
pid: 45845
time_since_restore: 10.747072219848633
time_this_iter_s: 0.1084749698638916
time_total_s: 10.747072219848633
timestamp: 1658499522
timesteps_since_restore: 0
training_iteration: 100
trial_id: 28682bfa
warmup_time: 0.003072023391723633
Result for objective_28788388:
date: 2022-07-22_15-18-42
done: true
experiment_id: e959dae493a240c489a3193352c46f3a
experiment_tag: 8_activation=relu_tanh,height=90.6787,steps=100,width=13.7072
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 10.141019147716873
neg_mean_loss: -10.141019147716873
node_ip: 127.0.0.1
pid: 45848
time_since_restore: 10.752525091171265
time_this_iter_s: 0.10643196105957031
time_total_s: 10.752525091171265
timestamp: 1658499522
timesteps_since_restore: 0
training_iteration: 100
trial_id: '28788388'
warmup_time: 0.0027070045471191406
Result for objective_3046455a:
date: 2022-07-22_15-18-44
done: false
experiment_id: f282080699d848f69c4b8626bd4c2d91
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 13.250521336587763
neg_mean_loss: -13.250521336587763
node_ip: 127.0.0.1
pid: 45869
time_since_restore: 0.10329103469848633
time_this_iter_s: 0.10329103469848633
time_total_s: 0.10329103469848633
timestamp: 1658499524
timesteps_since_restore: 0
training_iteration: 1
trial_id: 3046455a
warmup_time: 0.0027010440826416016
Result for objective_2e3cd63e:
date: 2022-07-22_15-18-46
done: false
experiment_id: a2f52d042c5d43bcbb59c6a16249fdb9
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 8.5370623175226
neg_mean_loss: -8.5370623175226
node_ip: 127.0.0.1
pid: 45864
time_since_restore: 5.129793882369995
time_this_iter_s: 0.10869097709655762
time_total_s: 5.129793882369995
timestamp: 1658499526
timesteps_since_restore: 0
training_iteration: 48
trial_id: 2e3cd63e
warmup_time: 0.002730131149291992
Result for objective_3046455a:
date: 2022-07-22_15-18-49
done: false
experiment_id: f282080699d848f69c4b8626bd4c2d91
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 3.3797430580474837
neg_mean_loss: -3.3797430580474837
node_ip: 127.0.0.1
pid: 45869
time_since_restore: 5.139970064163208
time_this_iter_s: 0.10870695114135742
time_total_s: 5.139970064163208
timestamp: 1658499529
timesteps_since_restore: 0
training_iteration: 48
trial_id: 3046455a
warmup_time: 0.0027010440826416016
Result for objective_2e3cd63e:
date: 2022-07-22_15-18-51
done: false
experiment_id: a2f52d042c5d43bcbb59c6a16249fdb9
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 6.983470378488619
neg_mean_loss: -6.983470378488619
node_ip: 127.0.0.1
pid: 45864
time_since_restore: 10.16285490989685
time_this_iter_s: 0.1078939437866211
time_total_s: 10.16285490989685
timestamp: 1658499531
timesteps_since_restore: 0
training_iteration: 95
trial_id: 2e3cd63e
warmup_time: 0.002730131149291992
Result for objective_2e3cd63e:
date: 2022-07-22_15-18-51
done: true
experiment_id: a2f52d042c5d43bcbb59c6a16249fdb9
experiment_tag: 9_activation=relu_tanh,height=2.4385,steps=100,width=0.0790
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 6.8562837197212945
neg_mean_loss: -6.8562837197212945
node_ip: 127.0.0.1
pid: 45864
time_since_restore: 10.700610876083374
time_this_iter_s: 0.10786271095275879
time_total_s: 10.700610876083374
timestamp: 1658499531
timesteps_since_restore: 0
training_iteration: 100
trial_id: 2e3cd63e
warmup_time: 0.002730131149291992
Result for objective_3046455a:
date: 2022-07-22_15-18-54
done: false
experiment_id: f282080699d848f69c4b8626bd4c2d91
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 3.315552368411653
neg_mean_loss: -3.315552368411653
node_ip: 127.0.0.1
pid: 45869
time_since_restore: 10.18196415901184
time_this_iter_s: 0.10727214813232422
time_total_s: 10.18196415901184
timestamp: 1658499534
timesteps_since_restore: 0
training_iteration: 95
trial_id: 3046455a
warmup_time: 0.0027010440826416016
Result for objective_3046455a:
date: 2022-07-22_15-18-55
done: true
experiment_id: f282080699d848f69c4b8626bd4c2d91
experiment_tag: 10_activation=relu_tanh,height=22.5052,steps=100,width=16.2524
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 3.3122882595656677
neg_mean_loss: -3.3122882595656677
node_ip: 127.0.0.1
pid: 45869
time_since_restore: 10.717580080032349
time_this_iter_s: 0.1054232120513916
time_total_s: 10.717580080032349
timestamp: 1658499535
timesteps_since_restore: 0
training_iteration: 100
trial_id: 3046455a
warmup_time: 0.0027010440826416016
Here are the hyperparamters found to minimize the mean loss of the defined objective.
print("Best hyperparameters found were: ", results.get_best_result().config)
Best hyperparameters found were: {'steps': 100, 'width': 15.42641286533492, 'height': -95.8496101281197, 'activation': 'relu, tanh'}
Incorporating a time budget to the experiment#
Define the time budget in seconds:
time_budget_s = 30
Similarly we define a search space, but this time we feed it as an argument to
BlendSearch rather than Tuner()’s param_space argument.
We next define the time budget via set_search_properties.
And once again include the ConcurrencyLimiter.
algo = BlendSearch(
metric="mean_loss",
mode="min",
space={
"width": tune.uniform(0, 20),
"height": tune.uniform(-100, 100),
"activation": tune.choice(["relu", "tanh"]),
},
)
algo.set_search_properties(config={"time_budget_s": time_budget_s})
algo = ConcurrencyLimiter(algo, max_concurrent=4)
You passed a `space` parameter to OptunaSearch that contained unresolved search space definitions. OptunaSearch should however be instantiated with fully configured search spaces only. To use Ray Tune's automatic search space conversion, pass the space definition as part of the `param_space` argument to `Tuner()` instead.
[I 2022-07-22 15:18:55,531] A new study created in memory with name: optuna
Now we run the experiment, this time with the time_budget included as an argument.
Note: We allow for virtually infinite num_samples by passing -1, so that the
experiment is stopped according to the time budget rather than a sample limit.
tuner = tune.Tuner(
objective,
tune_config=tune.TuneConfig(
metric="mean_loss",
mode="min",
search_alg=algo,
num_samples=-1,
time_budget_s=time_budget_s,
),
param_space={"steps": 100},
)
results = tuner.fit()
== Status ==
Current time: 2022-07-22 15:19:27 (running for 00:00:32.42)
Memory usage on this node: 10.2/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/4.69 GiB heap, 0.0/2.0 GiB objects
Current best trial: 421d81ee with mean_loss=-8.596112689018389 and parameters={'steps': 100, 'width': 13.171830039895717, 'height': -96.72215542618497, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/blendsearch_exp
Number of trials: 12/infinite (12 TERMINATED)
Current time: 2022-07-22 15:19:27 (running for 00:00:32.42)
Memory usage on this node: 10.2/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/4.69 GiB heap, 0.0/2.0 GiB objects
Current best trial: 421d81ee with mean_loss=-8.596112689018389 and parameters={'steps': 100, 'width': 13.171830039895717, 'height': -96.72215542618497, 'activation': 'tanh'}
Result logdir: /Users/kai/ray_results/blendsearch_exp
Number of trials: 12/infinite (12 TERMINATED)
| Trial name | status | loc | activation | height | steps | width | loss | iter | total time (s) | iterations | neg_mean_loss |
|---|---|---|---|---|---|---|---|---|---|---|---|
| objective_38449df6 | TERMINATED | 127.0.0.1:45883 | relu | 29.4532 | 100 | 1.94864 | 13.4381 | 100 | 10.6555 | 99 | -13.4381 |
| objective_39d44fea | TERMINATED | 127.0.0.1:45889 | tanh | -95.8496 | 100 | 15.4264 | -8.51991 | 100 | 11.2945 | 99 | 8.51991 |
| objective_39d5c046 | TERMINATED | 127.0.0.1:45890 | tanh | -55.0407 | 100 | 9.97014 | -4.40377 | 100 | 11.169 | 99 | 4.40377 |
| objective_39d71608 | TERMINATED | 127.0.0.1:45891 | tanh | -82.332 | 100 | 3.38222 | -6.94321 | 100 | 11.3021 | 99 | 6.94321 |
| objective_402f9534 | TERMINATED | 127.0.0.1:45909 | relu | 2.43845 | 100 | 0.0789653 | 15.8563 | 100 | 10.7275 | 99 | -15.8563 |
| objective_4208c6aa | TERMINATED | 127.0.0.1:45914 | relu | -41.6248 | 100 | 14.4351 | 5.90701 | 100 | 10.811 | 99 | -5.90701 |
| objective_421aeb64 | TERMINATED | 127.0.0.1:45917 | relu | -94.9771 | 100 | 17.681 | 0.559098 | 100 | 10.8502 | 99 | -0.559098 |
| objective_421d81ee | TERMINATED | 127.0.0.1:45918 | tanh | -96.7222 | 100 | 13.1718 | -8.59611 | 100 | 10.8456 | 99 | 8.59611 |
| objective_481b0de6 | TERMINATED | 127.0.0.1:45933 | relu | -81.3401 | 100 | 15.2514 | 2.48132 | 11 | 1.17882 | 10 | -2.48132 |
| objective_4a00fc10 | TERMINATED | 127.0.0.1:45940 | tanh | -100 | 100 | 11.0922 | |||||
| objective_4a167946 | TERMINATED | 127.0.0.1:45945 | tanh | -71.566 | 100 | 10.8509 | |||||
| objective_4a1908a0 | TERMINATED | 127.0.0.1:45946 | relu | -100 | 100 | 12.4575 |
Result for objective_38449df6:
date: 2022-07-22_15-18-58
done: false
experiment_id: 22ed7d2e0f154e3d89fddc8f7f0b743f
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 22.945321626643842
neg_mean_loss: -22.945321626643842
node_ip: 127.0.0.1
pid: 45883
time_since_restore: 0.1004023551940918
time_this_iter_s: 0.1004023551940918
time_total_s: 0.1004023551940918
timestamp: 1658499538
timesteps_since_restore: 0
training_iteration: 1
trial_id: 38449df6
warmup_time: 0.0025548934936523438
Result for objective_39d5c046:
date: 2022-07-22_15-19-00
done: false
experiment_id: faa00d9b1a2b4014bf01bab455bf9dbc
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 5.495932910616953
neg_mean_loss: -5.495932910616953
node_ip: 127.0.0.1
pid: 45890
time_since_restore: 0.10018515586853027
time_this_iter_s: 0.10018515586853027
time_total_s: 0.10018515586853027
timestamp: 1658499540
timesteps_since_restore: 0
training_iteration: 1
trial_id: 39d5c046
warmup_time: 0.0029740333557128906
Result for objective_39d71608:
date: 2022-07-22_15-19-00
done: false
experiment_id: 43ea2d3839fb42cabf83009ddd82acbd
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 2.766796283480206
neg_mean_loss: -2.766796283480206
node_ip: 127.0.0.1
pid: 45891
time_since_restore: 0.10022497177124023
time_this_iter_s: 0.10022497177124023
time_total_s: 0.10022497177124023
timestamp: 1658499540
timesteps_since_restore: 0
training_iteration: 1
trial_id: 39d71608
warmup_time: 0.003571033477783203
Result for objective_39d44fea:
date: 2022-07-22_15-19-00
done: false
experiment_id: d45c2138630140c3915a785b0e544a10
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 1.4150389871880282
neg_mean_loss: -1.4150389871880282
node_ip: 127.0.0.1
pid: 45889
time_since_restore: 0.10021686553955078
time_this_iter_s: 0.10021686553955078
time_total_s: 0.10021686553955078
timestamp: 1658499540
timesteps_since_restore: 0
training_iteration: 1
trial_id: 39d44fea
warmup_time: 0.003281831741333008
Result for objective_38449df6:
date: 2022-07-22_15-19-03
done: false
experiment_id: 22ed7d2e0f154e3d89fddc8f7f0b743f
hostname: Kais-MacBook-Pro.local
iterations: 48
iterations_since_restore: 49
mean_loss: 13.91118054261762
neg_mean_loss: -13.91118054261762
node_ip: 127.0.0.1
pid: 45883
time_since_restore: 5.199498176574707
time_this_iter_s: 0.10724401473999023
time_total_s: 5.199498176574707
timestamp: 1658499543
timesteps_since_restore: 0
training_iteration: 49
trial_id: 38449df6
warmup_time: 0.0025548934936523438
Result for objective_39d5c046:
date: 2022-07-22_15-19-05
done: false
experiment_id: faa00d9b1a2b4014bf01bab455bf9dbc
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -4.295122851662673
neg_mean_loss: 4.295122851662673
node_ip: 127.0.0.1
pid: 45890
time_since_restore: 5.137373924255371
time_this_iter_s: 0.1063838005065918
time_total_s: 5.137373924255371
timestamp: 1658499545
timesteps_since_restore: 0
training_iteration: 48
trial_id: 39d5c046
warmup_time: 0.0029740333557128906
Result for objective_39d44fea:
date: 2022-07-22_15-19-05
done: false
experiment_id: d45c2138630140c3915a785b0e544a10
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -8.44891425494968
neg_mean_loss: 8.44891425494968
node_ip: 127.0.0.1
pid: 45889
time_since_restore: 5.140375852584839
time_this_iter_s: 0.10918521881103516
time_total_s: 5.140375852584839
timestamp: 1658499545
timesteps_since_restore: 0
training_iteration: 48
trial_id: 39d44fea
warmup_time: 0.003281831741333008
Result for objective_39d71608:
date: 2022-07-22_15-19-05
done: false
experiment_id: 43ea2d3839fb42cabf83009ddd82acbd
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -6.641362320132167
neg_mean_loss: 6.641362320132167
node_ip: 127.0.0.1
pid: 45891
time_since_restore: 5.144212007522583
time_this_iter_s: 0.1064291000366211
time_total_s: 5.144212007522583
timestamp: 1658499545
timesteps_since_restore: 0
training_iteration: 48
trial_id: 39d71608
warmup_time: 0.003571033477783203
Result for objective_38449df6:
date: 2022-07-22_15-19-08
done: false
experiment_id: 22ed7d2e0f154e3d89fddc8f7f0b743f
hostname: Kais-MacBook-Pro.local
iterations: 95
iterations_since_restore: 96
mean_loss: 13.457824286707531
neg_mean_loss: -13.457824286707531
node_ip: 127.0.0.1
pid: 45883
time_since_restore: 10.220572233200073
time_this_iter_s: 0.10792016983032227
time_total_s: 10.220572233200073
timestamp: 1658499548
timesteps_since_restore: 0
training_iteration: 96
trial_id: 38449df6
warmup_time: 0.0025548934936523438
Result for objective_38449df6:
date: 2022-07-22_15-19-08
done: true
experiment_id: 22ed7d2e0f154e3d89fddc8f7f0b743f
experiment_tag: 1_activation=relu,height=29.4532,steps=100,width=1.9486
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 13.43813759132241
neg_mean_loss: -13.43813759132241
node_ip: 127.0.0.1
pid: 45883
time_since_restore: 10.655535221099854
time_this_iter_s: 0.10872602462768555
time_total_s: 10.655535221099854
timestamp: 1658499548
timesteps_since_restore: 0
training_iteration: 100
trial_id: 38449df6
warmup_time: 0.0025548934936523438
Result for objective_39d5c046:
date: 2022-07-22_15-19-10
done: false
experiment_id: faa00d9b1a2b4014bf01bab455bf9dbc
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: -4.398492005249027
neg_mean_loss: 4.398492005249027
node_ip: 127.0.0.1
pid: 45890
time_since_restore: 10.1424720287323
time_this_iter_s: 0.10685110092163086
time_total_s: 10.1424720287323
timestamp: 1658499550
timesteps_since_restore: 0
training_iteration: 95
trial_id: 39d5c046
warmup_time: 0.0029740333557128906
Result for objective_39d44fea:
date: 2022-07-22_15-19-10
done: false
experiment_id: d45c2138630140c3915a785b0e544a10
hostname: Kais-MacBook-Pro.local
iterations: 93
iterations_since_restore: 94
mean_loss: -8.515740400738455
neg_mean_loss: 8.515740400738455
node_ip: 127.0.0.1
pid: 45889
time_since_restore: 10.051257848739624
time_this_iter_s: 0.10633492469787598
time_total_s: 10.051257848739624
timestamp: 1658499550
timesteps_since_restore: 0
training_iteration: 94
trial_id: 39d44fea
warmup_time: 0.003281831741333008
Result for objective_39d71608:
date: 2022-07-22_15-19-10
done: false
experiment_id: 43ea2d3839fb42cabf83009ddd82acbd
hostname: Kais-MacBook-Pro.local
iterations: 93
iterations_since_restore: 94
mean_loss: -6.925081133185133
neg_mean_loss: 6.925081133185133
node_ip: 127.0.0.1
pid: 45891
time_since_restore: 10.069098949432373
time_this_iter_s: 0.10451602935791016
time_total_s: 10.069098949432373
timestamp: 1658499550
timesteps_since_restore: 0
training_iteration: 94
trial_id: 39d71608
warmup_time: 0.003571033477783203
Result for objective_402f9534:
date: 2022-07-22_15-19-11
done: false
experiment_id: 66af4092dc134e049ed47a56532133ce
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 20.243845267715532
neg_mean_loss: -20.243845267715532
node_ip: 127.0.0.1
pid: 45909
time_since_restore: 0.10260891914367676
time_this_iter_s: 0.10260891914367676
time_total_s: 0.10260891914367676
timestamp: 1658499551
timesteps_since_restore: 0
training_iteration: 1
trial_id: 402f9534
warmup_time: 0.0026721954345703125
Result for objective_39d5c046:
date: 2022-07-22_15-19-11
done: true
experiment_id: faa00d9b1a2b4014bf01bab455bf9dbc
experiment_tag: 3_activation=tanh,height=-55.0407,steps=100,width=9.9701
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -4.403770601366095
neg_mean_loss: 4.403770601366095
node_ip: 127.0.0.1
pid: 45890
time_since_restore: 11.169048070907593
time_this_iter_s: 0.1081700325012207
time_total_s: 11.169048070907593
timestamp: 1658499551
timesteps_since_restore: 0
training_iteration: 100
trial_id: 39d5c046
warmup_time: 0.0029740333557128906
Result for objective_39d44fea:
date: 2022-07-22_15-19-12
done: true
experiment_id: d45c2138630140c3915a785b0e544a10
experiment_tag: 2_activation=tanh,height=-95.8496,steps=100,width=15.4264
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -8.519908298508359
neg_mean_loss: 8.519908298508359
node_ip: 127.0.0.1
pid: 45889
time_since_restore: 11.29450273513794
time_this_iter_s: 0.1167898178100586
time_total_s: 11.29450273513794
timestamp: 1658499552
timesteps_since_restore: 0
training_iteration: 100
trial_id: 39d44fea
warmup_time: 0.003281831741333008
Result for objective_39d71608:
date: 2022-07-22_15-19-12
done: true
experiment_id: 43ea2d3839fb42cabf83009ddd82acbd
experiment_tag: 4_activation=tanh,height=-82.3320,steps=100,width=3.3822
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -6.943213699003365
neg_mean_loss: 6.943213699003365
node_ip: 127.0.0.1
pid: 45891
time_since_restore: 11.302148818969727
time_this_iter_s: 0.12157797813415527
time_total_s: 11.302148818969727
timestamp: 1658499552
timesteps_since_restore: 0
training_iteration: 100
trial_id: 39d71608
warmup_time: 0.003571033477783203
Result for objective_4208c6aa:
date: 2022-07-22_15-19-14
done: false
experiment_id: ce292c59f7c74e1ab42de6004eb3d846
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 15.837521363412662
neg_mean_loss: -15.837521363412662
node_ip: 127.0.0.1
pid: 45914
time_since_restore: 0.10442900657653809
time_this_iter_s: 0.10442900657653809
time_total_s: 0.10442900657653809
timestamp: 1658499554
timesteps_since_restore: 0
training_iteration: 1
trial_id: 4208c6aa
warmup_time: 0.002707958221435547
Result for objective_421aeb64:
date: 2022-07-22_15-19-14
done: false
experiment_id: a4b264c05660423ea645cef7a9cd38bb
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 10.502293516994555
neg_mean_loss: -10.502293516994555
node_ip: 127.0.0.1
pid: 45917
time_since_restore: 0.1030268669128418
time_this_iter_s: 0.1030268669128418
time_total_s: 0.1030268669128418
timestamp: 1658499554
timesteps_since_restore: 0
training_iteration: 1
trial_id: 421aeb64
warmup_time: 0.0031020641326904297
Result for objective_421d81ee:
date: 2022-07-22_15-19-14
done: false
experiment_id: 836a62c82d6c467fb232414586872fd0
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 1.3277844573815027
neg_mean_loss: -1.3277844573815027
node_ip: 127.0.0.1
pid: 45918
time_since_restore: 0.10497617721557617
time_this_iter_s: 0.10497617721557617
time_total_s: 0.10497617721557617
timestamp: 1658499554
timesteps_since_restore: 0
training_iteration: 1
trial_id: 421d81ee
warmup_time: 0.002989053726196289
Result for objective_402f9534:
date: 2022-07-22_15-19-16
done: false
experiment_id: 66af4092dc134e049ed47a56532133ce
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 17.5370623175226
neg_mean_loss: -17.5370623175226
node_ip: 127.0.0.1
pid: 45909
time_since_restore: 5.13571572303772
time_this_iter_s: 0.1079568862915039
time_total_s: 5.13571572303772
timestamp: 1658499556
timesteps_since_restore: 0
training_iteration: 48
trial_id: 402f9534
warmup_time: 0.0026721954345703125
Result for objective_4208c6aa:
date: 2022-07-22_15-19-19
done: false
experiment_id: ce292c59f7c74e1ab42de6004eb3d846
hostname: Kais-MacBook-Pro.local
iterations: 46
iterations_since_restore: 47
mean_loss: 5.98588603781584
neg_mean_loss: -5.98588603781584
node_ip: 127.0.0.1
pid: 45914
time_since_restore: 5.11784291267395
time_this_iter_s: 0.106842041015625
time_total_s: 5.11784291267395
timestamp: 1658499559
timesteps_since_restore: 0
training_iteration: 47
trial_id: 4208c6aa
warmup_time: 0.002707958221435547
Result for objective_421aeb64:
date: 2022-07-22_15-19-19
done: false
experiment_id: a4b264c05660423ea645cef7a9cd38bb
hostname: Kais-MacBook-Pro.local
iterations: 46
iterations_since_restore: 47
mean_loss: 0.6237521179327707
neg_mean_loss: -0.6237521179327707
node_ip: 127.0.0.1
pid: 45917
time_since_restore: 5.119860887527466
time_this_iter_s: 0.10904288291931152
time_total_s: 5.119860887527466
timestamp: 1658499559
timesteps_since_restore: 0
training_iteration: 47
trial_id: 421aeb64
warmup_time: 0.0031020641326904297
Result for objective_421d81ee:
date: 2022-07-22_15-19-19
done: false
experiment_id: 836a62c82d6c467fb232414586872fd0
hostname: Kais-MacBook-Pro.local
iterations: 46
iterations_since_restore: 47
mean_loss: -8.509852624896407
neg_mean_loss: 8.509852624896407
node_ip: 127.0.0.1
pid: 45918
time_since_restore: 5.103626012802124
time_this_iter_s: 0.10568881034851074
time_total_s: 5.103626012802124
timestamp: 1658499559
timesteps_since_restore: 0
training_iteration: 47
trial_id: 421d81ee
warmup_time: 0.002989053726196289
Result for objective_402f9534:
date: 2022-07-22_15-19-21
done: false
experiment_id: 66af4092dc134e049ed47a56532133ce
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 15.983470378488619
neg_mean_loss: -15.983470378488619
node_ip: 127.0.0.1
pid: 45909
time_since_restore: 10.194732666015625
time_this_iter_s: 0.11167168617248535
time_total_s: 10.194732666015625
timestamp: 1658499561
timesteps_since_restore: 0
training_iteration: 95
trial_id: 402f9534
warmup_time: 0.0026721954345703125
Result for objective_402f9534:
date: 2022-07-22_15-19-22
done: true
experiment_id: 66af4092dc134e049ed47a56532133ce
experiment_tag: 5_activation=relu,height=2.4385,steps=100,width=0.0790
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 15.856283719721294
neg_mean_loss: -15.856283719721294
node_ip: 127.0.0.1
pid: 45909
time_since_restore: 10.727468013763428
time_this_iter_s: 0.10604405403137207
time_total_s: 10.727468013763428
timestamp: 1658499562
timesteps_since_restore: 0
training_iteration: 100
trial_id: 402f9534
warmup_time: 0.0026721954345703125
Result for objective_4208c6aa:
date: 2022-07-22_15-19-24
done: false
experiment_id: ce292c59f7c74e1ab42de6004eb3d846
hostname: Kais-MacBook-Pro.local
iterations: 93
iterations_since_restore: 94
mean_loss: 5.911460436218253
neg_mean_loss: -5.911460436218253
node_ip: 127.0.0.1
pid: 45914
time_since_restore: 10.168487071990967
time_this_iter_s: 0.10619020462036133
time_total_s: 10.168487071990967
timestamp: 1658499564
timesteps_since_restore: 0
training_iteration: 94
trial_id: 4208c6aa
warmup_time: 0.002707958221435547
Result for objective_481b0de6:
date: 2022-07-22_15-19-24
done: false
experiment_id: ceafc8f530054af6bc676fd081f84c46
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 11.865986796947288
neg_mean_loss: -11.865986796947288
node_ip: 127.0.0.1
pid: 45933
time_since_restore: 0.10329604148864746
time_this_iter_s: 0.10329604148864746
time_total_s: 0.10329604148864746
timestamp: 1658499564
timesteps_since_restore: 0
training_iteration: 1
trial_id: 481b0de6
warmup_time: 0.0028421878814697266
Result for objective_421aeb64:
date: 2022-07-22_15-19-24
done: false
experiment_id: a4b264c05660423ea645cef7a9cd38bb
hostname: Kais-MacBook-Pro.local
iterations: 93
iterations_since_restore: 94
mean_loss: 0.5627408539120644
neg_mean_loss: -0.5627408539120644
node_ip: 127.0.0.1
pid: 45917
time_since_restore: 10.207238912582397
time_this_iter_s: 0.1081080436706543
time_total_s: 10.207238912582397
timestamp: 1658499564
timesteps_since_restore: 0
training_iteration: 94
trial_id: 421aeb64
warmup_time: 0.0031020641326904297
Result for objective_421d81ee:
date: 2022-07-22_15-19-24
done: false
experiment_id: 836a62c82d6c467fb232414586872fd0
hostname: Kais-MacBook-Pro.local
iterations: 93
iterations_since_restore: 94
mean_loss: -8.591242584097142
neg_mean_loss: 8.591242584097142
node_ip: 127.0.0.1
pid: 45918
time_since_restore: 10.2047119140625
time_this_iter_s: 0.1085958480834961
time_total_s: 10.2047119140625
timestamp: 1658499564
timesteps_since_restore: 0
training_iteration: 94
trial_id: 421d81ee
warmup_time: 0.002989053726196289
Result for objective_4208c6aa:
date: 2022-07-22_15-19-25
done: true
experiment_id: ce292c59f7c74e1ab42de6004eb3d846
experiment_tag: 6_activation=relu,height=-41.6248,steps=100,width=14.4351
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 5.907010419420164
neg_mean_loss: -5.907010419420164
node_ip: 127.0.0.1
pid: 45914
time_since_restore: 10.811041831970215
time_this_iter_s: 0.10398101806640625
time_total_s: 10.811041831970215
timestamp: 1658499565
timesteps_since_restore: 0
training_iteration: 100
trial_id: 4208c6aa
warmup_time: 0.002707958221435547
Result for objective_421aeb64:
date: 2022-07-22_15-19-25
done: true
experiment_id: a4b264c05660423ea645cef7a9cd38bb
experiment_tag: 7_activation=relu,height=-94.9771,steps=100,width=17.6810
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 0.559098189196817
neg_mean_loss: -0.559098189196817
node_ip: 127.0.0.1
pid: 45917
time_since_restore: 10.850210905075073
time_this_iter_s: 0.10677385330200195
time_total_s: 10.850210905075073
timestamp: 1658499565
timesteps_since_restore: 0
training_iteration: 100
trial_id: 421aeb64
warmup_time: 0.0031020641326904297
Result for objective_421d81ee:
date: 2022-07-22_15-19-25
done: true
experiment_id: 836a62c82d6c467fb232414586872fd0
experiment_tag: 8_activation=tanh,height=-96.7222,steps=100,width=13.1718
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -8.596112689018389
neg_mean_loss: 8.596112689018389
node_ip: 127.0.0.1
pid: 45918
time_since_restore: 10.845621109008789
time_this_iter_s: 0.10427713394165039
time_total_s: 10.845621109008789
timestamp: 1658499565
timesteps_since_restore: 0
training_iteration: 100
trial_id: 421d81ee
warmup_time: 0.002989053726196289
print("Best hyperparameters found were: ", results.get_best_result().config)
Best hyperparameters found were: {'steps': 100, 'width': 13.171830039895717, 'height': -96.72215542618497, 'activation': 'tanh'}
Running Tune experiments with CFO#
This example demonstrates the usage of Frugal Optimization for Cost-related Hyperparameters (CFO) with Ray Tune.
We now define the search algorithm as built from CFO, constrained to a maximum of 4
concurrent trials with a ConcurrencyLimiter.
algo = CFO()
algo = ConcurrencyLimiter(algo, max_concurrent=4)
The number of samples is the number of hyperparameter combinations that will be
tried out. This Tune run is set to 1000 samples.
(you can decrease this if it takes too long on your machine).
num_samples = 1000
Next we define a search space. The critical assumption is that the optimal hyperparameters live within this space. Yet, if the space is very large, then those hyperparameters may be difficult to find in a short amount of time.
search_config = {
"steps": 100,
"width": tune.uniform(0, 20),
"height": tune.uniform(-100, 100),
"activation": tune.choice(["relu, tanh"])
}
Finally, we run the experiment to "min"imize the “mean_loss” of the objective
by searching search_config via algo, num_samples times. This previous sentence
is fully characterizes the search problem we aim to solve. With this in mind,
notice how efficient it is to execute tuner.fit().
tuner = tune.Tuner(
objective,
tune_config=tune.TuneConfig(
metric="mean_loss",
mode="min",
search_alg=algo,
num_samples=num_samples,
),
param_space=search_config,
)
results = tuner.fit()
== Status ==
Current time: 2022-07-22 15:20:11 (running for 00:00:43.01)
Memory usage on this node: 10.0/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/4.69 GiB heap, 0.0/2.0 GiB objects
Current best trial: 4d51b38c with mean_loss=-8.661424748129757 and parameters={'steps': 100, 'width': 15.124213652112319, 'height': -97.27768667042203, 'activation': 'relu, tanh'}
Result logdir: /Users/kai/ray_results/cfo_exp
Number of trials: 10/10 (10 TERMINATED)
Current time: 2022-07-22 15:20:11 (running for 00:00:43.01)
Memory usage on this node: 10.0/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/4.69 GiB heap, 0.0/2.0 GiB objects
Current best trial: 4d51b38c with mean_loss=-8.661424748129757 and parameters={'steps': 100, 'width': 15.124213652112319, 'height': -97.27768667042203, 'activation': 'relu, tanh'}
Result logdir: /Users/kai/ray_results/cfo_exp
Number of trials: 10/10 (10 TERMINATED)
| Trial name | status | loc | activation | height | steps | width | loss | iter | total time (s) | iterations | neg_mean_loss |
|---|---|---|---|---|---|---|---|---|---|---|---|
| objective_4bc281fe | TERMINATED | 127.0.0.1:45958 | relu, tanh | 29.4532 | 100 | 1.94864 | 4.43814 | 100 | 10.7193 | 99 | -4.43814 |
| objective_4d4ed536 | TERMINATED | 127.0.0.1:45963 | relu, tanh | -26.2681 | 100 | 7.25618 | -1.48952 | 100 | 11.1932 | 99 | 1.48952 |
| objective_4d5064aa | TERMINATED | 127.0.0.1:45964 | relu, tanh | -46.7215 | 100 | 15.1097 | -3.60574 | 100 | 11.1983 | 99 | 3.60574 |
| objective_4d51b38c | TERMINATED | 127.0.0.1:45965 | relu, tanh | -97.2777 | 100 | 15.1242 | -8.66142 | 100 | 11.1977 | 99 | 8.66142 |
| objective_53b39e84 | TERMINATED | 127.0.0.1:45983 | relu, tanh | 25.7812 | 100 | 1.77704 | 4.11597 | 100 | 10.7964 | 99 | -4.11597 |
| objective_558c6132 | TERMINATED | 127.0.0.1:45990 | relu, tanh | 33.1252 | 100 | 2.12024 | 4.76727 | 100 | 10.7659 | 99 | -4.76727 |
| objective_558f27aa | TERMINATED | 127.0.0.1:45991 | relu, tanh | 40.929 | 100 | 0.0138032 | 13.8907 | 100 | 10.785 | 99 | -13.8907 |
| objective_5592ca68 | TERMINATED | 127.0.0.1:45992 | relu, tanh | 17.9775 | 100 | 3.88348 | 3.05125 | 100 | 10.7802 | 99 | -3.05125 |
| objective_5b999414 | TERMINATED | 127.0.0.1:46011 | relu, tanh | 11.0354 | 100 | 3.27423 | 2.40281 | 100 | 12.6262 | 99 | -2.40281 |
| objective_5d8462c2 | TERMINATED | 127.0.0.1:46018 | relu, tanh | 24.9195 | 100 | 4.49273 | 3.71184 | 100 | 10.7339 | 99 | -3.71184 |
Result for objective_4bc281fe:
date: 2022-07-22_15-19-30
done: false
experiment_id: 129b24a058ee40fd991fb40844988057
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 13.945321626643842
neg_mean_loss: -13.945321626643842
node_ip: 127.0.0.1
pid: 45958
time_since_restore: 0.10440444946289062
time_this_iter_s: 0.10440444946289062
time_total_s: 0.10440444946289062
timestamp: 1658499570
timesteps_since_restore: 0
training_iteration: 1
trial_id: 4bc281fe
warmup_time: 0.003987789154052734
Result for objective_4d4ed536:
date: 2022-07-22_15-19-33
done: false
experiment_id: 6c18c359d1574792ac99ed37265d5d87
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 8.373189005362395
neg_mean_loss: -8.373189005362395
node_ip: 127.0.0.1
pid: 45963
time_since_restore: 0.10463309288024902
time_this_iter_s: 0.10463309288024902
time_total_s: 0.10463309288024902
timestamp: 1658499573
timesteps_since_restore: 0
training_iteration: 1
trial_id: 4d4ed536
warmup_time: 0.0028281211853027344
Result for objective_4d51b38c:
date: 2022-07-22_15-19-33
done: false
experiment_id: 5c9cf6fc35c4411b8ec13e0b90b791a4
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 1.2722313329577961
neg_mean_loss: -1.2722313329577961
node_ip: 127.0.0.1
pid: 45965
time_since_restore: 0.10346817970275879
time_this_iter_s: 0.10346817970275879
time_total_s: 0.10346817970275879
timestamp: 1658499573
timesteps_since_restore: 0
training_iteration: 1
trial_id: 4d51b38c
warmup_time: 0.0027761459350585938
Result for objective_4d5064aa:
date: 2022-07-22_15-19-33
done: false
experiment_id: 93ed3aa2df8c4f71b0b1e9437c062ccf
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 6.327848408616047
neg_mean_loss: -6.327848408616047
node_ip: 127.0.0.1
pid: 45964
time_since_restore: 0.10395598411560059
time_this_iter_s: 0.10395598411560059
time_total_s: 0.10395598411560059
timestamp: 1658499573
timesteps_since_restore: 0
training_iteration: 1
trial_id: 4d5064aa
warmup_time: 0.002546072006225586
Result for objective_4bc281fe:
date: 2022-07-22_15-19-35
done: false
experiment_id: 129b24a058ee40fd991fb40844988057
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 4.9297078000588614
neg_mean_loss: -4.9297078000588614
node_ip: 127.0.0.1
pid: 45958
time_since_restore: 5.122281312942505
time_this_iter_s: 0.10893416404724121
time_total_s: 5.122281312942505
timestamp: 1658499575
timesteps_since_restore: 0
training_iteration: 48
trial_id: 4bc281fe
warmup_time: 0.003987789154052734
Result for objective_4d4ed536:
date: 2022-07-22_15-19-38
done: false
experiment_id: 6c18c359d1574792ac99ed37265d5d87
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -1.3419437208788234
neg_mean_loss: 1.3419437208788234
node_ip: 127.0.0.1
pid: 45963
time_since_restore: 5.132510185241699
time_this_iter_s: 0.10718894004821777
time_total_s: 5.132510185241699
timestamp: 1658499578
timesteps_since_restore: 0
training_iteration: 48
trial_id: 4d4ed536
warmup_time: 0.0028281211853027344
Result for objective_4d51b38c:
date: 2022-07-22_15-19-38
done: false
experiment_id: 5c9cf6fc35c4411b8ec13e0b90b791a4
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -8.58904124947806
neg_mean_loss: 8.58904124947806
node_ip: 127.0.0.1
pid: 45965
time_since_restore: 5.1296000480651855
time_this_iter_s: 0.10642719268798828
time_total_s: 5.1296000480651855
timestamp: 1658499578
timesteps_since_restore: 0
training_iteration: 48
trial_id: 4d51b38c
warmup_time: 0.0027761459350585938
Result for objective_4d5064aa:
date: 2022-07-22_15-19-38
done: false
experiment_id: 93ed3aa2df8c4f71b0b1e9437c062ccf
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: -3.5332931133029772
neg_mean_loss: 3.5332931133029772
node_ip: 127.0.0.1
pid: 45964
time_since_restore: 5.1527419090271
time_this_iter_s: 0.10794281959533691
time_total_s: 5.1527419090271
timestamp: 1658499578
timesteps_since_restore: 0
training_iteration: 48
trial_id: 4d5064aa
warmup_time: 0.002546072006225586
Result for objective_4bc281fe:
date: 2022-07-22_15-19-41
done: false
experiment_id: 129b24a058ee40fd991fb40844988057
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 4.462994199505408
neg_mean_loss: -4.462994199505408
node_ip: 127.0.0.1
pid: 45958
time_since_restore: 10.180925369262695
time_this_iter_s: 0.10581827163696289
time_total_s: 10.180925369262695
timestamp: 1658499581
timesteps_since_restore: 0
training_iteration: 95
trial_id: 4bc281fe
warmup_time: 0.003987789154052734
Result for objective_4bc281fe:
date: 2022-07-22_15-19-41
done: true
experiment_id: 129b24a058ee40fd991fb40844988057
experiment_tag: 1_activation=relu_tanh,height=29.4532,steps=100,width=1.9486
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 4.438137591322411
neg_mean_loss: -4.438137591322411
node_ip: 127.0.0.1
pid: 45958
time_since_restore: 10.71930718421936
time_this_iter_s: 0.10692596435546875
time_total_s: 10.71930718421936
timestamp: 1658499581
timesteps_since_restore: 0
training_iteration: 100
trial_id: 4bc281fe
warmup_time: 0.003987789154052734
Result for objective_4d4ed536:
date: 2022-07-22_15-19-43
done: false
experiment_id: 6c18c359d1574792ac99ed37265d5d87
hostname: Kais-MacBook-Pro.local
iterations: 93
iterations_since_restore: 94
mean_loss: -1.48078832328251
neg_mean_loss: 1.48078832328251
node_ip: 127.0.0.1
pid: 45963
time_since_restore: 10.033550262451172
time_this_iter_s: 0.10465312004089355
time_total_s: 10.033550262451172
timestamp: 1658499583
timesteps_since_restore: 0
training_iteration: 94
trial_id: 4d4ed536
warmup_time: 0.0028281211853027344
Result for objective_4d51b38c:
date: 2022-07-22_15-19-43
done: false
experiment_id: 5c9cf6fc35c4411b8ec13e0b90b791a4
hostname: Kais-MacBook-Pro.local
iterations: 93
iterations_since_restore: 94
mean_loss: -8.657174711294605
neg_mean_loss: 8.657174711294605
node_ip: 127.0.0.1
pid: 45965
time_since_restore: 10.026285171508789
time_this_iter_s: 0.10726714134216309
time_total_s: 10.026285171508789
timestamp: 1658499583
timesteps_since_restore: 0
training_iteration: 94
trial_id: 4d51b38c
warmup_time: 0.0027761459350585938
Result for objective_4d5064aa:
date: 2022-07-22_15-19-43
done: false
experiment_id: 93ed3aa2df8c4f71b0b1e9437c062ccf
hostname: Kais-MacBook-Pro.local
iterations: 93
iterations_since_restore: 94
mean_loss: -3.601490481891755
neg_mean_loss: 3.601490481891755
node_ip: 127.0.0.1
pid: 45964
time_since_restore: 10.049538135528564
time_this_iter_s: 0.10654830932617188
time_total_s: 10.049538135528564
timestamp: 1658499583
timesteps_since_restore: 0
training_iteration: 94
trial_id: 4d5064aa
warmup_time: 0.002546072006225586
Result for objective_53b39e84:
date: 2022-07-22_15-19-44
done: false
experiment_id: abfeeaaa7eae4ad798c655bed2d1e223
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 13.57812082128205
neg_mean_loss: -13.57812082128205
node_ip: 127.0.0.1
pid: 45983
time_since_restore: 0.10458230972290039
time_this_iter_s: 0.10458230972290039
time_total_s: 0.10458230972290039
timestamp: 1658499584
timesteps_since_restore: 0
training_iteration: 1
trial_id: 53b39e84
warmup_time: 0.002878904342651367
Result for objective_4d4ed536:
date: 2022-07-22_15-19-44
done: true
experiment_id: 6c18c359d1574792ac99ed37265d5d87
experiment_tag: 2_activation=relu_tanh,height=-26.2681,steps=100,width=7.2562
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -1.489516678620475
neg_mean_loss: 1.489516678620475
node_ip: 127.0.0.1
pid: 45963
time_since_restore: 11.193175077438354
time_this_iter_s: 0.11231780052185059
time_total_s: 11.193175077438354
timestamp: 1658499584
timesteps_since_restore: 0
training_iteration: 100
trial_id: 4d4ed536
warmup_time: 0.0028281211853027344
Result for objective_4d51b38c:
date: 2022-07-22_15-19-44
done: true
experiment_id: 5c9cf6fc35c4411b8ec13e0b90b791a4
experiment_tag: 4_activation=relu_tanh,height=-97.2777,steps=100,width=15.1242
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -8.661424748129757
neg_mean_loss: 8.661424748129757
node_ip: 127.0.0.1
pid: 45965
time_since_restore: 11.19774603843689
time_this_iter_s: 0.10802578926086426
time_total_s: 11.19774603843689
timestamp: 1658499584
timesteps_since_restore: 0
training_iteration: 100
trial_id: 4d51b38c
warmup_time: 0.0027761459350585938
Result for objective_4d5064aa:
date: 2022-07-22_15-19-44
done: true
experiment_id: 93ed3aa2df8c4f71b0b1e9437c062ccf
experiment_tag: 3_activation=relu_tanh,height=-46.7215,steps=100,width=15.1097
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: -3.6057445346106167
neg_mean_loss: 3.6057445346106167
node_ip: 127.0.0.1
pid: 45964
time_since_restore: 11.198285102844238
time_this_iter_s: 0.10615801811218262
time_total_s: 11.198285102844238
timestamp: 1658499584
timesteps_since_restore: 0
training_iteration: 100
trial_id: 4d5064aa
warmup_time: 0.002546072006225586
Result for objective_558c6132:
date: 2022-07-22_15-19-47
done: false
experiment_id: 98df2ca60c5f4a33a3da6870c3d8db4c
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 14.31252243200564
neg_mean_loss: -14.31252243200564
node_ip: 127.0.0.1
pid: 45990
time_since_restore: 0.10317206382751465
time_this_iter_s: 0.10317206382751465
time_total_s: 0.10317206382751465
timestamp: 1658499587
timesteps_since_restore: 0
training_iteration: 1
trial_id: 558c6132
warmup_time: 0.003049135208129883
Result for objective_558f27aa:
date: 2022-07-22_15-19-47
done: false
experiment_id: 73def1ffc7f443669328b88695453248
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 15.092898207127158
neg_mean_loss: -15.092898207127158
node_ip: 127.0.0.1
pid: 45991
time_since_restore: 0.10495114326477051
time_this_iter_s: 0.10495114326477051
time_total_s: 0.10495114326477051
timestamp: 1658499587
timesteps_since_restore: 0
training_iteration: 1
trial_id: 558f27aa
warmup_time: 0.0029478073120117188
Result for objective_5592ca68:
date: 2022-07-22_15-19-47
done: false
experiment_id: 1de409fb58a84c3bba522620ec7e1fe4
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 12.79774504616053
neg_mean_loss: -12.79774504616053
node_ip: 127.0.0.1
pid: 45992
time_since_restore: 0.10448837280273438
time_this_iter_s: 0.10448837280273438
time_total_s: 0.10448837280273438
timestamp: 1658499587
timesteps_since_restore: 0
training_iteration: 1
trial_id: 5592ca68
warmup_time: 0.0027618408203125
Result for objective_53b39e84:
date: 2022-07-22_15-19-49
done: false
experiment_id: abfeeaaa7eae4ad798c655bed2d1e223
hostname: Kais-MacBook-Pro.local
iterations: 46
iterations_since_restore: 47
mean_loss: 4.668111815725785
neg_mean_loss: -4.668111815725785
node_ip: 127.0.0.1
pid: 45983
time_since_restore: 5.113715171813965
time_this_iter_s: 0.10550904273986816
time_total_s: 5.113715171813965
timestamp: 1658499589
timesteps_since_restore: 0
training_iteration: 47
trial_id: 53b39e84
warmup_time: 0.002878904342651367
Result for objective_558c6132:
date: 2022-07-22_15-19-52
done: false
experiment_id: 98df2ca60c5f4a33a3da6870c3d8db4c
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 5.2245036670526375
neg_mean_loss: -5.2245036670526375
node_ip: 127.0.0.1
pid: 45990
time_since_restore: 5.1552488803863525
time_this_iter_s: 0.10927176475524902
time_total_s: 5.1552488803863525
timestamp: 1658499592
timesteps_since_restore: 0
training_iteration: 48
trial_id: 558c6132
warmup_time: 0.003049135208129883
Result for objective_558f27aa:
date: 2022-07-22_15-19-52
done: false
experiment_id: 73def1ffc7f443669328b88695453248
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 14.483670237907232
neg_mean_loss: -14.483670237907232
node_ip: 127.0.0.1
pid: 45991
time_since_restore: 5.174212217330933
time_this_iter_s: 0.10701107978820801
time_total_s: 5.174212217330933
timestamp: 1658499592
timesteps_since_restore: 0
training_iteration: 48
trial_id: 558f27aa
warmup_time: 0.0029478073120117188
Result for objective_5592ca68:
date: 2022-07-22_15-19-52
done: false
experiment_id: 1de409fb58a84c3bba522620ec7e1fe4
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 3.3171620341364583
neg_mean_loss: -3.3171620341364583
node_ip: 127.0.0.1
pid: 45992
time_since_restore: 5.171269178390503
time_this_iter_s: 0.10331320762634277
time_total_s: 5.171269178390503
timestamp: 1658499592
timesteps_since_restore: 0
training_iteration: 48
trial_id: 5592ca68
warmup_time: 0.0027618408203125
Result for objective_53b39e84:
date: 2022-07-22_15-19-54
done: false
experiment_id: abfeeaaa7eae4ad798c655bed2d1e223
hostname: Kais-MacBook-Pro.local
iterations: 93
iterations_since_restore: 94
mean_loss: 4.148686061272061
neg_mean_loss: -4.148686061272061
node_ip: 127.0.0.1
pid: 45983
time_since_restore: 10.137071132659912
time_this_iter_s: 0.10901093482971191
time_total_s: 10.137071132659912
timestamp: 1658499594
timesteps_since_restore: 0
training_iteration: 94
trial_id: 53b39e84
warmup_time: 0.002878904342651367
Result for objective_53b39e84:
date: 2022-07-22_15-19-54
done: true
experiment_id: abfeeaaa7eae4ad798c655bed2d1e223
experiment_tag: 5_activation=relu_tanh,height=25.7812,steps=100,width=1.7770
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 4.1159662035734215
neg_mean_loss: -4.1159662035734215
node_ip: 127.0.0.1
pid: 45983
time_since_restore: 10.796404361724854
time_this_iter_s: 0.10543107986450195
time_total_s: 10.796404361724854
timestamp: 1658499594
timesteps_since_restore: 0
training_iteration: 100
trial_id: 53b39e84
warmup_time: 0.002878904342651367
Result for objective_5b999414:
date: 2022-07-22_15-19-57
done: false
experiment_id: 5548713c4f5046c0a44e1946db840d73
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 12.103540718376786
neg_mean_loss: -12.103540718376786
node_ip: 127.0.0.1
pid: 46011
time_since_restore: 0.10435795783996582
time_this_iter_s: 0.10435795783996582
time_total_s: 0.10435795783996582
timestamp: 1658499597
timesteps_since_restore: 0
training_iteration: 1
trial_id: 5b999414
warmup_time: 0.002862691879272461
Result for objective_558c6132:
date: 2022-07-22_15-19-57
done: false
experiment_id: 98df2ca60c5f4a33a3da6870c3d8db4c
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 4.790299222921972
neg_mean_loss: -4.790299222921972
node_ip: 127.0.0.1
pid: 45990
time_since_restore: 10.228116989135742
time_this_iter_s: 0.146165132522583
time_total_s: 10.228116989135742
timestamp: 1658499597
timesteps_since_restore: 0
training_iteration: 95
trial_id: 558c6132
warmup_time: 0.003049135208129883
Result for objective_558f27aa:
date: 2022-07-22_15-19-57
done: false
experiment_id: 73def1ffc7f443669328b88695453248
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 13.944411303108408
neg_mean_loss: -13.944411303108408
node_ip: 127.0.0.1
pid: 45991
time_since_restore: 10.248456001281738
time_this_iter_s: 0.13899707794189453
time_total_s: 10.248456001281738
timestamp: 1658499597
timesteps_since_restore: 0
training_iteration: 95
trial_id: 558f27aa
warmup_time: 0.0029478073120117188
Result for objective_5592ca68:
date: 2022-07-22_15-19-57
done: false
experiment_id: 1de409fb58a84c3bba522620ec7e1fe4
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 3.064378230421662
neg_mean_loss: -3.064378230421662
node_ip: 127.0.0.1
pid: 45992
time_since_restore: 10.244185209274292
time_this_iter_s: 0.16223502159118652
time_total_s: 10.244185209274292
timestamp: 1658499597
timesteps_since_restore: 0
training_iteration: 95
trial_id: 5592ca68
warmup_time: 0.0027618408203125
Result for objective_558c6132:
date: 2022-07-22_15-19-58
done: true
experiment_id: 98df2ca60c5f4a33a3da6870c3d8db4c
experiment_tag: 6_activation=relu_tanh,height=33.1252,steps=100,width=2.1202
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 4.767266385536331
neg_mean_loss: -4.767266385536331
node_ip: 127.0.0.1
pid: 45990
time_since_restore: 10.765881776809692
time_this_iter_s: 0.10750794410705566
time_total_s: 10.765881776809692
timestamp: 1658499598
timesteps_since_restore: 0
training_iteration: 100
trial_id: 558c6132
warmup_time: 0.003049135208129883
Result for objective_558f27aa:
date: 2022-07-22_15-19-58
done: true
experiment_id: 73def1ffc7f443669328b88695453248
experiment_tag: 7_activation=relu_tanh,height=40.9290,steps=100,width=0.0138
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 13.890665978269285
neg_mean_loss: -13.890665978269285
node_ip: 127.0.0.1
pid: 45991
time_since_restore: 10.784975051879883
time_this_iter_s: 0.10729503631591797
time_total_s: 10.784975051879883
timestamp: 1658499598
timesteps_since_restore: 0
training_iteration: 100
trial_id: 558f27aa
warmup_time: 0.0029478073120117188
Result for objective_5592ca68:
date: 2022-07-22_15-19-58
done: true
experiment_id: 1de409fb58a84c3bba522620ec7e1fe4
experiment_tag: 8_activation=relu_tanh,height=17.9775,steps=100,width=3.8835
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 3.0512532903599245
neg_mean_loss: -3.0512532903599245
node_ip: 127.0.0.1
pid: 45992
time_since_restore: 10.780152082443237
time_this_iter_s: 0.10820770263671875
time_total_s: 10.780152082443237
timestamp: 1658499598
timesteps_since_restore: 0
training_iteration: 100
trial_id: 5592ca68
warmup_time: 0.0027618408203125
Result for objective_5d8462c2:
date: 2022-07-22_15-20-00
done: false
experiment_id: 37975742a6894daeb580d222effadf3c
hostname: Kais-MacBook-Pro.local
iterations: 0
iterations_since_restore: 1
mean_loss: 13.491949373944276
neg_mean_loss: -13.491949373944276
node_ip: 127.0.0.1
pid: 46018
time_since_restore: 0.1021728515625
time_this_iter_s: 0.1021728515625
time_total_s: 0.1021728515625
timestamp: 1658499600
timesteps_since_restore: 0
training_iteration: 1
trial_id: 5d8462c2
warmup_time: 0.0030527114868164062
Result for objective_5b999414:
date: 2022-07-22_15-20-02
done: false
experiment_id: 5548713c4f5046c0a44e1946db840d73
hostname: Kais-MacBook-Pro.local
iterations: 29
iterations_since_restore: 30
mean_loss: 3.0563505542773735
neg_mean_loss: -3.0563505542773735
node_ip: 127.0.0.1
pid: 46011
time_since_restore: 5.116016149520874
time_this_iter_s: 0.1060340404510498
time_total_s: 5.116016149520874
timestamp: 1658499602
timesteps_since_restore: 0
training_iteration: 30
trial_id: 5b999414
warmup_time: 0.002862691879272461
Result for objective_5d8462c2:
date: 2022-07-22_15-20-05
done: false
experiment_id: 37975742a6894daeb580d222effadf3c
hostname: Kais-MacBook-Pro.local
iterations: 47
iterations_since_restore: 48
mean_loss: 3.9441144421537513
neg_mean_loss: -3.9441144421537513
node_ip: 127.0.0.1
pid: 46018
time_since_restore: 5.143904685974121
time_this_iter_s: 0.1046457290649414
time_total_s: 5.143904685974121
timestamp: 1658499605
timesteps_since_restore: 0
training_iteration: 48
trial_id: 5d8462c2
warmup_time: 0.0030527114868164062
Result for objective_5b999414:
date: 2022-07-22_15-20-07
done: false
experiment_id: 5548713c4f5046c0a44e1946db840d73
hostname: Kais-MacBook-Pro.local
iterations: 76
iterations_since_restore: 77
mean_loss: 2.4898772989369
neg_mean_loss: -2.4898772989369
node_ip: 127.0.0.1
pid: 46011
time_since_restore: 10.162328004837036
time_this_iter_s: 0.10597825050354004
time_total_s: 10.162328004837036
timestamp: 1658499607
timesteps_since_restore: 0
training_iteration: 77
trial_id: 5b999414
warmup_time: 0.002862691879272461
Result for objective_5b999414:
date: 2022-07-22_15-20-09
done: true
experiment_id: 5548713c4f5046c0a44e1946db840d73
experiment_tag: 9_activation=relu_tanh,height=11.0354,steps=100,width=3.2742
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 2.4028084118568103
neg_mean_loss: -2.4028084118568103
node_ip: 127.0.0.1
pid: 46011
time_since_restore: 12.626188039779663
time_this_iter_s: 0.10697293281555176
time_total_s: 12.626188039779663
timestamp: 1658499609
timesteps_since_restore: 0
training_iteration: 100
trial_id: 5b999414
warmup_time: 0.002862691879272461
Result for objective_5d8462c2:
date: 2022-07-22_15-20-10
done: false
experiment_id: 37975742a6894daeb580d222effadf3c
hostname: Kais-MacBook-Pro.local
iterations: 94
iterations_since_restore: 95
mean_loss: 3.723261470545891
neg_mean_loss: -3.723261470545891
node_ip: 127.0.0.1
pid: 46018
time_since_restore: 10.198927879333496
time_this_iter_s: 0.10776281356811523
time_total_s: 10.198927879333496
timestamp: 1658499610
timesteps_since_restore: 0
training_iteration: 95
trial_id: 5d8462c2
warmup_time: 0.0030527114868164062
Result for objective_5d8462c2:
date: 2022-07-22_15-20-11
done: true
experiment_id: 37975742a6894daeb580d222effadf3c
experiment_tag: 10_activation=relu_tanh,height=24.9195,steps=100,width=4.4927
hostname: Kais-MacBook-Pro.local
iterations: 99
iterations_since_restore: 100
mean_loss: 3.711835922326217
neg_mean_loss: -3.711835922326217
node_ip: 127.0.0.1
pid: 46018
time_since_restore: 10.733852863311768
time_this_iter_s: 0.10374593734741211
time_total_s: 10.733852863311768
timestamp: 1658499611
timesteps_since_restore: 0
training_iteration: 100
trial_id: 5d8462c2
warmup_time: 0.0030527114868164062
Here are the hyperparameters found to minimize the mean loss of the defined objective.
print("Best hyperparameters found were: ", results.get_best_result().config)
Best hyperparameters found were: {'steps': 100, 'width': 15.124213652112319, 'height': -97.27768667042203, 'activation': 'relu, tanh'}