Using Keras & TensorFlow with Tune

Keras & TensorFlow Logo

Example

import argparse
import os

from filelock import FileLock
from tensorflow.keras.datasets import mnist

import ray
from ray import air, tune
from ray.tune.schedulers import AsyncHyperBandScheduler
from ray.tune.integration.keras import TuneReportCallback


def train_mnist(config):
    # https://github.com/tensorflow/tensorflow/issues/32159
    import tensorflow as tf

    batch_size = 128
    num_classes = 10
    epochs = 12

    with FileLock(os.path.expanduser("~/.data.lock")):
        (x_train, y_train), (x_test, y_test) = mnist.load_data()
    x_train, x_test = x_train / 255.0, x_test / 255.0
    model = tf.keras.models.Sequential(
        [
            tf.keras.layers.Flatten(input_shape=(28, 28)),
            tf.keras.layers.Dense(config["hidden"], activation="relu"),
            tf.keras.layers.Dropout(0.2),
            tf.keras.layers.Dense(num_classes, activation="softmax"),
        ]
    )

    model.compile(
        loss="sparse_categorical_crossentropy",
        optimizer=tf.keras.optimizers.SGD(lr=config["lr"], momentum=config["momentum"]),
        metrics=["accuracy"],
    )

    model.fit(
        x_train,
        y_train,
        batch_size=batch_size,
        epochs=epochs,
        verbose=0,
        validation_data=(x_test, y_test),
        callbacks=[TuneReportCallback({"mean_accuracy": "accuracy"})],
    )


def tune_mnist(num_training_iterations):
    sched = AsyncHyperBandScheduler(
        time_attr="training_iteration", max_t=400, grace_period=20
    )
    
    tuner = tune.Tuner(
        tune.with_resources(
            train_mnist,
            resources={"cpu": 2, "gpu": 0}
        ),
        tune_config=tune.TuneConfig(
            metric="mean_accuracy",
            mode="max",
            scheduler=sched,
            num_samples=10,
        ),
        run_config=air.RunConfig(
            name="exp",
            stop={"mean_accuracy": 0.99, "training_iteration": num_training_iterations},
        ),
        param_space={
            "threads": 2,
            "lr": tune.uniform(0.001, 0.1),
            "momentum": tune.uniform(0.1, 0.9),
            "hidden": tune.randint(32, 512),
        },
    )
    results = tuner.fit()

    print("Best hyperparameters found were: ", results.get_best_result().config)


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--smoke-test", action="store_true", help="Finish quickly for testing"
    )
    parser.add_argument(
        "--server-address",
        type=str,
        default=None,
        required=False,
        help="The address of server to connect to if using " "Ray Client.",
    )
    args, _ = parser.parse_known_args()
    if args.smoke_test:
        ray.init(num_cpus=4)
    elif args.server_address:
        ray.init(f"ray://{args.server_address}")

    tune_mnist(num_training_iterations=5 if args.smoke_test else 300)
2022-07-22 16:16:58,114	INFO services.py:1483 -- View the Ray dashboard at http://127.0.0.1:8269
2022-07-22 16:17:00,822	WARNING function_trainable.py:619 -- Function checkpointing is disabled. This may result in unexpected behavior when using checkpointing features or certain schedulers. To enable, set the train function arguments to be `func(config, checkpoint_dir=None)`.
== Status ==
Current time: 2022-07-22 16:18:36 (running for 00:01:35.04)
Memory usage on this node: 9.0/16.0 GiB
Using AsyncHyperBand: num_stopped=0 Bracket: Iter 320.000: None | Iter 80.000: None | Iter 20.000: None
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.47 GiB heap, 0.0/2.0 GiB objects
Current best trial: 55a9b_00002 with mean_accuracy=0.9904166460037231 and parameters={'threads': 2, 'lr': 0.09518133271957563, 'momentum': 0.8254987643140009, 'hidden': 258}
Result logdir: /Users/kai/ray_results/exp
Number of trials: 10/10 (10 TERMINATED)
Trial name status loc hidden lr momentum acc iter total time (s)
train_mnist_55a9b_00000TERMINATED127.0.0.1:51968 2760.0406397 0.8177880.98455 12 78.3252
train_mnist_55a9b_00001TERMINATED127.0.0.1:51977 3800.0873557 0.5246340.983717 12 74.9888
train_mnist_55a9b_00002TERMINATED127.0.0.1:51984 2580.0951813 0.8254990.990417 11 64.1272
train_mnist_55a9b_00003TERMINATED127.0.0.1:51991 2550.0971683 0.23161 0.977633 12 60.8475
train_mnist_55a9b_00004TERMINATED127.0.0.1:52000 3030.00440117 0.3254390.90775 12 55.5722
train_mnist_55a9b_00005TERMINATED127.0.0.1:52007 920.0651919 0.7101830.974867 12 44.8092
train_mnist_55a9b_00006TERMINATED127.0.0.1:52016 2110.0731116 0.1277510.97025 12 42.1217
train_mnist_55a9b_00007TERMINATED127.0.0.1:52021 1810.0362389 0.7903450.979967 12 41.7632
train_mnist_55a9b_00008TERMINATED127.0.0.1:52007 1420.0323741 0.6604180.969367 12 14.1527
train_mnist_55a9b_00009TERMINATED127.0.0.1:51984 970.0244971 0.1750450.9407 12 12.6405


2022-07-22 16:17:01,834	INFO plugin_schema_manager.py:52 -- Loading the default runtime env schemas: ['/Users/kai/coding/ray/python/ray/_private/runtime_env/../../runtime_env/schemas/working_dir_schema.json', '/Users/kai/coding/ray/python/ray/_private/runtime_env/../../runtime_env/schemas/pip_schema.json'].
(train_mnist pid=51968) 2022-07-22 16:17:08.627419: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
(train_mnist pid=51968) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
(train_mnist pid=51968) /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/keras/optimizer_v2/optimizer_v2.py:356: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
(train_mnist pid=51968)   "The `lr` argument is deprecated, use `learning_rate` instead.")
(train_mnist pid=51968) 2022-07-22 16:17:08.947939: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
(train_mnist pid=51977) 2022-07-22 16:17:14.473677: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
(train_mnist pid=51977) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
(train_mnist pid=51977) /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/keras/optimizer_v2/optimizer_v2.py:356: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
(train_mnist pid=51977)   "The `lr` argument is deprecated, use `learning_rate` instead.")
(train_mnist pid=51977) 2022-07-22 16:17:14.635104: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
(train_mnist pid=51984) 2022-07-22 16:17:20.406624: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
(train_mnist pid=51984) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
(train_mnist pid=51984) /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/keras/optimizer_v2/optimizer_v2.py:356: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
(train_mnist pid=51984)   "The `lr` argument is deprecated, use `learning_rate` instead.")
(train_mnist pid=51984) 2022-07-22 16:17:20.681960: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
(train_mnist pid=51991) 2022-07-22 16:17:26.109460: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
(train_mnist pid=51991) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
(train_mnist pid=51991) /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/keras/optimizer_v2/optimizer_v2.py:356: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
(train_mnist pid=51991)   "The `lr` argument is deprecated, use `learning_rate` instead.")
(train_mnist pid=51991) 2022-07-22 16:17:26.303375: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
(train_mnist pid=52000) 2022-07-22 16:17:31.899252: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
(train_mnist pid=52000) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
(train_mnist pid=52000) /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/keras/optimizer_v2/optimizer_v2.py:356: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
(train_mnist pid=52000)   "The `lr` argument is deprecated, use `learning_rate` instead.")
(train_mnist pid=52000) 2022-07-22 16:17:32.300424: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
(train_mnist pid=52007) 2022-07-22 16:17:37.937471: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
(train_mnist pid=52007) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
(train_mnist pid=52007) /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/keras/optimizer_v2/optimizer_v2.py:356: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
(train_mnist pid=52007)   "The `lr` argument is deprecated, use `learning_rate` instead.")
(train_mnist pid=52007) 2022-07-22 16:17:38.263888: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
(train_mnist pid=52016) 2022-07-22 16:17:43.657379: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
(train_mnist pid=52016) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
(train_mnist pid=52016) /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/keras/optimizer_v2/optimizer_v2.py:356: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
(train_mnist pid=52016)   "The `lr` argument is deprecated, use `learning_rate` instead.")
(train_mnist pid=52016) 2022-07-22 16:17:43.828809: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
Result for train_mnist_55a9b_00000:
  date: 2022-07-22_16-17-10
  done: false
  experiment_id: 3659349c38c746cfb71b4db5eb9302a0
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 1
  mean_accuracy: 0.8903833627700806
  node_ip: 127.0.0.1
  pid: 51968
  time_since_restore: 2.439258098602295
  time_this_iter_s: 2.439258098602295
  time_total_s: 2.439258098602295
  timestamp: 1658503030
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 55a9b_00000
  warmup_time: 0.003445863723754883
  
Result for train_mnist_55a9b_00004:
  date: 2022-07-22_16-17-33
  done: false
  experiment_id: 6eb62b7cb38f442a867a9094f0664701
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 1
  mean_accuracy: 0.6376166939735413
  node_ip: 127.0.0.1
  pid: 52000
  time_since_restore: 2.4364511966705322
  time_this_iter_s: 2.4364511966705322
  time_total_s: 2.4364511966705322
  timestamp: 1658503053
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 55a9b_00004
  warmup_time: 0.0030939579010009766
  
Result for train_mnist_55a9b_00006:
  date: 2022-07-22_16-17-45
  done: false
  experiment_id: 9594405e38084311a891b48addd13f75
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 1
  mean_accuracy: 0.8557000160217285
  node_ip: 127.0.0.1
  pid: 52016
  time_since_restore: 1.8570480346679688
  time_this_iter_s: 1.8570480346679688
  time_total_s: 1.8570480346679688
  timestamp: 1658503065
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 55a9b_00006
  warmup_time: 0.003566741943359375
  
Result for train_mnist_55a9b_00001:
  date: 2022-07-22_16-17-15
  done: false
  experiment_id: fcdeb049f9614755a9b7c9420ca2ae5e
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 1
  mean_accuracy: 0.8887666463851929
  node_ip: 127.0.0.1
  pid: 51977
  time_since_restore: 1.9353628158569336
  time_this_iter_s: 1.9353628158569336
  time_total_s: 1.9353628158569336
  timestamp: 1658503035
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 55a9b_00001
  warmup_time: 0.0029449462890625
  
Result for train_mnist_55a9b_00005:
  date: 2022-07-22_16-17-39
  done: false
  experiment_id: 8dbd22e6caed4fe39351dffa3ef14eac
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 1
  mean_accuracy: 0.8789666891098022
  node_ip: 127.0.0.1
  pid: 52007
  time_since_restore: 2.3337321281433105
  time_this_iter_s: 2.3337321281433105
  time_total_s: 2.3337321281433105
  timestamp: 1658503059
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 55a9b_00005
  warmup_time: 0.005449056625366211
  
Result for train_mnist_55a9b_00002:
  date: 2022-07-22_16-17-21
  done: false
  experiment_id: c4f803baf65f4d4e9fd6abc85b2fd00c
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 1
  mean_accuracy: 0.9112833142280579
  node_ip: 127.0.0.1
  pid: 51984
  time_since_restore: 2.3220012187957764
  time_this_iter_s: 2.3220012187957764
  time_total_s: 2.3220012187957764
  timestamp: 1658503041
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 55a9b_00002
  warmup_time: 0.0028328895568847656
  
Result for train_mnist_55a9b_00003:
  date: 2022-07-22_16-17-27
  done: false
  experiment_id: 469478f02b4a43f5b44c40e59989ad39
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 1
  mean_accuracy: 0.8743166923522949
  node_ip: 127.0.0.1
  pid: 51991
  time_since_restore: 2.0278611183166504
  time_this_iter_s: 2.0278611183166504
  time_total_s: 2.0278611183166504
  timestamp: 1658503047
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 55a9b_00003
  warmup_time: 0.0033779144287109375
  
(train_mnist pid=52021) 2022-07-22 16:17:51.567914: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
(train_mnist pid=52021) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
(train_mnist pid=52021) /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/keras/optimizer_v2/optimizer_v2.py:356: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
(train_mnist pid=52021)   "The `lr` argument is deprecated, use `learning_rate` instead.")
(train_mnist pid=52021) 2022-07-22 16:17:52.977183: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
Result for train_mnist_55a9b_00005:
  date: 2022-07-22_16-17-54
  done: false
  experiment_id: 8dbd22e6caed4fe39351dffa3ef14eac
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 3
  mean_accuracy: 0.9490833282470703
  node_ip: 127.0.0.1
  pid: 52007
  time_since_restore: 17.22033405303955
  time_this_iter_s: 2.672102928161621
  time_total_s: 17.22033405303955
  timestamp: 1658503074
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: 55a9b_00005
  warmup_time: 0.005449056625366211
  
Result for train_mnist_55a9b_00006:
  date: 2022-07-22_16-17-54
  done: false
  experiment_id: 9594405e38084311a891b48addd13f75
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 3
  mean_accuracy: 0.9327999949455261
  node_ip: 127.0.0.1
  pid: 52016
  time_since_restore: 11.758372068405151
  time_this_iter_s: 3.0426323413848877
  time_total_s: 11.758372068405151
  timestamp: 1658503074
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: 55a9b_00006
  warmup_time: 0.003566741943359375
  
Result for train_mnist_55a9b_00003:
  date: 2022-07-22_16-17-55
  done: false
  experiment_id: 469478f02b4a43f5b44c40e59989ad39
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 3
  mean_accuracy: 0.9454166889190674
  node_ip: 127.0.0.1
  pid: 51991
  time_since_restore: 29.733185052871704
  time_this_iter_s: 3.0363340377807617
  time_total_s: 29.733185052871704
  timestamp: 1658503075
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: 55a9b_00003
  warmup_time: 0.0033779144287109375
  
Result for train_mnist_55a9b_00000:
  date: 2022-07-22_16-17-55
  done: false
  experiment_id: 3659349c38c746cfb71b4db5eb9302a0
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 3
  mean_accuracy: 0.958216667175293
  node_ip: 127.0.0.1
  pid: 51968
  time_since_restore: 47.272178173065186
  time_this_iter_s: 3.2986061573028564
  time_total_s: 47.272178173065186
  timestamp: 1658503075
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: 55a9b_00000
  warmup_time: 0.003445863723754883
  
Result for train_mnist_55a9b_00004:
  date: 2022-07-22_16-17-55
  done: false
  experiment_id: 6eb62b7cb38f442a867a9094f0664701
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 3
  mean_accuracy: 0.8524500131607056
  node_ip: 127.0.0.1
  pid: 52000
  time_since_restore: 24.11396098136902
  time_this_iter_s: 3.2331089973449707
  time_total_s: 24.11396098136902
  timestamp: 1658503075
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: 55a9b_00004
  warmup_time: 0.0030939579010009766
  
Result for train_mnist_55a9b_00002:
  date: 2022-07-22_16-17-55
  done: false
  experiment_id: c4f803baf65f4d4e9fd6abc85b2fd00c
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 3
  mean_accuracy: 0.9695500135421753
  node_ip: 127.0.0.1
  pid: 51984
  time_since_restore: 35.78592824935913
  time_this_iter_s: 3.021165132522583
  time_total_s: 35.78592824935913
  timestamp: 1658503075
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: 55a9b_00002
  warmup_time: 0.0028328895568847656
  
Result for train_mnist_55a9b_00001:
  date: 2022-07-22_16-17-56
  done: false
  experiment_id: fcdeb049f9614755a9b7c9420ca2ae5e
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 3
  mean_accuracy: 0.9560333490371704
  node_ip: 127.0.0.1
  pid: 51977
  time_since_restore: 42.38909387588501
  time_this_iter_s: 3.753290891647339
  time_total_s: 42.38909387588501
  timestamp: 1658503076
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: 55a9b_00001
  warmup_time: 0.0029449462890625
  
Result for train_mnist_55a9b_00005:
  date: 2022-07-22_16-18-00
  done: false
  experiment_id: 8dbd22e6caed4fe39351dffa3ef14eac
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 5
  mean_accuracy: 0.9611166715621948
  node_ip: 127.0.0.1
  pid: 52007
  time_since_restore: 23.303561210632324
  time_this_iter_s: 2.933852195739746
  time_total_s: 23.303561210632324
  timestamp: 1658503080
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: 55a9b_00005
  warmup_time: 0.005449056625366211
  
Result for train_mnist_55a9b_00007:
  date: 2022-07-22_16-18-01
  done: false
  experiment_id: d9469b1fc58b41db88da5446dc2a3b23
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 1
  mean_accuracy: 0.8797500133514404
  node_ip: 127.0.0.1
  pid: 52021
  time_since_restore: 12.469872951507568
  time_this_iter_s: 12.469872951507568
  time_total_s: 12.469872951507568
  timestamp: 1658503081
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 55a9b_00007
  warmup_time: 0.0028028488159179688
  
Result for train_mnist_55a9b_00006:
  date: 2022-07-22_16-18-01
  done: false
  experiment_id: 9594405e38084311a891b48addd13f75
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 5
  mean_accuracy: 0.9499499797821045
  node_ip: 127.0.0.1
  pid: 52016
  time_since_restore: 18.780059814453125
  time_this_iter_s: 3.3080599308013916
  time_total_s: 18.780059814453125
  timestamp: 1658503081
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: 55a9b_00006
  warmup_time: 0.003566741943359375
  
Result for train_mnist_55a9b_00003:
  date: 2022-07-22_16-18-02
  done: false
  experiment_id: 469478f02b4a43f5b44c40e59989ad39
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 5
  mean_accuracy: 0.9601166844367981
  node_ip: 127.0.0.1
  pid: 51991
  time_since_restore: 36.93912100791931
  time_this_iter_s: 3.4057939052581787
  time_total_s: 36.93912100791931
  timestamp: 1658503082
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: 55a9b_00003
  warmup_time: 0.0033779144287109375
  
Result for train_mnist_55a9b_00000:
  date: 2022-07-22_16-18-02
  done: false
  experiment_id: 3659349c38c746cfb71b4db5eb9302a0
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 5
  mean_accuracy: 0.970466673374176
  node_ip: 127.0.0.1
  pid: 51968
  time_since_restore: 54.49850010871887
  time_this_iter_s: 3.4417831897735596
  time_total_s: 54.49850010871887
  timestamp: 1658503082
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: 55a9b_00000
  warmup_time: 0.003445863723754883
  
Result for train_mnist_55a9b_00004:
  date: 2022-07-22_16-18-02
  done: false
  experiment_id: 6eb62b7cb38f442a867a9094f0664701
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 5
  mean_accuracy: 0.8777499794960022
  node_ip: 127.0.0.1
  pid: 52000
  time_since_restore: 31.513713121414185
  time_this_iter_s: 3.506195068359375
  time_total_s: 31.513713121414185
  timestamp: 1658503082
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: 55a9b_00004
  warmup_time: 0.0030939579010009766
  
Result for train_mnist_55a9b_00002:
  date: 2022-07-22_16-18-02
  done: false
  experiment_id: c4f803baf65f4d4e9fd6abc85b2fd00c
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 5
  mean_accuracy: 0.979283332824707
  node_ip: 127.0.0.1
  pid: 51984
  time_since_restore: 43.266417026519775
  time_this_iter_s: 3.3383469581604004
  time_total_s: 43.266417026519775
  timestamp: 1658503082
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: 55a9b_00002
  warmup_time: 0.0028328895568847656
  
Result for train_mnist_55a9b_00001:
  date: 2022-07-22_16-18-04
  done: false
  experiment_id: fcdeb049f9614755a9b7c9420ca2ae5e
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 5
  mean_accuracy: 0.9692999720573425
  node_ip: 127.0.0.1
  pid: 51977
  time_since_restore: 50.620792865753174
  time_this_iter_s: 4.001068115234375
  time_total_s: 50.620792865753174
  timestamp: 1658503084
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: 55a9b_00001
  warmup_time: 0.0029449462890625
  
Result for train_mnist_55a9b_00005:
  date: 2022-07-22_16-18-06
  done: false
  experiment_id: 8dbd22e6caed4fe39351dffa3ef14eac
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 7
  mean_accuracy: 0.96711665391922
  node_ip: 127.0.0.1
  pid: 52007
  time_since_restore: 29.40476107597351
  time_this_iter_s: 2.976076126098633
  time_total_s: 29.40476107597351
  timestamp: 1658503086
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: 55a9b_00005
  warmup_time: 0.005449056625366211
  
Result for train_mnist_55a9b_00007:
  date: 2022-07-22_16-18-07
  done: false
  experiment_id: d9469b1fc58b41db88da5446dc2a3b23
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 3
  mean_accuracy: 0.951033353805542
  node_ip: 127.0.0.1
  pid: 52021
  time_since_restore: 18.96213722229004
  time_this_iter_s: 3.252371311187744
  time_total_s: 18.96213722229004
  timestamp: 1658503087
  timesteps_since_restore: 0
  training_iteration: 3
  trial_id: 55a9b_00007
  warmup_time: 0.0028028488159179688
  
Result for train_mnist_55a9b_00006:
  date: 2022-07-22_16-18-08
  done: false
  experiment_id: 9594405e38084311a891b48addd13f75
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 7
  mean_accuracy: 0.9584500193595886
  node_ip: 127.0.0.1
  pid: 52016
  time_since_restore: 25.336583852767944
  time_this_iter_s: 3.311979055404663
  time_total_s: 25.336583852767944
  timestamp: 1658503088
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: 55a9b_00006
  warmup_time: 0.003566741943359375
  
Result for train_mnist_55a9b_00003:
  date: 2022-07-22_16-18-09
  done: false
  experiment_id: 469478f02b4a43f5b44c40e59989ad39
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 7
  mean_accuracy: 0.9675499796867371
  node_ip: 127.0.0.1
  pid: 51991
  time_since_restore: 43.7107310295105
  time_this_iter_s: 3.3927559852600098
  time_total_s: 43.7107310295105
  timestamp: 1658503089
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: 55a9b_00003
  warmup_time: 0.0033779144287109375
  
Result for train_mnist_55a9b_00000:
  date: 2022-07-22_16-18-09
  done: false
  experiment_id: 3659349c38c746cfb71b4db5eb9302a0
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 7
  mean_accuracy: 0.9763000011444092
  node_ip: 127.0.0.1
  pid: 51968
  time_since_restore: 61.30248522758484
  time_this_iter_s: 3.4063682556152344
  time_total_s: 61.30248522758484
  timestamp: 1658503089
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: 55a9b_00000
  warmup_time: 0.003445863723754883
  
Result for train_mnist_55a9b_00002:
  date: 2022-07-22_16-18-09
  done: false
  experiment_id: c4f803baf65f4d4e9fd6abc85b2fd00c
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 7
  mean_accuracy: 0.9840666651725769
  node_ip: 127.0.0.1
  pid: 51984
  time_since_restore: 50.212465047836304
  time_this_iter_s: 3.43766188621521
  time_total_s: 50.212465047836304
  timestamp: 1658503089
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: 55a9b_00002
  warmup_time: 0.0028328895568847656
  
Result for train_mnist_55a9b_00004:
  date: 2022-07-22_16-18-09
  done: false
  experiment_id: 6eb62b7cb38f442a867a9094f0664701
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 7
  mean_accuracy: 0.8899999856948853
  node_ip: 127.0.0.1
  pid: 52000
  time_since_restore: 38.63890194892883
  time_this_iter_s: 3.5783908367156982
  time_total_s: 38.63890194892883
  timestamp: 1658503089
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: 55a9b_00004
  warmup_time: 0.0030939579010009766
  
Result for train_mnist_55a9b_00005:
  date: 2022-07-22_16-18-12
  done: false
  experiment_id: 8dbd22e6caed4fe39351dffa3ef14eac
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 9
  mean_accuracy: 0.9712333083152771
  node_ip: 127.0.0.1
  pid: 52007
  time_since_restore: 35.3185760974884
  time_this_iter_s: 3.0241990089416504
  time_total_s: 35.3185760974884
  timestamp: 1658503092
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: 55a9b_00005
  warmup_time: 0.005449056625366211
  
Result for train_mnist_55a9b_00001:
  date: 2022-07-22_16-18-12
  done: false
  experiment_id: fcdeb049f9614755a9b7c9420ca2ae5e
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 7
  mean_accuracy: 0.9755333065986633
  node_ip: 127.0.0.1
  pid: 51977
  time_since_restore: 58.57745599746704
  time_this_iter_s: 3.936232089996338
  time_total_s: 58.57745599746704
  timestamp: 1658503092
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: 55a9b_00001
  warmup_time: 0.0029449462890625
  
Result for train_mnist_55a9b_00007:
  date: 2022-07-22_16-18-14
  done: false
  experiment_id: d9469b1fc58b41db88da5446dc2a3b23
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 5
  mean_accuracy: 0.9648333191871643
  node_ip: 127.0.0.1
  pid: 52021
  time_since_restore: 25.25843620300293
  time_this_iter_s: 3.094501256942749
  time_total_s: 25.25843620300293
  timestamp: 1658503094
  timesteps_since_restore: 0
  training_iteration: 5
  trial_id: 55a9b_00007
  warmup_time: 0.0028028488159179688
  
Result for train_mnist_55a9b_00006:
  date: 2022-07-22_16-18-15
  done: false
  experiment_id: 9594405e38084311a891b48addd13f75
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 9
  mean_accuracy: 0.9646666646003723
  node_ip: 127.0.0.1
  pid: 52016
  time_since_restore: 32.048911809921265
  time_this_iter_s: 3.315690755844116
  time_total_s: 32.048911809921265
  timestamp: 1658503095
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: 55a9b_00006
  warmup_time: 0.003566741943359375
  
Result for train_mnist_55a9b_00003:
  date: 2022-07-22_16-18-15
  done: false
  experiment_id: 469478f02b4a43f5b44c40e59989ad39
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 9
  mean_accuracy: 0.9729499816894531
  node_ip: 127.0.0.1
  pid: 51991
  time_since_restore: 50.50909209251404
  time_this_iter_s: 3.4110782146453857
  time_total_s: 50.50909209251404
  timestamp: 1658503095
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: 55a9b_00003
  warmup_time: 0.0033779144287109375
  
Result for train_mnist_55a9b_00000:
  date: 2022-07-22_16-18-16
  done: false
  experiment_id: 3659349c38c746cfb71b4db5eb9302a0
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 9
  mean_accuracy: 0.9807666540145874
  node_ip: 127.0.0.1
  pid: 51968
  time_since_restore: 68.26757216453552
  time_this_iter_s: 3.4475879669189453
  time_total_s: 68.26757216453552
  timestamp: 1658503096
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: 55a9b_00000
  warmup_time: 0.003445863723754883
  
Result for train_mnist_55a9b_00002:
  date: 2022-07-22_16-18-16
  done: false
  experiment_id: c4f803baf65f4d4e9fd6abc85b2fd00c
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 9
  mean_accuracy: 0.9872999787330627
  node_ip: 127.0.0.1
  pid: 51984
  time_since_restore: 57.01431703567505
  time_this_iter_s: 3.3804008960723877
  time_total_s: 57.01431703567505
  timestamp: 1658503096
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: 55a9b_00002
  warmup_time: 0.0028328895568847656
  
Result for train_mnist_55a9b_00004:
  date: 2022-07-22_16-18-16
  done: false
  experiment_id: 6eb62b7cb38f442a867a9094f0664701
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 9
  mean_accuracy: 0.8989166617393494
  node_ip: 127.0.0.1
  pid: 52000
  time_since_restore: 45.67929005622864
  time_this_iter_s: 3.4561610221862793
  time_total_s: 45.67929005622864
  timestamp: 1658503096
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: 55a9b_00004
  warmup_time: 0.0030939579010009766
  
Result for train_mnist_55a9b_00005:
  date: 2022-07-22_16-18-18
  done: false
  experiment_id: 8dbd22e6caed4fe39351dffa3ef14eac
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 11
  mean_accuracy: 0.9744333624839783
  node_ip: 127.0.0.1
  pid: 52007
  time_since_restore: 41.49077916145325
  time_this_iter_s: 3.172250270843506
  time_total_s: 41.49077916145325
  timestamp: 1658503098
  timesteps_since_restore: 0
  training_iteration: 11
  trial_id: 55a9b_00005
  warmup_time: 0.005449056625366211
  
Result for train_mnist_55a9b_00001:
  date: 2022-07-22_16-18-20
  done: false
  experiment_id: fcdeb049f9614755a9b7c9420ca2ae5e
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 9
  mean_accuracy: 0.9806166887283325
  node_ip: 127.0.0.1
  pid: 51977
  time_since_restore: 66.64132380485535
  time_this_iter_s: 4.0674309730529785
  time_total_s: 66.64132380485535
  timestamp: 1658503100
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: 55a9b_00001
  warmup_time: 0.0029449462890625
  
Result for train_mnist_55a9b_00007:
  date: 2022-07-22_16-18-20
  done: false
  experiment_id: d9469b1fc58b41db88da5446dc2a3b23
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 7
  mean_accuracy: 0.970716655254364
  node_ip: 127.0.0.1
  pid: 52021
  time_since_restore: 31.897236108779907
  time_this_iter_s: 3.3691420555114746
  time_total_s: 31.897236108779907
  timestamp: 1658503100
  timesteps_since_restore: 0
  training_iteration: 7
  trial_id: 55a9b_00007
  warmup_time: 0.0028028488159179688
  
Result for train_mnist_55a9b_00005:
  date: 2022-07-22_16-18-21
  done: true
  experiment_id: 8dbd22e6caed4fe39351dffa3ef14eac
  experiment_tag: 5_hidden=92,lr=0.0652,momentum=0.7102
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 12
  mean_accuracy: 0.9748666882514954
  node_ip: 127.0.0.1
  pid: 52007
  time_since_restore: 44.80922222137451
  time_this_iter_s: 3.3184430599212646
  time_total_s: 44.80922222137451
  timestamp: 1658503101
  timesteps_since_restore: 0
  training_iteration: 12
  trial_id: 55a9b_00005
  warmup_time: 0.005449056625366211
  
Result for train_mnist_55a9b_00006:
  date: 2022-07-22_16-18-22
  done: false
  experiment_id: 9594405e38084311a891b48addd13f75
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 11
  mean_accuracy: 0.9679166674613953
  node_ip: 127.0.0.1
  pid: 52016
  time_since_restore: 39.08963179588318
  time_this_iter_s: 3.4860758781433105
  time_total_s: 39.08963179588318
  timestamp: 1658503102
  timesteps_since_restore: 0
  training_iteration: 11
  trial_id: 55a9b_00006
  warmup_time: 0.003566741943359375
  
Result for train_mnist_55a9b_00003:
  date: 2022-07-22_16-18-23
  done: false
  experiment_id: 469478f02b4a43f5b44c40e59989ad39
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 11
  mean_accuracy: 0.9771833419799805
  node_ip: 127.0.0.1
  pid: 51991
  time_since_restore: 57.6213219165802
  time_this_iter_s: 3.4615819454193115
  time_total_s: 57.6213219165802
  timestamp: 1658503103
  timesteps_since_restore: 0
  training_iteration: 11
  trial_id: 55a9b_00003
  warmup_time: 0.0033779144287109375
  
Result for train_mnist_55a9b_00000:
  date: 2022-07-22_16-18-23
  done: false
  experiment_id: 3659349c38c746cfb71b4db5eb9302a0
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 11
  mean_accuracy: 0.98416668176651
  node_ip: 127.0.0.1
  pid: 51968
  time_since_restore: 75.32713007926941
  time_this_iter_s: 3.443808078765869
  time_total_s: 75.32713007926941
  timestamp: 1658503103
  timesteps_since_restore: 0
  training_iteration: 11
  trial_id: 55a9b_00000
  warmup_time: 0.003445863723754883
  
Result for train_mnist_55a9b_00002:
  date: 2022-07-22_16-18-23
  done: true
  experiment_id: c4f803baf65f4d4e9fd6abc85b2fd00c
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 11
  mean_accuracy: 0.9904166460037231
  node_ip: 127.0.0.1
  pid: 51984
  time_since_restore: 64.12720203399658
  time_this_iter_s: 3.508151054382324
  time_total_s: 64.12720203399658
  timestamp: 1658503103
  timesteps_since_restore: 0
  training_iteration: 11
  trial_id: 55a9b_00002
  warmup_time: 0.0028328895568847656
  
Result for train_mnist_55a9b_00004:
  date: 2022-07-22_16-18-23
  done: false
  experiment_id: 6eb62b7cb38f442a867a9094f0664701
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 11
  mean_accuracy: 0.9052166938781738
  node_ip: 127.0.0.1
  pid: 52000
  time_since_restore: 52.687995195388794
  time_this_iter_s: 3.420351982116699
  time_total_s: 52.687995195388794
  timestamp: 1658503103
  timesteps_since_restore: 0
  training_iteration: 11
  trial_id: 55a9b_00004
  warmup_time: 0.0030939579010009766
  
Result for train_mnist_55a9b_00006:
  date: 2022-07-22_16-18-25
  done: true
  experiment_id: 9594405e38084311a891b48addd13f75
  experiment_tag: 6_hidden=211,lr=0.0731,momentum=0.1278
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 12
  mean_accuracy: 0.9702500104904175
  node_ip: 127.0.0.1
  pid: 52016
  time_since_restore: 42.1216938495636
  time_this_iter_s: 3.03206205368042
  time_total_s: 42.1216938495636
  timestamp: 1658503105
  timesteps_since_restore: 0
  training_iteration: 12
  trial_id: 55a9b_00006
  warmup_time: 0.003566741943359375
  
Result for train_mnist_55a9b_00003:
  date: 2022-07-22_16-18-26
  done: true
  experiment_id: 469478f02b4a43f5b44c40e59989ad39
  experiment_tag: 3_hidden=255,lr=0.0972,momentum=0.2316
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 12
  mean_accuracy: 0.9776333570480347
  node_ip: 127.0.0.1
  pid: 51991
  time_since_restore: 60.8474760055542
  time_this_iter_s: 3.226154088973999
  time_total_s: 60.8474760055542
  timestamp: 1658503106
  timesteps_since_restore: 0
  training_iteration: 12
  trial_id: 55a9b_00003
  warmup_time: 0.0033779144287109375
  
Result for train_mnist_55a9b_00000:
  date: 2022-07-22_16-18-26
  done: true
  experiment_id: 3659349c38c746cfb71b4db5eb9302a0
  experiment_tag: 0_hidden=276,lr=0.0406,momentum=0.8178
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 12
  mean_accuracy: 0.9845499992370605
  node_ip: 127.0.0.1
  pid: 51968
  time_since_restore: 78.32520508766174
  time_this_iter_s: 2.998075008392334
  time_total_s: 78.32520508766174
  timestamp: 1658503106
  timesteps_since_restore: 0
  training_iteration: 12
  trial_id: 55a9b_00000
  warmup_time: 0.003445863723754883
  
Result for train_mnist_55a9b_00007:
  date: 2022-07-22_16-18-26
  done: false
  experiment_id: d9469b1fc58b41db88da5446dc2a3b23
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 9
  mean_accuracy: 0.9751333594322205
  node_ip: 127.0.0.1
  pid: 52021
  time_since_restore: 37.76195311546326
  time_this_iter_s: 2.7159180641174316
  time_total_s: 37.76195311546326
  timestamp: 1658503106
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: 55a9b_00007
  warmup_time: 0.0028028488159179688
  
Result for train_mnist_55a9b_00004:
  date: 2022-07-22_16-18-26
  done: true
  experiment_id: 6eb62b7cb38f442a867a9094f0664701
  experiment_tag: 4_hidden=303,lr=0.0044,momentum=0.3254
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 12
  mean_accuracy: 0.9077500104904175
  node_ip: 127.0.0.1
  pid: 52000
  time_since_restore: 55.57219409942627
  time_this_iter_s: 2.8841989040374756
  time_total_s: 55.57219409942627
  timestamp: 1658503106
  timesteps_since_restore: 0
  training_iteration: 12
  trial_id: 55a9b_00004
  warmup_time: 0.0030939579010009766
  
Result for train_mnist_55a9b_00001:
  date: 2022-07-22_16-18-27
  done: false
  experiment_id: fcdeb049f9614755a9b7c9420ca2ae5e
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 11
  mean_accuracy: 0.9830166697502136
  node_ip: 127.0.0.1
  pid: 51977
  time_since_restore: 73.19760584831238
  time_this_iter_s: 2.7281620502471924
  time_total_s: 73.19760584831238
  timestamp: 1658503107
  timesteps_since_restore: 0
  training_iteration: 11
  trial_id: 55a9b_00001
  warmup_time: 0.0029449462890625
  
Result for train_mnist_55a9b_00008:
  date: 2022-07-22_16-18-28
  done: false
  experiment_id: 8dbd22e6caed4fe39351dffa3ef14eac
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 1
  mean_accuracy: 0.8477166891098022
  node_ip: 127.0.0.1
  pid: 52007
  time_since_restore: 6.2436230182647705
  time_this_iter_s: 6.2436230182647705
  time_total_s: 6.2436230182647705
  timestamp: 1658503108
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 55a9b_00008
  warmup_time: 0.005449056625366211
  
Result for train_mnist_55a9b_00001:
  date: 2022-07-22_16-18-28
  done: true
  experiment_id: fcdeb049f9614755a9b7c9420ca2ae5e
  experiment_tag: 1_hidden=380,lr=0.0874,momentum=0.5246
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 12
  mean_accuracy: 0.9837166666984558
  node_ip: 127.0.0.1
  pid: 51977
  time_since_restore: 74.98881888389587
  time_this_iter_s: 1.791213035583496
  time_total_s: 74.98881888389587
  timestamp: 1658503108
  timesteps_since_restore: 0
  training_iteration: 12
  trial_id: 55a9b_00001
  warmup_time: 0.0029449462890625
  
Result for train_mnist_55a9b_00009:
  date: 2022-07-22_16-18-29
  done: false
  experiment_id: c4f803baf65f4d4e9fd6abc85b2fd00c
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 1
  mean_accuracy: 0.7675999999046326
  node_ip: 127.0.0.1
  pid: 51984
  time_since_restore: 5.303471088409424
  time_this_iter_s: 5.303471088409424
  time_total_s: 5.303471088409424
  timestamp: 1658503109
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 55a9b_00009
  warmup_time: 0.0028328895568847656
  
Result for train_mnist_55a9b_00007:
  date: 2022-07-22_16-18-30
  done: true
  experiment_id: d9469b1fc58b41db88da5446dc2a3b23
  experiment_tag: 7_hidden=181,lr=0.0362,momentum=0.7903
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 12
  mean_accuracy: 0.9799666404724121
  node_ip: 127.0.0.1
  pid: 52021
  time_since_restore: 41.763158082962036
  time_this_iter_s: 1.0622038841247559
  time_total_s: 41.763158082962036
  timestamp: 1658503110
  timesteps_since_restore: 0
  training_iteration: 12
  trial_id: 55a9b_00007
  warmup_time: 0.0028028488159179688
  
Result for train_mnist_55a9b_00008:
  date: 2022-07-22_16-18-33
  done: false
  experiment_id: 8dbd22e6caed4fe39351dffa3ef14eac
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 8
  mean_accuracy: 0.9599000215530396
  node_ip: 127.0.0.1
  pid: 52007
  time_since_restore: 11.612935304641724
  time_this_iter_s: 0.6818761825561523
  time_total_s: 11.612935304641724
  timestamp: 1658503113
  timesteps_since_restore: 0
  training_iteration: 8
  trial_id: 55a9b_00008
  warmup_time: 0.005449056625366211
  
Result for train_mnist_55a9b_00009:
  date: 2022-07-22_16-18-34
  done: false
  experiment_id: c4f803baf65f4d4e9fd6abc85b2fd00c
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 9
  mean_accuracy: 0.9319833517074585
  node_ip: 127.0.0.1
  pid: 51984
  time_since_restore: 10.803268194198608
  time_this_iter_s: 0.606992244720459
  time_total_s: 10.803268194198608
  timestamp: 1658503114
  timesteps_since_restore: 0
  training_iteration: 9
  trial_id: 55a9b_00009
  warmup_time: 0.0028328895568847656
  
Result for train_mnist_55a9b_00008:
  date: 2022-07-22_16-18-36
  done: true
  experiment_id: 8dbd22e6caed4fe39351dffa3ef14eac
  experiment_tag: 8_hidden=142,lr=0.0324,momentum=0.6604
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 12
  mean_accuracy: 0.9693666696548462
  node_ip: 127.0.0.1
  pid: 52007
  time_since_restore: 14.152745008468628
  time_this_iter_s: 0.5980076789855957
  time_total_s: 14.152745008468628
  timestamp: 1658503116
  timesteps_since_restore: 0
  training_iteration: 12
  trial_id: 55a9b_00008
  warmup_time: 0.005449056625366211
  
Result for train_mnist_55a9b_00009:
  date: 2022-07-22_16-18-36
  done: true
  experiment_id: c4f803baf65f4d4e9fd6abc85b2fd00c
  experiment_tag: 9_hidden=97,lr=0.0245,momentum=0.1750
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 12
  mean_accuracy: 0.9406999945640564
  node_ip: 127.0.0.1
  pid: 51984
  time_since_restore: 12.640528202056885
  time_this_iter_s: 0.5808131694793701
  time_total_s: 12.640528202056885
  timestamp: 1658503116
  timesteps_since_restore: 0
  training_iteration: 12
  trial_id: 55a9b_00009
  warmup_time: 0.0028328895568847656
  
2022-07-22 16:18:36,803	INFO tune.py:738 -- Total run time: 95.98 seconds (95.03 seconds for the tuning loop).
Best hyperparameters found were:  {'threads': 2, 'lr': 0.09518133271957563, 'momentum': 0.8254987643140009, 'hidden': 258}

More Keras and TensorFlow Examples

  • Memory NN Example: Example of training a Memory NN on bAbI with Keras using PBT.

  • TensorFlow MNIST Example: Converts the Advanced TF2.0 MNIST example to use Tune with the Trainable. This uses tf.function. Original code from tensorflow: https://www.tensorflow.org/tutorials/quickstart/advanced

  • Keras Cifar10 Example: A contributed example of tuning a Keras model on CIFAR10 with the PopulationBasedTraining scheduler.