{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"id": "3b05af3b",
"metadata": {},
"source": [
"(tune-mnist-keras)=\n",
"\n",
"# Using Keras & TensorFlow with Tune\n",
"\n",
"```{image} /images/tf_keras_logo.jpeg\n",
":align: center\n",
":alt: Keras & TensorFlow Logo\n",
":height: 120px\n",
":target: https://keras.io\n",
"```\n",
"\n",
"```{contents}\n",
":backlinks: none\n",
":local: true\n",
"```\n",
"\n",
"## Example"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "19e3c389",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2022-07-22 16:16:58,114\tINFO services.py:1483 -- View the Ray dashboard at \u001b[1m\u001b[32mhttp://127.0.0.1:8269\u001b[39m\u001b[22m\n",
"2022-07-22 16:17:00,822\tWARNING function_trainable.py:619 -- \n"
]
},
{
"data": {
"text/html": [
"== Status ==
Current time: 2022-07-22 16:18:36 (running for 00:01:35.04)
Memory usage on this node: 9.0/16.0 GiB
Using AsyncHyperBand: num_stopped=0\n",
"Bracket: Iter 320.000: None | Iter 80.000: None | Iter 20.000: None
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.47 GiB heap, 0.0/2.0 GiB objects
Current best trial: 55a9b_00002 with mean_accuracy=0.9904166460037231 and parameters={'threads': 2, 'lr': 0.09518133271957563, 'momentum': 0.8254987643140009, 'hidden': 258}
Result logdir: /Users/kai/ray_results/exp
Number of trials: 10/10 (10 TERMINATED)
\n",
"\n",
"| Trial name | status | loc | hidden | lr | momentum | acc | iter | total time (s) |
|---|
\n",
"\n",
"\n",
"| train_mnist_55a9b_00000 | TERMINATED | 127.0.0.1:51968 | 276 | 0.0406397 | 0.817788 | 0.98455 | 12 | 78.3252 |
\n",
"| train_mnist_55a9b_00001 | TERMINATED | 127.0.0.1:51977 | 380 | 0.0873557 | 0.524634 | 0.983717 | 12 | 74.9888 |
\n",
"| train_mnist_55a9b_00002 | TERMINATED | 127.0.0.1:51984 | 258 | 0.0951813 | 0.825499 | 0.990417 | 11 | 64.1272 |
\n",
"| train_mnist_55a9b_00003 | TERMINATED | 127.0.0.1:51991 | 255 | 0.0971683 | 0.23161 | 0.977633 | 12 | 60.8475 |
\n",
"| train_mnist_55a9b_00004 | TERMINATED | 127.0.0.1:52000 | 303 | 0.00440117 | 0.325439 | 0.90775 | 12 | 55.5722 |
\n",
"| train_mnist_55a9b_00005 | TERMINATED | 127.0.0.1:52007 | 92 | 0.0651919 | 0.710183 | 0.974867 | 12 | 44.8092 |
\n",
"| train_mnist_55a9b_00006 | TERMINATED | 127.0.0.1:52016 | 211 | 0.0731116 | 0.127751 | 0.97025 | 12 | 42.1217 |
\n",
"| train_mnist_55a9b_00007 | TERMINATED | 127.0.0.1:52021 | 181 | 0.0362389 | 0.790345 | 0.979967 | 12 | 41.7632 |
\n",
"| train_mnist_55a9b_00008 | TERMINATED | 127.0.0.1:52007 | 142 | 0.0323741 | 0.660418 | 0.969367 | 12 | 14.1527 |
\n",
"| train_mnist_55a9b_00009 | TERMINATED | 127.0.0.1:51984 | 97 | 0.0244971 | 0.175045 | 0.9407 | 12 | 12.6405 |
\n",
"\n",
"
"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2022-07-22 16:17:01,834\tINFO plugin_schema_manager.py:52 -- Loading the default runtime env schemas: ['/Users/kai/coding/ray/python/ray/_private/runtime_env/../../runtime_env/schemas/working_dir_schema.json', '/Users/kai/coding/ray/python/ray/_private/runtime_env/../../runtime_env/schemas/pip_schema.json'].\n",
"\u001b[2m\u001b[36m(train_mnist pid=51968)\u001b[0m 2022-07-22 16:17:08.627419: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA\n",
"\u001b[2m\u001b[36m(train_mnist pid=51968)\u001b[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
"\u001b[2m\u001b[36m(train_mnist pid=51968)\u001b[0m /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/keras/optimizer_v2/optimizer_v2.py:356: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
"\u001b[2m\u001b[36m(train_mnist pid=51968)\u001b[0m \"The `lr` argument is deprecated, use `learning_rate` instead.\")\n",
"\u001b[2m\u001b[36m(train_mnist pid=51968)\u001b[0m 2022-07-22 16:17:08.947939: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)\n",
"\u001b[2m\u001b[36m(train_mnist pid=51977)\u001b[0m 2022-07-22 16:17:14.473677: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA\n",
"\u001b[2m\u001b[36m(train_mnist pid=51977)\u001b[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
"\u001b[2m\u001b[36m(train_mnist pid=51977)\u001b[0m /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/keras/optimizer_v2/optimizer_v2.py:356: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
"\u001b[2m\u001b[36m(train_mnist pid=51977)\u001b[0m \"The `lr` argument is deprecated, use `learning_rate` instead.\")\n",
"\u001b[2m\u001b[36m(train_mnist pid=51977)\u001b[0m 2022-07-22 16:17:14.635104: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)\n",
"\u001b[2m\u001b[36m(train_mnist pid=51984)\u001b[0m 2022-07-22 16:17:20.406624: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA\n",
"\u001b[2m\u001b[36m(train_mnist pid=51984)\u001b[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
"\u001b[2m\u001b[36m(train_mnist pid=51984)\u001b[0m /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/keras/optimizer_v2/optimizer_v2.py:356: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
"\u001b[2m\u001b[36m(train_mnist pid=51984)\u001b[0m \"The `lr` argument is deprecated, use `learning_rate` instead.\")\n",
"\u001b[2m\u001b[36m(train_mnist pid=51984)\u001b[0m 2022-07-22 16:17:20.681960: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)\n",
"\u001b[2m\u001b[36m(train_mnist pid=51991)\u001b[0m 2022-07-22 16:17:26.109460: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA\n",
"\u001b[2m\u001b[36m(train_mnist pid=51991)\u001b[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
"\u001b[2m\u001b[36m(train_mnist pid=51991)\u001b[0m /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/keras/optimizer_v2/optimizer_v2.py:356: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
"\u001b[2m\u001b[36m(train_mnist pid=51991)\u001b[0m \"The `lr` argument is deprecated, use `learning_rate` instead.\")\n",
"\u001b[2m\u001b[36m(train_mnist pid=51991)\u001b[0m 2022-07-22 16:17:26.303375: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)\n",
"\u001b[2m\u001b[36m(train_mnist pid=52000)\u001b[0m 2022-07-22 16:17:31.899252: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA\n",
"\u001b[2m\u001b[36m(train_mnist pid=52000)\u001b[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
"\u001b[2m\u001b[36m(train_mnist pid=52000)\u001b[0m /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/keras/optimizer_v2/optimizer_v2.py:356: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
"\u001b[2m\u001b[36m(train_mnist pid=52000)\u001b[0m \"The `lr` argument is deprecated, use `learning_rate` instead.\")\n",
"\u001b[2m\u001b[36m(train_mnist pid=52000)\u001b[0m 2022-07-22 16:17:32.300424: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)\n",
"\u001b[2m\u001b[36m(train_mnist pid=52007)\u001b[0m 2022-07-22 16:17:37.937471: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA\n",
"\u001b[2m\u001b[36m(train_mnist pid=52007)\u001b[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
"\u001b[2m\u001b[36m(train_mnist pid=52007)\u001b[0m /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/keras/optimizer_v2/optimizer_v2.py:356: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
"\u001b[2m\u001b[36m(train_mnist pid=52007)\u001b[0m \"The `lr` argument is deprecated, use `learning_rate` instead.\")\n",
"\u001b[2m\u001b[36m(train_mnist pid=52007)\u001b[0m 2022-07-22 16:17:38.263888: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)\n",
"\u001b[2m\u001b[36m(train_mnist pid=52016)\u001b[0m 2022-07-22 16:17:43.657379: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA\n",
"\u001b[2m\u001b[36m(train_mnist pid=52016)\u001b[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
"\u001b[2m\u001b[36m(train_mnist pid=52016)\u001b[0m /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/keras/optimizer_v2/optimizer_v2.py:356: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
"\u001b[2m\u001b[36m(train_mnist pid=52016)\u001b[0m \"The `lr` argument is deprecated, use `learning_rate` instead.\")\n",
"\u001b[2m\u001b[36m(train_mnist pid=52016)\u001b[0m 2022-07-22 16:17:43.828809: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for train_mnist_55a9b_00000:\n",
" date: 2022-07-22_16-17-10\n",
" done: false\n",
" experiment_id: 3659349c38c746cfb71b4db5eb9302a0\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" mean_accuracy: 0.8903833627700806\n",
" node_ip: 127.0.0.1\n",
" pid: 51968\n",
" time_since_restore: 2.439258098602295\n",
" time_this_iter_s: 2.439258098602295\n",
" time_total_s: 2.439258098602295\n",
" timestamp: 1658503030\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 55a9b_00000\n",
" warmup_time: 0.003445863723754883\n",
" \n",
"Result for train_mnist_55a9b_00004:\n",
" date: 2022-07-22_16-17-33\n",
" done: false\n",
" experiment_id: 6eb62b7cb38f442a867a9094f0664701\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" mean_accuracy: 0.6376166939735413\n",
" node_ip: 127.0.0.1\n",
" pid: 52000\n",
" time_since_restore: 2.4364511966705322\n",
" time_this_iter_s: 2.4364511966705322\n",
" time_total_s: 2.4364511966705322\n",
" timestamp: 1658503053\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 55a9b_00004\n",
" warmup_time: 0.0030939579010009766\n",
" \n",
"Result for train_mnist_55a9b_00006:\n",
" date: 2022-07-22_16-17-45\n",
" done: false\n",
" experiment_id: 9594405e38084311a891b48addd13f75\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" mean_accuracy: 0.8557000160217285\n",
" node_ip: 127.0.0.1\n",
" pid: 52016\n",
" time_since_restore: 1.8570480346679688\n",
" time_this_iter_s: 1.8570480346679688\n",
" time_total_s: 1.8570480346679688\n",
" timestamp: 1658503065\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 55a9b_00006\n",
" warmup_time: 0.003566741943359375\n",
" \n",
"Result for train_mnist_55a9b_00001:\n",
" date: 2022-07-22_16-17-15\n",
" done: false\n",
" experiment_id: fcdeb049f9614755a9b7c9420ca2ae5e\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" mean_accuracy: 0.8887666463851929\n",
" node_ip: 127.0.0.1\n",
" pid: 51977\n",
" time_since_restore: 1.9353628158569336\n",
" time_this_iter_s: 1.9353628158569336\n",
" time_total_s: 1.9353628158569336\n",
" timestamp: 1658503035\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 55a9b_00001\n",
" warmup_time: 0.0029449462890625\n",
" \n",
"Result for train_mnist_55a9b_00005:\n",
" date: 2022-07-22_16-17-39\n",
" done: false\n",
" experiment_id: 8dbd22e6caed4fe39351dffa3ef14eac\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" mean_accuracy: 0.8789666891098022\n",
" node_ip: 127.0.0.1\n",
" pid: 52007\n",
" time_since_restore: 2.3337321281433105\n",
" time_this_iter_s: 2.3337321281433105\n",
" time_total_s: 2.3337321281433105\n",
" timestamp: 1658503059\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 55a9b_00005\n",
" warmup_time: 0.005449056625366211\n",
" \n",
"Result for train_mnist_55a9b_00002:\n",
" date: 2022-07-22_16-17-21\n",
" done: false\n",
" experiment_id: c4f803baf65f4d4e9fd6abc85b2fd00c\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" mean_accuracy: 0.9112833142280579\n",
" node_ip: 127.0.0.1\n",
" pid: 51984\n",
" time_since_restore: 2.3220012187957764\n",
" time_this_iter_s: 2.3220012187957764\n",
" time_total_s: 2.3220012187957764\n",
" timestamp: 1658503041\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 55a9b_00002\n",
" warmup_time: 0.0028328895568847656\n",
" \n",
"Result for train_mnist_55a9b_00003:\n",
" date: 2022-07-22_16-17-27\n",
" done: false\n",
" experiment_id: 469478f02b4a43f5b44c40e59989ad39\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" mean_accuracy: 0.8743166923522949\n",
" node_ip: 127.0.0.1\n",
" pid: 51991\n",
" time_since_restore: 2.0278611183166504\n",
" time_this_iter_s: 2.0278611183166504\n",
" time_total_s: 2.0278611183166504\n",
" timestamp: 1658503047\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 55a9b_00003\n",
" warmup_time: 0.0033779144287109375\n",
" \n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\u001b[2m\u001b[36m(train_mnist pid=52021)\u001b[0m 2022-07-22 16:17:51.567914: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA\n",
"\u001b[2m\u001b[36m(train_mnist pid=52021)\u001b[0m To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
"\u001b[2m\u001b[36m(train_mnist pid=52021)\u001b[0m /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/keras/optimizer_v2/optimizer_v2.py:356: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
"\u001b[2m\u001b[36m(train_mnist pid=52021)\u001b[0m \"The `lr` argument is deprecated, use `learning_rate` instead.\")\n",
"\u001b[2m\u001b[36m(train_mnist pid=52021)\u001b[0m 2022-07-22 16:17:52.977183: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Result for train_mnist_55a9b_00005:\n",
" date: 2022-07-22_16-17-54\n",
" done: false\n",
" experiment_id: 8dbd22e6caed4fe39351dffa3ef14eac\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 3\n",
" mean_accuracy: 0.9490833282470703\n",
" node_ip: 127.0.0.1\n",
" pid: 52007\n",
" time_since_restore: 17.22033405303955\n",
" time_this_iter_s: 2.672102928161621\n",
" time_total_s: 17.22033405303955\n",
" timestamp: 1658503074\n",
" timesteps_since_restore: 0\n",
" training_iteration: 3\n",
" trial_id: 55a9b_00005\n",
" warmup_time: 0.005449056625366211\n",
" \n",
"Result for train_mnist_55a9b_00006:\n",
" date: 2022-07-22_16-17-54\n",
" done: false\n",
" experiment_id: 9594405e38084311a891b48addd13f75\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 3\n",
" mean_accuracy: 0.9327999949455261\n",
" node_ip: 127.0.0.1\n",
" pid: 52016\n",
" time_since_restore: 11.758372068405151\n",
" time_this_iter_s: 3.0426323413848877\n",
" time_total_s: 11.758372068405151\n",
" timestamp: 1658503074\n",
" timesteps_since_restore: 0\n",
" training_iteration: 3\n",
" trial_id: 55a9b_00006\n",
" warmup_time: 0.003566741943359375\n",
" \n",
"Result for train_mnist_55a9b_00003:\n",
" date: 2022-07-22_16-17-55\n",
" done: false\n",
" experiment_id: 469478f02b4a43f5b44c40e59989ad39\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 3\n",
" mean_accuracy: 0.9454166889190674\n",
" node_ip: 127.0.0.1\n",
" pid: 51991\n",
" time_since_restore: 29.733185052871704\n",
" time_this_iter_s: 3.0363340377807617\n",
" time_total_s: 29.733185052871704\n",
" timestamp: 1658503075\n",
" timesteps_since_restore: 0\n",
" training_iteration: 3\n",
" trial_id: 55a9b_00003\n",
" warmup_time: 0.0033779144287109375\n",
" \n",
"Result for train_mnist_55a9b_00000:\n",
" date: 2022-07-22_16-17-55\n",
" done: false\n",
" experiment_id: 3659349c38c746cfb71b4db5eb9302a0\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 3\n",
" mean_accuracy: 0.958216667175293\n",
" node_ip: 127.0.0.1\n",
" pid: 51968\n",
" time_since_restore: 47.272178173065186\n",
" time_this_iter_s: 3.2986061573028564\n",
" time_total_s: 47.272178173065186\n",
" timestamp: 1658503075\n",
" timesteps_since_restore: 0\n",
" training_iteration: 3\n",
" trial_id: 55a9b_00000\n",
" warmup_time: 0.003445863723754883\n",
" \n",
"Result for train_mnist_55a9b_00004:\n",
" date: 2022-07-22_16-17-55\n",
" done: false\n",
" experiment_id: 6eb62b7cb38f442a867a9094f0664701\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 3\n",
" mean_accuracy: 0.8524500131607056\n",
" node_ip: 127.0.0.1\n",
" pid: 52000\n",
" time_since_restore: 24.11396098136902\n",
" time_this_iter_s: 3.2331089973449707\n",
" time_total_s: 24.11396098136902\n",
" timestamp: 1658503075\n",
" timesteps_since_restore: 0\n",
" training_iteration: 3\n",
" trial_id: 55a9b_00004\n",
" warmup_time: 0.0030939579010009766\n",
" \n",
"Result for train_mnist_55a9b_00002:\n",
" date: 2022-07-22_16-17-55\n",
" done: false\n",
" experiment_id: c4f803baf65f4d4e9fd6abc85b2fd00c\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 3\n",
" mean_accuracy: 0.9695500135421753\n",
" node_ip: 127.0.0.1\n",
" pid: 51984\n",
" time_since_restore: 35.78592824935913\n",
" time_this_iter_s: 3.021165132522583\n",
" time_total_s: 35.78592824935913\n",
" timestamp: 1658503075\n",
" timesteps_since_restore: 0\n",
" training_iteration: 3\n",
" trial_id: 55a9b_00002\n",
" warmup_time: 0.0028328895568847656\n",
" \n",
"Result for train_mnist_55a9b_00001:\n",
" date: 2022-07-22_16-17-56\n",
" done: false\n",
" experiment_id: fcdeb049f9614755a9b7c9420ca2ae5e\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 3\n",
" mean_accuracy: 0.9560333490371704\n",
" node_ip: 127.0.0.1\n",
" pid: 51977\n",
" time_since_restore: 42.38909387588501\n",
" time_this_iter_s: 3.753290891647339\n",
" time_total_s: 42.38909387588501\n",
" timestamp: 1658503076\n",
" timesteps_since_restore: 0\n",
" training_iteration: 3\n",
" trial_id: 55a9b_00001\n",
" warmup_time: 0.0029449462890625\n",
" \n",
"Result for train_mnist_55a9b_00005:\n",
" date: 2022-07-22_16-18-00\n",
" done: false\n",
" experiment_id: 8dbd22e6caed4fe39351dffa3ef14eac\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 5\n",
" mean_accuracy: 0.9611166715621948\n",
" node_ip: 127.0.0.1\n",
" pid: 52007\n",
" time_since_restore: 23.303561210632324\n",
" time_this_iter_s: 2.933852195739746\n",
" time_total_s: 23.303561210632324\n",
" timestamp: 1658503080\n",
" timesteps_since_restore: 0\n",
" training_iteration: 5\n",
" trial_id: 55a9b_00005\n",
" warmup_time: 0.005449056625366211\n",
" \n",
"Result for train_mnist_55a9b_00007:\n",
" date: 2022-07-22_16-18-01\n",
" done: false\n",
" experiment_id: d9469b1fc58b41db88da5446dc2a3b23\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" mean_accuracy: 0.8797500133514404\n",
" node_ip: 127.0.0.1\n",
" pid: 52021\n",
" time_since_restore: 12.469872951507568\n",
" time_this_iter_s: 12.469872951507568\n",
" time_total_s: 12.469872951507568\n",
" timestamp: 1658503081\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 55a9b_00007\n",
" warmup_time: 0.0028028488159179688\n",
" \n",
"Result for train_mnist_55a9b_00006:\n",
" date: 2022-07-22_16-18-01\n",
" done: false\n",
" experiment_id: 9594405e38084311a891b48addd13f75\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 5\n",
" mean_accuracy: 0.9499499797821045\n",
" node_ip: 127.0.0.1\n",
" pid: 52016\n",
" time_since_restore: 18.780059814453125\n",
" time_this_iter_s: 3.3080599308013916\n",
" time_total_s: 18.780059814453125\n",
" timestamp: 1658503081\n",
" timesteps_since_restore: 0\n",
" training_iteration: 5\n",
" trial_id: 55a9b_00006\n",
" warmup_time: 0.003566741943359375\n",
" \n",
"Result for train_mnist_55a9b_00003:\n",
" date: 2022-07-22_16-18-02\n",
" done: false\n",
" experiment_id: 469478f02b4a43f5b44c40e59989ad39\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 5\n",
" mean_accuracy: 0.9601166844367981\n",
" node_ip: 127.0.0.1\n",
" pid: 51991\n",
" time_since_restore: 36.93912100791931\n",
" time_this_iter_s: 3.4057939052581787\n",
" time_total_s: 36.93912100791931\n",
" timestamp: 1658503082\n",
" timesteps_since_restore: 0\n",
" training_iteration: 5\n",
" trial_id: 55a9b_00003\n",
" warmup_time: 0.0033779144287109375\n",
" \n",
"Result for train_mnist_55a9b_00000:\n",
" date: 2022-07-22_16-18-02\n",
" done: false\n",
" experiment_id: 3659349c38c746cfb71b4db5eb9302a0\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 5\n",
" mean_accuracy: 0.970466673374176\n",
" node_ip: 127.0.0.1\n",
" pid: 51968\n",
" time_since_restore: 54.49850010871887\n",
" time_this_iter_s: 3.4417831897735596\n",
" time_total_s: 54.49850010871887\n",
" timestamp: 1658503082\n",
" timesteps_since_restore: 0\n",
" training_iteration: 5\n",
" trial_id: 55a9b_00000\n",
" warmup_time: 0.003445863723754883\n",
" \n",
"Result for train_mnist_55a9b_00004:\n",
" date: 2022-07-22_16-18-02\n",
" done: false\n",
" experiment_id: 6eb62b7cb38f442a867a9094f0664701\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 5\n",
" mean_accuracy: 0.8777499794960022\n",
" node_ip: 127.0.0.1\n",
" pid: 52000\n",
" time_since_restore: 31.513713121414185\n",
" time_this_iter_s: 3.506195068359375\n",
" time_total_s: 31.513713121414185\n",
" timestamp: 1658503082\n",
" timesteps_since_restore: 0\n",
" training_iteration: 5\n",
" trial_id: 55a9b_00004\n",
" warmup_time: 0.0030939579010009766\n",
" \n",
"Result for train_mnist_55a9b_00002:\n",
" date: 2022-07-22_16-18-02\n",
" done: false\n",
" experiment_id: c4f803baf65f4d4e9fd6abc85b2fd00c\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 5\n",
" mean_accuracy: 0.979283332824707\n",
" node_ip: 127.0.0.1\n",
" pid: 51984\n",
" time_since_restore: 43.266417026519775\n",
" time_this_iter_s: 3.3383469581604004\n",
" time_total_s: 43.266417026519775\n",
" timestamp: 1658503082\n",
" timesteps_since_restore: 0\n",
" training_iteration: 5\n",
" trial_id: 55a9b_00002\n",
" warmup_time: 0.0028328895568847656\n",
" \n",
"Result for train_mnist_55a9b_00001:\n",
" date: 2022-07-22_16-18-04\n",
" done: false\n",
" experiment_id: fcdeb049f9614755a9b7c9420ca2ae5e\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 5\n",
" mean_accuracy: 0.9692999720573425\n",
" node_ip: 127.0.0.1\n",
" pid: 51977\n",
" time_since_restore: 50.620792865753174\n",
" time_this_iter_s: 4.001068115234375\n",
" time_total_s: 50.620792865753174\n",
" timestamp: 1658503084\n",
" timesteps_since_restore: 0\n",
" training_iteration: 5\n",
" trial_id: 55a9b_00001\n",
" warmup_time: 0.0029449462890625\n",
" \n",
"Result for train_mnist_55a9b_00005:\n",
" date: 2022-07-22_16-18-06\n",
" done: false\n",
" experiment_id: 8dbd22e6caed4fe39351dffa3ef14eac\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 7\n",
" mean_accuracy: 0.96711665391922\n",
" node_ip: 127.0.0.1\n",
" pid: 52007\n",
" time_since_restore: 29.40476107597351\n",
" time_this_iter_s: 2.976076126098633\n",
" time_total_s: 29.40476107597351\n",
" timestamp: 1658503086\n",
" timesteps_since_restore: 0\n",
" training_iteration: 7\n",
" trial_id: 55a9b_00005\n",
" warmup_time: 0.005449056625366211\n",
" \n",
"Result for train_mnist_55a9b_00007:\n",
" date: 2022-07-22_16-18-07\n",
" done: false\n",
" experiment_id: d9469b1fc58b41db88da5446dc2a3b23\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 3\n",
" mean_accuracy: 0.951033353805542\n",
" node_ip: 127.0.0.1\n",
" pid: 52021\n",
" time_since_restore: 18.96213722229004\n",
" time_this_iter_s: 3.252371311187744\n",
" time_total_s: 18.96213722229004\n",
" timestamp: 1658503087\n",
" timesteps_since_restore: 0\n",
" training_iteration: 3\n",
" trial_id: 55a9b_00007\n",
" warmup_time: 0.0028028488159179688\n",
" \n",
"Result for train_mnist_55a9b_00006:\n",
" date: 2022-07-22_16-18-08\n",
" done: false\n",
" experiment_id: 9594405e38084311a891b48addd13f75\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 7\n",
" mean_accuracy: 0.9584500193595886\n",
" node_ip: 127.0.0.1\n",
" pid: 52016\n",
" time_since_restore: 25.336583852767944\n",
" time_this_iter_s: 3.311979055404663\n",
" time_total_s: 25.336583852767944\n",
" timestamp: 1658503088\n",
" timesteps_since_restore: 0\n",
" training_iteration: 7\n",
" trial_id: 55a9b_00006\n",
" warmup_time: 0.003566741943359375\n",
" \n",
"Result for train_mnist_55a9b_00003:\n",
" date: 2022-07-22_16-18-09\n",
" done: false\n",
" experiment_id: 469478f02b4a43f5b44c40e59989ad39\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 7\n",
" mean_accuracy: 0.9675499796867371\n",
" node_ip: 127.0.0.1\n",
" pid: 51991\n",
" time_since_restore: 43.7107310295105\n",
" time_this_iter_s: 3.3927559852600098\n",
" time_total_s: 43.7107310295105\n",
" timestamp: 1658503089\n",
" timesteps_since_restore: 0\n",
" training_iteration: 7\n",
" trial_id: 55a9b_00003\n",
" warmup_time: 0.0033779144287109375\n",
" \n",
"Result for train_mnist_55a9b_00000:\n",
" date: 2022-07-22_16-18-09\n",
" done: false\n",
" experiment_id: 3659349c38c746cfb71b4db5eb9302a0\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 7\n",
" mean_accuracy: 0.9763000011444092\n",
" node_ip: 127.0.0.1\n",
" pid: 51968\n",
" time_since_restore: 61.30248522758484\n",
" time_this_iter_s: 3.4063682556152344\n",
" time_total_s: 61.30248522758484\n",
" timestamp: 1658503089\n",
" timesteps_since_restore: 0\n",
" training_iteration: 7\n",
" trial_id: 55a9b_00000\n",
" warmup_time: 0.003445863723754883\n",
" \n",
"Result for train_mnist_55a9b_00002:\n",
" date: 2022-07-22_16-18-09\n",
" done: false\n",
" experiment_id: c4f803baf65f4d4e9fd6abc85b2fd00c\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 7\n",
" mean_accuracy: 0.9840666651725769\n",
" node_ip: 127.0.0.1\n",
" pid: 51984\n",
" time_since_restore: 50.212465047836304\n",
" time_this_iter_s: 3.43766188621521\n",
" time_total_s: 50.212465047836304\n",
" timestamp: 1658503089\n",
" timesteps_since_restore: 0\n",
" training_iteration: 7\n",
" trial_id: 55a9b_00002\n",
" warmup_time: 0.0028328895568847656\n",
" \n",
"Result for train_mnist_55a9b_00004:\n",
" date: 2022-07-22_16-18-09\n",
" done: false\n",
" experiment_id: 6eb62b7cb38f442a867a9094f0664701\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 7\n",
" mean_accuracy: 0.8899999856948853\n",
" node_ip: 127.0.0.1\n",
" pid: 52000\n",
" time_since_restore: 38.63890194892883\n",
" time_this_iter_s: 3.5783908367156982\n",
" time_total_s: 38.63890194892883\n",
" timestamp: 1658503089\n",
" timesteps_since_restore: 0\n",
" training_iteration: 7\n",
" trial_id: 55a9b_00004\n",
" warmup_time: 0.0030939579010009766\n",
" \n",
"Result for train_mnist_55a9b_00005:\n",
" date: 2022-07-22_16-18-12\n",
" done: false\n",
" experiment_id: 8dbd22e6caed4fe39351dffa3ef14eac\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 9\n",
" mean_accuracy: 0.9712333083152771\n",
" node_ip: 127.0.0.1\n",
" pid: 52007\n",
" time_since_restore: 35.3185760974884\n",
" time_this_iter_s: 3.0241990089416504\n",
" time_total_s: 35.3185760974884\n",
" timestamp: 1658503092\n",
" timesteps_since_restore: 0\n",
" training_iteration: 9\n",
" trial_id: 55a9b_00005\n",
" warmup_time: 0.005449056625366211\n",
" \n",
"Result for train_mnist_55a9b_00001:\n",
" date: 2022-07-22_16-18-12\n",
" done: false\n",
" experiment_id: fcdeb049f9614755a9b7c9420ca2ae5e\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 7\n",
" mean_accuracy: 0.9755333065986633\n",
" node_ip: 127.0.0.1\n",
" pid: 51977\n",
" time_since_restore: 58.57745599746704\n",
" time_this_iter_s: 3.936232089996338\n",
" time_total_s: 58.57745599746704\n",
" timestamp: 1658503092\n",
" timesteps_since_restore: 0\n",
" training_iteration: 7\n",
" trial_id: 55a9b_00001\n",
" warmup_time: 0.0029449462890625\n",
" \n",
"Result for train_mnist_55a9b_00007:\n",
" date: 2022-07-22_16-18-14\n",
" done: false\n",
" experiment_id: d9469b1fc58b41db88da5446dc2a3b23\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 5\n",
" mean_accuracy: 0.9648333191871643\n",
" node_ip: 127.0.0.1\n",
" pid: 52021\n",
" time_since_restore: 25.25843620300293\n",
" time_this_iter_s: 3.094501256942749\n",
" time_total_s: 25.25843620300293\n",
" timestamp: 1658503094\n",
" timesteps_since_restore: 0\n",
" training_iteration: 5\n",
" trial_id: 55a9b_00007\n",
" warmup_time: 0.0028028488159179688\n",
" \n",
"Result for train_mnist_55a9b_00006:\n",
" date: 2022-07-22_16-18-15\n",
" done: false\n",
" experiment_id: 9594405e38084311a891b48addd13f75\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 9\n",
" mean_accuracy: 0.9646666646003723\n",
" node_ip: 127.0.0.1\n",
" pid: 52016\n",
" time_since_restore: 32.048911809921265\n",
" time_this_iter_s: 3.315690755844116\n",
" time_total_s: 32.048911809921265\n",
" timestamp: 1658503095\n",
" timesteps_since_restore: 0\n",
" training_iteration: 9\n",
" trial_id: 55a9b_00006\n",
" warmup_time: 0.003566741943359375\n",
" \n",
"Result for train_mnist_55a9b_00003:\n",
" date: 2022-07-22_16-18-15\n",
" done: false\n",
" experiment_id: 469478f02b4a43f5b44c40e59989ad39\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 9\n",
" mean_accuracy: 0.9729499816894531\n",
" node_ip: 127.0.0.1\n",
" pid: 51991\n",
" time_since_restore: 50.50909209251404\n",
" time_this_iter_s: 3.4110782146453857\n",
" time_total_s: 50.50909209251404\n",
" timestamp: 1658503095\n",
" timesteps_since_restore: 0\n",
" training_iteration: 9\n",
" trial_id: 55a9b_00003\n",
" warmup_time: 0.0033779144287109375\n",
" \n",
"Result for train_mnist_55a9b_00000:\n",
" date: 2022-07-22_16-18-16\n",
" done: false\n",
" experiment_id: 3659349c38c746cfb71b4db5eb9302a0\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 9\n",
" mean_accuracy: 0.9807666540145874\n",
" node_ip: 127.0.0.1\n",
" pid: 51968\n",
" time_since_restore: 68.26757216453552\n",
" time_this_iter_s: 3.4475879669189453\n",
" time_total_s: 68.26757216453552\n",
" timestamp: 1658503096\n",
" timesteps_since_restore: 0\n",
" training_iteration: 9\n",
" trial_id: 55a9b_00000\n",
" warmup_time: 0.003445863723754883\n",
" \n",
"Result for train_mnist_55a9b_00002:\n",
" date: 2022-07-22_16-18-16\n",
" done: false\n",
" experiment_id: c4f803baf65f4d4e9fd6abc85b2fd00c\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 9\n",
" mean_accuracy: 0.9872999787330627\n",
" node_ip: 127.0.0.1\n",
" pid: 51984\n",
" time_since_restore: 57.01431703567505\n",
" time_this_iter_s: 3.3804008960723877\n",
" time_total_s: 57.01431703567505\n",
" timestamp: 1658503096\n",
" timesteps_since_restore: 0\n",
" training_iteration: 9\n",
" trial_id: 55a9b_00002\n",
" warmup_time: 0.0028328895568847656\n",
" \n",
"Result for train_mnist_55a9b_00004:\n",
" date: 2022-07-22_16-18-16\n",
" done: false\n",
" experiment_id: 6eb62b7cb38f442a867a9094f0664701\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 9\n",
" mean_accuracy: 0.8989166617393494\n",
" node_ip: 127.0.0.1\n",
" pid: 52000\n",
" time_since_restore: 45.67929005622864\n",
" time_this_iter_s: 3.4561610221862793\n",
" time_total_s: 45.67929005622864\n",
" timestamp: 1658503096\n",
" timesteps_since_restore: 0\n",
" training_iteration: 9\n",
" trial_id: 55a9b_00004\n",
" warmup_time: 0.0030939579010009766\n",
" \n",
"Result for train_mnist_55a9b_00005:\n",
" date: 2022-07-22_16-18-18\n",
" done: false\n",
" experiment_id: 8dbd22e6caed4fe39351dffa3ef14eac\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 11\n",
" mean_accuracy: 0.9744333624839783\n",
" node_ip: 127.0.0.1\n",
" pid: 52007\n",
" time_since_restore: 41.49077916145325\n",
" time_this_iter_s: 3.172250270843506\n",
" time_total_s: 41.49077916145325\n",
" timestamp: 1658503098\n",
" timesteps_since_restore: 0\n",
" training_iteration: 11\n",
" trial_id: 55a9b_00005\n",
" warmup_time: 0.005449056625366211\n",
" \n",
"Result for train_mnist_55a9b_00001:\n",
" date: 2022-07-22_16-18-20\n",
" done: false\n",
" experiment_id: fcdeb049f9614755a9b7c9420ca2ae5e\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 9\n",
" mean_accuracy: 0.9806166887283325\n",
" node_ip: 127.0.0.1\n",
" pid: 51977\n",
" time_since_restore: 66.64132380485535\n",
" time_this_iter_s: 4.0674309730529785\n",
" time_total_s: 66.64132380485535\n",
" timestamp: 1658503100\n",
" timesteps_since_restore: 0\n",
" training_iteration: 9\n",
" trial_id: 55a9b_00001\n",
" warmup_time: 0.0029449462890625\n",
" \n",
"Result for train_mnist_55a9b_00007:\n",
" date: 2022-07-22_16-18-20\n",
" done: false\n",
" experiment_id: d9469b1fc58b41db88da5446dc2a3b23\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 7\n",
" mean_accuracy: 0.970716655254364\n",
" node_ip: 127.0.0.1\n",
" pid: 52021\n",
" time_since_restore: 31.897236108779907\n",
" time_this_iter_s: 3.3691420555114746\n",
" time_total_s: 31.897236108779907\n",
" timestamp: 1658503100\n",
" timesteps_since_restore: 0\n",
" training_iteration: 7\n",
" trial_id: 55a9b_00007\n",
" warmup_time: 0.0028028488159179688\n",
" \n",
"Result for train_mnist_55a9b_00005:\n",
" date: 2022-07-22_16-18-21\n",
" done: true\n",
" experiment_id: 8dbd22e6caed4fe39351dffa3ef14eac\n",
" experiment_tag: 5_hidden=92,lr=0.0652,momentum=0.7102\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 12\n",
" mean_accuracy: 0.9748666882514954\n",
" node_ip: 127.0.0.1\n",
" pid: 52007\n",
" time_since_restore: 44.80922222137451\n",
" time_this_iter_s: 3.3184430599212646\n",
" time_total_s: 44.80922222137451\n",
" timestamp: 1658503101\n",
" timesteps_since_restore: 0\n",
" training_iteration: 12\n",
" trial_id: 55a9b_00005\n",
" warmup_time: 0.005449056625366211\n",
" \n",
"Result for train_mnist_55a9b_00006:\n",
" date: 2022-07-22_16-18-22\n",
" done: false\n",
" experiment_id: 9594405e38084311a891b48addd13f75\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 11\n",
" mean_accuracy: 0.9679166674613953\n",
" node_ip: 127.0.0.1\n",
" pid: 52016\n",
" time_since_restore: 39.08963179588318\n",
" time_this_iter_s: 3.4860758781433105\n",
" time_total_s: 39.08963179588318\n",
" timestamp: 1658503102\n",
" timesteps_since_restore: 0\n",
" training_iteration: 11\n",
" trial_id: 55a9b_00006\n",
" warmup_time: 0.003566741943359375\n",
" \n",
"Result for train_mnist_55a9b_00003:\n",
" date: 2022-07-22_16-18-23\n",
" done: false\n",
" experiment_id: 469478f02b4a43f5b44c40e59989ad39\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 11\n",
" mean_accuracy: 0.9771833419799805\n",
" node_ip: 127.0.0.1\n",
" pid: 51991\n",
" time_since_restore: 57.6213219165802\n",
" time_this_iter_s: 3.4615819454193115\n",
" time_total_s: 57.6213219165802\n",
" timestamp: 1658503103\n",
" timesteps_since_restore: 0\n",
" training_iteration: 11\n",
" trial_id: 55a9b_00003\n",
" warmup_time: 0.0033779144287109375\n",
" \n",
"Result for train_mnist_55a9b_00000:\n",
" date: 2022-07-22_16-18-23\n",
" done: false\n",
" experiment_id: 3659349c38c746cfb71b4db5eb9302a0\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 11\n",
" mean_accuracy: 0.98416668176651\n",
" node_ip: 127.0.0.1\n",
" pid: 51968\n",
" time_since_restore: 75.32713007926941\n",
" time_this_iter_s: 3.443808078765869\n",
" time_total_s: 75.32713007926941\n",
" timestamp: 1658503103\n",
" timesteps_since_restore: 0\n",
" training_iteration: 11\n",
" trial_id: 55a9b_00000\n",
" warmup_time: 0.003445863723754883\n",
" \n",
"Result for train_mnist_55a9b_00002:\n",
" date: 2022-07-22_16-18-23\n",
" done: true\n",
" experiment_id: c4f803baf65f4d4e9fd6abc85b2fd00c\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 11\n",
" mean_accuracy: 0.9904166460037231\n",
" node_ip: 127.0.0.1\n",
" pid: 51984\n",
" time_since_restore: 64.12720203399658\n",
" time_this_iter_s: 3.508151054382324\n",
" time_total_s: 64.12720203399658\n",
" timestamp: 1658503103\n",
" timesteps_since_restore: 0\n",
" training_iteration: 11\n",
" trial_id: 55a9b_00002\n",
" warmup_time: 0.0028328895568847656\n",
" \n",
"Result for train_mnist_55a9b_00004:\n",
" date: 2022-07-22_16-18-23\n",
" done: false\n",
" experiment_id: 6eb62b7cb38f442a867a9094f0664701\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 11\n",
" mean_accuracy: 0.9052166938781738\n",
" node_ip: 127.0.0.1\n",
" pid: 52000\n",
" time_since_restore: 52.687995195388794\n",
" time_this_iter_s: 3.420351982116699\n",
" time_total_s: 52.687995195388794\n",
" timestamp: 1658503103\n",
" timesteps_since_restore: 0\n",
" training_iteration: 11\n",
" trial_id: 55a9b_00004\n",
" warmup_time: 0.0030939579010009766\n",
" \n",
"Result for train_mnist_55a9b_00006:\n",
" date: 2022-07-22_16-18-25\n",
" done: true\n",
" experiment_id: 9594405e38084311a891b48addd13f75\n",
" experiment_tag: 6_hidden=211,lr=0.0731,momentum=0.1278\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 12\n",
" mean_accuracy: 0.9702500104904175\n",
" node_ip: 127.0.0.1\n",
" pid: 52016\n",
" time_since_restore: 42.1216938495636\n",
" time_this_iter_s: 3.03206205368042\n",
" time_total_s: 42.1216938495636\n",
" timestamp: 1658503105\n",
" timesteps_since_restore: 0\n",
" training_iteration: 12\n",
" trial_id: 55a9b_00006\n",
" warmup_time: 0.003566741943359375\n",
" \n",
"Result for train_mnist_55a9b_00003:\n",
" date: 2022-07-22_16-18-26\n",
" done: true\n",
" experiment_id: 469478f02b4a43f5b44c40e59989ad39\n",
" experiment_tag: 3_hidden=255,lr=0.0972,momentum=0.2316\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 12\n",
" mean_accuracy: 0.9776333570480347\n",
" node_ip: 127.0.0.1\n",
" pid: 51991\n",
" time_since_restore: 60.8474760055542\n",
" time_this_iter_s: 3.226154088973999\n",
" time_total_s: 60.8474760055542\n",
" timestamp: 1658503106\n",
" timesteps_since_restore: 0\n",
" training_iteration: 12\n",
" trial_id: 55a9b_00003\n",
" warmup_time: 0.0033779144287109375\n",
" \n",
"Result for train_mnist_55a9b_00000:\n",
" date: 2022-07-22_16-18-26\n",
" done: true\n",
" experiment_id: 3659349c38c746cfb71b4db5eb9302a0\n",
" experiment_tag: 0_hidden=276,lr=0.0406,momentum=0.8178\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 12\n",
" mean_accuracy: 0.9845499992370605\n",
" node_ip: 127.0.0.1\n",
" pid: 51968\n",
" time_since_restore: 78.32520508766174\n",
" time_this_iter_s: 2.998075008392334\n",
" time_total_s: 78.32520508766174\n",
" timestamp: 1658503106\n",
" timesteps_since_restore: 0\n",
" training_iteration: 12\n",
" trial_id: 55a9b_00000\n",
" warmup_time: 0.003445863723754883\n",
" \n",
"Result for train_mnist_55a9b_00007:\n",
" date: 2022-07-22_16-18-26\n",
" done: false\n",
" experiment_id: d9469b1fc58b41db88da5446dc2a3b23\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 9\n",
" mean_accuracy: 0.9751333594322205\n",
" node_ip: 127.0.0.1\n",
" pid: 52021\n",
" time_since_restore: 37.76195311546326\n",
" time_this_iter_s: 2.7159180641174316\n",
" time_total_s: 37.76195311546326\n",
" timestamp: 1658503106\n",
" timesteps_since_restore: 0\n",
" training_iteration: 9\n",
" trial_id: 55a9b_00007\n",
" warmup_time: 0.0028028488159179688\n",
" \n",
"Result for train_mnist_55a9b_00004:\n",
" date: 2022-07-22_16-18-26\n",
" done: true\n",
" experiment_id: 6eb62b7cb38f442a867a9094f0664701\n",
" experiment_tag: 4_hidden=303,lr=0.0044,momentum=0.3254\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 12\n",
" mean_accuracy: 0.9077500104904175\n",
" node_ip: 127.0.0.1\n",
" pid: 52000\n",
" time_since_restore: 55.57219409942627\n",
" time_this_iter_s: 2.8841989040374756\n",
" time_total_s: 55.57219409942627\n",
" timestamp: 1658503106\n",
" timesteps_since_restore: 0\n",
" training_iteration: 12\n",
" trial_id: 55a9b_00004\n",
" warmup_time: 0.0030939579010009766\n",
" \n",
"Result for train_mnist_55a9b_00001:\n",
" date: 2022-07-22_16-18-27\n",
" done: false\n",
" experiment_id: fcdeb049f9614755a9b7c9420ca2ae5e\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 11\n",
" mean_accuracy: 0.9830166697502136\n",
" node_ip: 127.0.0.1\n",
" pid: 51977\n",
" time_since_restore: 73.19760584831238\n",
" time_this_iter_s: 2.7281620502471924\n",
" time_total_s: 73.19760584831238\n",
" timestamp: 1658503107\n",
" timesteps_since_restore: 0\n",
" training_iteration: 11\n",
" trial_id: 55a9b_00001\n",
" warmup_time: 0.0029449462890625\n",
" \n",
"Result for train_mnist_55a9b_00008:\n",
" date: 2022-07-22_16-18-28\n",
" done: false\n",
" experiment_id: 8dbd22e6caed4fe39351dffa3ef14eac\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" mean_accuracy: 0.8477166891098022\n",
" node_ip: 127.0.0.1\n",
" pid: 52007\n",
" time_since_restore: 6.2436230182647705\n",
" time_this_iter_s: 6.2436230182647705\n",
" time_total_s: 6.2436230182647705\n",
" timestamp: 1658503108\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 55a9b_00008\n",
" warmup_time: 0.005449056625366211\n",
" \n",
"Result for train_mnist_55a9b_00001:\n",
" date: 2022-07-22_16-18-28\n",
" done: true\n",
" experiment_id: fcdeb049f9614755a9b7c9420ca2ae5e\n",
" experiment_tag: 1_hidden=380,lr=0.0874,momentum=0.5246\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 12\n",
" mean_accuracy: 0.9837166666984558\n",
" node_ip: 127.0.0.1\n",
" pid: 51977\n",
" time_since_restore: 74.98881888389587\n",
" time_this_iter_s: 1.791213035583496\n",
" time_total_s: 74.98881888389587\n",
" timestamp: 1658503108\n",
" timesteps_since_restore: 0\n",
" training_iteration: 12\n",
" trial_id: 55a9b_00001\n",
" warmup_time: 0.0029449462890625\n",
" \n",
"Result for train_mnist_55a9b_00009:\n",
" date: 2022-07-22_16-18-29\n",
" done: false\n",
" experiment_id: c4f803baf65f4d4e9fd6abc85b2fd00c\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 1\n",
" mean_accuracy: 0.7675999999046326\n",
" node_ip: 127.0.0.1\n",
" pid: 51984\n",
" time_since_restore: 5.303471088409424\n",
" time_this_iter_s: 5.303471088409424\n",
" time_total_s: 5.303471088409424\n",
" timestamp: 1658503109\n",
" timesteps_since_restore: 0\n",
" training_iteration: 1\n",
" trial_id: 55a9b_00009\n",
" warmup_time: 0.0028328895568847656\n",
" \n",
"Result for train_mnist_55a9b_00007:\n",
" date: 2022-07-22_16-18-30\n",
" done: true\n",
" experiment_id: d9469b1fc58b41db88da5446dc2a3b23\n",
" experiment_tag: 7_hidden=181,lr=0.0362,momentum=0.7903\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 12\n",
" mean_accuracy: 0.9799666404724121\n",
" node_ip: 127.0.0.1\n",
" pid: 52021\n",
" time_since_restore: 41.763158082962036\n",
" time_this_iter_s: 1.0622038841247559\n",
" time_total_s: 41.763158082962036\n",
" timestamp: 1658503110\n",
" timesteps_since_restore: 0\n",
" training_iteration: 12\n",
" trial_id: 55a9b_00007\n",
" warmup_time: 0.0028028488159179688\n",
" \n",
"Result for train_mnist_55a9b_00008:\n",
" date: 2022-07-22_16-18-33\n",
" done: false\n",
" experiment_id: 8dbd22e6caed4fe39351dffa3ef14eac\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 8\n",
" mean_accuracy: 0.9599000215530396\n",
" node_ip: 127.0.0.1\n",
" pid: 52007\n",
" time_since_restore: 11.612935304641724\n",
" time_this_iter_s: 0.6818761825561523\n",
" time_total_s: 11.612935304641724\n",
" timestamp: 1658503113\n",
" timesteps_since_restore: 0\n",
" training_iteration: 8\n",
" trial_id: 55a9b_00008\n",
" warmup_time: 0.005449056625366211\n",
" \n",
"Result for train_mnist_55a9b_00009:\n",
" date: 2022-07-22_16-18-34\n",
" done: false\n",
" experiment_id: c4f803baf65f4d4e9fd6abc85b2fd00c\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 9\n",
" mean_accuracy: 0.9319833517074585\n",
" node_ip: 127.0.0.1\n",
" pid: 51984\n",
" time_since_restore: 10.803268194198608\n",
" time_this_iter_s: 0.606992244720459\n",
" time_total_s: 10.803268194198608\n",
" timestamp: 1658503114\n",
" timesteps_since_restore: 0\n",
" training_iteration: 9\n",
" trial_id: 55a9b_00009\n",
" warmup_time: 0.0028328895568847656\n",
" \n",
"Result for train_mnist_55a9b_00008:\n",
" date: 2022-07-22_16-18-36\n",
" done: true\n",
" experiment_id: 8dbd22e6caed4fe39351dffa3ef14eac\n",
" experiment_tag: 8_hidden=142,lr=0.0324,momentum=0.6604\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 12\n",
" mean_accuracy: 0.9693666696548462\n",
" node_ip: 127.0.0.1\n",
" pid: 52007\n",
" time_since_restore: 14.152745008468628\n",
" time_this_iter_s: 0.5980076789855957\n",
" time_total_s: 14.152745008468628\n",
" timestamp: 1658503116\n",
" timesteps_since_restore: 0\n",
" training_iteration: 12\n",
" trial_id: 55a9b_00008\n",
" warmup_time: 0.005449056625366211\n",
" \n",
"Result for train_mnist_55a9b_00009:\n",
" date: 2022-07-22_16-18-36\n",
" done: true\n",
" experiment_id: c4f803baf65f4d4e9fd6abc85b2fd00c\n",
" experiment_tag: 9_hidden=97,lr=0.0245,momentum=0.1750\n",
" hostname: Kais-MacBook-Pro.local\n",
" iterations_since_restore: 12\n",
" mean_accuracy: 0.9406999945640564\n",
" node_ip: 127.0.0.1\n",
" pid: 51984\n",
" time_since_restore: 12.640528202056885\n",
" time_this_iter_s: 0.5808131694793701\n",
" time_total_s: 12.640528202056885\n",
" timestamp: 1658503116\n",
" timesteps_since_restore: 0\n",
" training_iteration: 12\n",
" trial_id: 55a9b_00009\n",
" warmup_time: 0.0028328895568847656\n",
" \n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"2022-07-22 16:18:36,803\tINFO tune.py:738 -- Total run time: 95.98 seconds (95.03 seconds for the tuning loop).\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Best hyperparameters found were: {'threads': 2, 'lr': 0.09518133271957563, 'momentum': 0.8254987643140009, 'hidden': 258}\n"
]
}
],
"source": [
"import argparse\n",
"import os\n",
"\n",
"from filelock import FileLock\n",
"from tensorflow.keras.datasets import mnist\n",
"\n",
"import ray\n",
"from ray import train, tune\n",
"from ray.tune.schedulers import AsyncHyperBandScheduler\n",
"from ray.air.integrations.keras import ReportCheckpointCallback\n",
"\n",
"\n",
"def train_mnist(config):\n",
" # https://github.com/tensorflow/tensorflow/issues/32159\n",
" import tensorflow as tf\n",
"\n",
" batch_size = 128\n",
" num_classes = 10\n",
" epochs = 12\n",
"\n",
" with FileLock(os.path.expanduser(\"~/.data.lock\")):\n",
" (x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
" x_train, x_test = x_train / 255.0, x_test / 255.0\n",
" model = tf.keras.models.Sequential(\n",
" [\n",
" tf.keras.layers.Flatten(input_shape=(28, 28)),\n",
" tf.keras.layers.Dense(config[\"hidden\"], activation=\"relu\"),\n",
" tf.keras.layers.Dropout(0.2),\n",
" tf.keras.layers.Dense(num_classes, activation=\"softmax\"),\n",
" ]\n",
" )\n",
"\n",
" model.compile(\n",
" loss=\"sparse_categorical_crossentropy\",\n",
" optimizer=tf.keras.optimizers.SGD(lr=config[\"lr\"], momentum=config[\"momentum\"]),\n",
" metrics=[\"accuracy\"],\n",
" )\n",
"\n",
" model.fit(\n",
" x_train,\n",
" y_train,\n",
" batch_size=batch_size,\n",
" epochs=epochs,\n",
" verbose=0,\n",
" validation_data=(x_test, y_test),\n",
" callbacks=[ReportCheckpointCallback(metrics={\"mean_accuracy\": \"accuracy\"})],\n",
" )\n",
"\n",
"\n",
"def tune_mnist():\n",
" sched = AsyncHyperBandScheduler(\n",
" time_attr=\"training_iteration\", max_t=400, grace_period=20\n",
" )\n",
"\n",
" tuner = tune.Tuner(\n",
" tune.with_resources(train_mnist, resources={\"cpu\": 2, \"gpu\": 0}),\n",
" tune_config=tune.TuneConfig(\n",
" metric=\"mean_accuracy\",\n",
" mode=\"max\",\n",
" scheduler=sched,\n",
" num_samples=10,\n",
" ),\n",
" run_config=train.RunConfig(\n",
" name=\"exp\",\n",
" stop={\"mean_accuracy\": 0.99},\n",
" ),\n",
" param_space={\n",
" \"threads\": 2,\n",
" \"lr\": tune.uniform(0.001, 0.1),\n",
" \"momentum\": tune.uniform(0.1, 0.9),\n",
" \"hidden\": tune.randint(32, 512),\n",
" },\n",
" )\n",
" results = tuner.fit()\n",
"\n",
" print(\"Best hyperparameters found were: \", results.get_best_result().config)\n",
"\n",
"tune_mnist()\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "d7e46189",
"metadata": {},
"source": [
"## More Keras and TensorFlow Examples\n",
"\n",
"- {doc}`/tune/examples/includes/pbt_memnn_example`: Example of training a Memory NN on bAbI with Keras using PBT.\n",
"- {doc}`/tune/examples/includes/tf_mnist_example`: Converts the Advanced TF2.0 MNIST example to use Tune\n",
" with the Trainable. This uses `tf.function`.\n",
" Original code from tensorflow: https://www.tensorflow.org/tutorials/quickstart/advanced\n",
"- {doc}`/tune/examples/includes/pbt_tune_cifar10_with_keras`:\n",
" A contributed example of tuning a Keras model on CIFAR10 with the PopulationBasedTraining scheduler.\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.13"
},
"orphan": true
},
"nbformat": 4,
"nbformat_minor": 5
}