{ "cells": [ { "attachments": {}, "cell_type": "markdown", "id": "ecad719c", "metadata": {}, "source": [ "(tune-aim-ref)=\n", "\n", "# Using Aim with Tune\n", "\n", "[Aim](https://aimstack.io) is an easy-to-use and supercharged open-source experiment tracker.\n", "Aim logs your training runs, enables a well-designed UI to compare them, and provides an API to query them programmatically.\n", "\n", "```{image} /images/aim_logo_full.png\n", ":align: center\n", ":alt: Aim\n", ":width: 100%\n", ":target: https://aimstack.io\n", "```\n", "\n", "Ray Tune currently offers built-in integration with Aim.\n", "The {ref}`AimLoggerCallback ` automatically logs metrics that are reported to Tune by using the Aim API.\n", "\n", "\n", "```{contents}\n", ":backlinks: none\n", ":local: true\n", "```\n", "\n", "## Logging Tune Hyperparameter Configurations and Results to Aim\n", "\n", "The following example demonstrates how the `AimLoggerCallback` can be used in a Tune experiment.\n", "Begin by installing and importing the necessary modules:" ] }, { "cell_type": "code", "execution_count": null, "id": "1290b5b5", "metadata": {}, "outputs": [], "source": [ "%pip install aim\n", "%pip install ray[tune]" ] }, { "cell_type": "code", "execution_count": 9, "id": "100bcf8a", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "\n", "import ray\n", "from ray import train, tune\n", "from ray.tune.logger.aim import AimLoggerCallback" ] }, { "attachments": {}, "cell_type": "markdown", "id": "9346c0f6", "metadata": {}, "source": [ "Next, define a simple `train_function`, which is a [`Trainable`](trainable-docs) that reports a loss to Tune.\n", "The objective function itself is not important for this example, as our main focus is on the integration with Aim." ] }, { "cell_type": "code", "execution_count": 2, "id": "e8b4fc4d", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "def train_function(config):\n", " for _ in range(50):\n", " loss = config[\"mean\"] + config[\"sd\"] * np.random.randn()\n", " train.report({\"loss\": loss})" ] }, { "attachments": {}, "cell_type": "markdown", "id": "831eed42", "metadata": {}, "source": [ "Here is an example of how you can use the `AimLoggerCallback` with simple grid-search Tune experiment.\n", "The logger will log each of the 9 grid-search trials as separate Aim runs." ] }, { "cell_type": "code", "execution_count": 3, "id": "52988599", "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2023-02-07 00:04:11,228\tINFO worker.py:1544 -- Started a local Ray instance. View the dashboard at \u001b[1m\u001b[32mhttp://127.0.0.1:8265 \u001b[39m\u001b[22m\n" ] }, { "data": { "text/html": [ "
\n", "
\n", "
\n", "

Tune Status

\n", " \n", "\n", "\n", "\n", "\n", "\n", "
Current time:2023-02-07 00:04:19
Running for: 00:00:06.86
Memory: 32.8/64.0 GiB
\n", "
\n", "
\n", "
\n", "

System Info

\n", " Using FIFO scheduling algorithm.
Resources requested: 0/10 CPUs, 0/0 GPUs, 0.0/26.93 GiB heap, 0.0/2.0 GiB objects\n", "
\n", " \n", "
\n", "
\n", "
\n", "

Trial Status

\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Trial name status loc mean sd iter total time (s) loss
train_function_01a3b_00000TERMINATED127.0.0.1:10277 10.385428 50 4.480311.01928
train_function_01a3b_00001TERMINATED127.0.0.1:10296 20.819716 50 2.972723.01491
train_function_01a3b_00002TERMINATED127.0.0.1:10301 30.769197 50 2.395723.87155
train_function_01a3b_00003TERMINATED127.0.0.1:10307 40.29466 50 2.415684.1507
train_function_01a3b_00004TERMINATED127.0.0.1:10313 50.152208 50 1.683835.10225
train_function_01a3b_00005TERMINATED127.0.0.1:10321 60.879814 50 1.540156.20238
train_function_01a3b_00006TERMINATED127.0.0.1:10329 70.487499 50 1.447067.79551
train_function_01a3b_00007TERMINATED127.0.0.1:10333 80.639783 50 1.4261 7.94189
train_function_01a3b_00008TERMINATED127.0.0.1:10341 90.12285 50 1.077018.82304
\n", "
\n", "
\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
\n", "

Trial Progress

\n", " \n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Trial name date done episodes_total experiment_id experiment_tag hostname iterations_since_restore lossnode_ip pid time_since_restore time_this_iter_s time_total_s timestamp timesteps_since_restoretimesteps_total training_iterationtrial_id warmup_time
train_function_01a3b_000002023-02-07_00-04-18True c8447fdceea6436c9edd6f030a5b1d820_mean=1,sd=0.3854Justins-MacBook-Pro-16 501.01928127.0.0.110277 4.48031 0.013865 4.48031 1675757058 0 5001a3b_00000 0.00264072
train_function_01a3b_000012023-02-07_00-04-18True 7dd6d3ee24244a0885b354c2850647281_mean=2,sd=0.8197Justins-MacBook-Pro-16 503.01491127.0.0.110296 2.97272 0.0584073 2.97272 1675757058 0 5001a3b_00001 0.0316792
train_function_01a3b_000022023-02-07_00-04-18True e3da49ebad034c4b8fdaf0aa87927b1a2_mean=3,sd=0.7692Justins-MacBook-Pro-16 503.87155127.0.0.110301 2.39572 0.0695491 2.39572 1675757058 0 5001a3b_00002 0.0315411
train_function_01a3b_000032023-02-07_00-04-18True 95c60c4f67c4481ebccff25b0a49e75d3_mean=4,sd=0.2947Justins-MacBook-Pro-16 504.1507 127.0.0.110307 2.41568 0.0175381 2.41568 1675757058 0 5001a3b_00003 0.0310779
train_function_01a3b_000042023-02-07_00-04-18True a216253cb41e47caa229e65488deb0194_mean=5,sd=0.1522Justins-MacBook-Pro-16 505.10225127.0.0.110313 1.68383 0.064441 1.68383 1675757058 0 5001a3b_00004 0.00450182
train_function_01a3b_000052023-02-07_00-04-18True 23834104277f476cb99d9c696281fceb5_mean=6,sd=0.8798Justins-MacBook-Pro-16 506.20238127.0.0.110321 1.54015 0.00910306 1.54015 1675757058 0 5001a3b_00005 0.0480251
train_function_01a3b_000062023-02-07_00-04-18True 15f650121df747c3bd2720481d47b2656_mean=7,sd=0.4875Justins-MacBook-Pro-16 507.79551127.0.0.110329 1.44706 0.00600386 1.44706 1675757058 0 5001a3b_00006 0.00202489
train_function_01a3b_000072023-02-07_00-04-19True 78b1673cf2034ed99135b80a0cb31e0e7_mean=8,sd=0.6398Justins-MacBook-Pro-16 507.94189127.0.0.110333 1.4261 0.00225306 1.4261 1675757059 0 5001a3b_00007 0.00209713
train_function_01a3b_000082023-02-07_00-04-19True c7f5d86154cb46b6aa27bef523edcd6f8_mean=9,sd=0.1228Justins-MacBook-Pro-16 508.82304127.0.0.110341 1.07701 0.00291467 1.07701 1675757059 0 5001a3b_00008 0.00240111
\n", "
\n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "2023-02-07 00:04:19,366\tINFO tune.py:798 -- Total run time: 7.38 seconds (6.85 seconds for the tuning loop).\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tuner = tune.Tuner(\n", " train_function,\n", " run_config=train.RunConfig(\n", " callbacks=[AimLoggerCallback()],\n", " storage_path=\"/tmp/ray_results\",\n", " name=\"aim_example\",\n", " ),\n", " param_space={\n", " \"mean\": tune.grid_search([1, 2, 3, 4, 5, 6, 7, 8, 9]),\n", " \"sd\": tune.uniform(0.1, 0.9),\n", " },\n", " tune_config=tune.TuneConfig(\n", " metric=\"loss\",\n", " mode=\"min\",\n", " ),\n", ")\n", "tuner.fit()\n" ] }, { "attachments": {}, "cell_type": "markdown", "id": "941f25f2", "metadata": {}, "source": [ "When the script executes, a grid-search is carried out and the results are saved to the Aim repo,\n", "stored at the default location -- the experiment log directory (in this case, it's at `/tmp/ray_results/aim_example`).\n", "\n", "### More Configuration Options for Aim\n", "\n", "In the example above, we used the default configuration for the `AimLoggerCallback`.\n", "There are a few options that can be configured as arguments to the callback. For example,\n", "setting `AimLoggerCallback(repo=\"/path/to/repo\")` will log results to the Aim repo at that\n", "filepath, which could be useful if you have a central location where the results of multiple\n", "Tune experiments are stored. Relative paths to the working directory where Tune script is\n", "launched can be used as well. By default, the repo will be set to the experiment log\n", "directory. See [the API reference](tune-aim-logger) for more configurations.\n", "\n", "## Launching the Aim UI\n", "\n", "Now that we have logged our results to the Aim repository, we can view it in Aim's web UI.\n", "To do this, we first find the directory where the Aim repository lives, then we use\n", "the Aim CLI to launch the web interface." ] }, { "cell_type": "code", "execution_count": 7, "id": "880f55aa", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "--------------------------------------------------------------------------\n", " Aim UI collects anonymous usage analytics. \n", " Read how to opt-out here: \n", " https://aimstack.readthedocs.io/en/latest/community/telemetry.html \n", "--------------------------------------------------------------------------\n", "\u001b[33mRunning Aim UI on repo ``\u001b[0m\n", "Open http://127.0.0.1:43800\n", "Press Ctrl+C to exit\n", "^C\n" ] } ], "source": [ "# Uncomment the following line to launch the Aim UI!\n", "#!aim up --repo=/tmp/ray_results/aim_example" ] }, { "attachments": {}, "cell_type": "markdown", "id": "adbe661a", "metadata": {}, "source": [ "After launching the Aim UI, we can open the web interface at `localhost:43800`." ] }, { "attachments": {}, "cell_type": "markdown", "id": "7bb97157", "metadata": {}, "source": [ "```{image} /images/aim_example_metrics_page.png\n", ":align: center\n", ":alt: Aim Metrics Explorer\n", ":target: https://aimstack.readthedocs.io/en/latest/ui/pages/explorers.html#metrics-explorer\n", "```" ] }, { "attachments": {}, "cell_type": "markdown", "id": "2f6e9138", "metadata": {}, "source": [ "The next sections contain more in-depth information on the API of the Tune-Aim integration.\n", "\n", "## Tune Aim Logger API\n", "\n", "(tune-aim-logger)=\n", "\n", "```{eval-rst}\n", ".. autoclass:: ray.tune.logger.aim.AimLoggerCallback\n", " :noindex:\n", "```" ] }, { "attachments": {}, "cell_type": "markdown", "id": "0ebd1904", "metadata": {}, "source": [] } ], "metadata": { "kernelspec": { "display_name": "ray_dev_py38", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.13" }, "orphan": true, "vscode": { "interpreter": { "hash": "265d195fda5292fe8f69c6e37c435a5634a1ed3b6799724e66a975f68fa21517" } } }, "nbformat": 4, "nbformat_minor": 5 }