Running Tune experiments with AxSearch¶

In this tutorial we introduce Ax, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with Ax and, as a result, allow you to seamlessly scale up a Ax optimization process - without sacrificing performance.

Ax is a platform for optimizing any kind of experiment, including machine learning experiments, A/B tests, and simulations. Ax can optimize discrete configurations (e.g., variants of an A/B test) using multi-armed bandit optimization, and continuous/ordered configurations (e.g. float/int parameters) using Bayesian optimization. Results of A/B tests and simulations with reinforcement learning agents often exhibit high amounts of noise. Ax supports state-of-the-art algorithms which work better than traditional Bayesian optimization in high-noise settings. Ax also supports multi-objective and constrained optimization which are common to real-world problems (e.g. improving load time without increasing data use). Ax belongs to the domain of “derivative-free” and “black-box” optimization.

In this example we minimize a simple objective to briefly demonstrate the usage of AxSearch with Ray Tune via AxSearch. It’s useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume ax-platform==0.2.4 library is installed withe python version >= 3.7. To learn more, please refer to the Ax website.

Click below to see all the imports we need for this example. You can also launch directly into a Binder instance to run this notebook yourself. Just click on the rocket symbol at the top of the navigation.

import numpy as np
import time

import ray
from ray import air, tune
from ray.air import session
from ray.tune.search.ax import AxSearch

Let’s start by defining a classic benchmark for global optimization. The form here is explicit for demonstration, yet it is typically a black-box. We artificially sleep for a bit (0.02 seconds) to simulate a long-running ML experiment. This setup assumes that we’re running multiple steps of an experiment and try to tune 6-dimensions of the x hyperparameter.

def landscape(x):
    """
    Hartmann 6D function containing 6 local minima.
    It is a classic benchmark for developing global optimization algorithms.
    """
    alpha = np.array([1.0, 1.2, 3.0, 3.2])
    A = np.array(
        [
            [10, 3, 17, 3.5, 1.7, 8],
            [0.05, 10, 17, 0.1, 8, 14],
            [3, 3.5, 1.7, 10, 17, 8],
            [17, 8, 0.05, 10, 0.1, 14],
        ]
    )
    P = 10 ** (-4) * np.array(
        [
            [1312, 1696, 5569, 124, 8283, 5886],
            [2329, 4135, 8307, 3736, 1004, 9991],
            [2348, 1451, 3522, 2883, 3047, 6650],
            [4047, 8828, 8732, 5743, 1091, 381],
        ]
    )
    y = 0.0
    for j, alpha_j in enumerate(alpha):
        t = 0
        for k in range(6):
            t += A[j, k] * ((x[k] - P[j, k]) ** 2)
        y -= alpha_j * np.exp(-t)
    return y

Next, our objective function takes a Tune config, evaluates the landscape of our experiment in a training loop, and uses session.report to report the landscape back to Tune.

def objective(config):
    for i in range(config["iterations"]):
        x = np.array([config.get("x{}".format(i + 1)) for i in range(6)])
        session.report(
            {"timesteps_total": i, "landscape": landscape(x), "l2norm": np.sqrt((x ** 2).sum())}
        )
        time.sleep(0.02)

Next we define a search space. The critical assumption is that the optimal hyperparamters live within this space. Yet, if the space is very large, then those hyperparamters may be difficult to find in a short amount of time.

search_space = {
    "iterations":100,
    "x1": tune.uniform(0.0, 1.0),
    "x2": tune.uniform(0.0, 1.0),
    "x3": tune.uniform(0.0, 1.0),
    "x4": tune.uniform(0.0, 1.0),
    "x5": tune.uniform(0.0, 1.0),
    "x6": tune.uniform(0.0, 1.0)
}

Now we define the search algorithm from AxSearch. If you want to constrain your parameters or even the space of outcomes, that can be easily done by passing the argumentsas below.

algo = AxSearch(
    parameter_constraints=["x1 + x2 <= 2.0"],
    outcome_constraints=["l2norm <= 1.25"],
)

We also use ConcurrencyLimiter to constrain to 4 concurrent trials.

algo = tune.search.ConcurrencyLimiter(algo, max_concurrent=4)

The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to 1000 samples. You can decrease this if it takes too long on your machine, or you can set a time limit easily through stop argument in the air.RunConfig() as we will show here.

num_samples = 100
stop_timesteps = 200

Finally, we run the experiment to find the global minimum of the provided landscape (which contains 5 false minima). The argument to metric, "landscape", is provided via the objective function’s session.report. The experiment "min"imizes the “mean_loss” of the landscape by searching within search_space via algo, num_samples times or when "timesteps_total": stop_timesteps. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, notice how efficient it is to execute tuner.fit().

tuner = tune.Tuner(
    objective,
    tune_config=tune.TuneConfig(
        metric="landscape",
        mode="min",
        search_alg=algo,
        num_samples=num_samples,
    ),
    run_config=air.RunConfig(
        name="ax",
        stop={"timesteps_total": stop_timesteps}
    ),
    param_space=search_space,
)
results = tuner.fit()

Function checkpointing is disabled. This may result in unexpected behavior when using checkpointing features or certain schedulers. To enable, set the train function arguments to be `func(config, checkpoint_dir=None)`.
[INFO 07-22 15:04:18] ax.service.ax_client: Starting optimization with verbose logging. To disable logging, set the `verbose_logging` argument to `False`. Note that float values in the logs are rounded to 6 decimal points.
[INFO 07-22 15:04:18] ax.service.utils.instantiation: Created search space: SearchSpace(parameters=[FixedParameter(name='iterations', parameter_type=INT, value=100), RangeParameter(name='x1', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x2', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x3', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x4', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x5', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='x6', parameter_type=FLOAT, range=[0.0, 1.0])], parameter_constraints=[ParameterConstraint(1.0*x1 + 1.0*x2 <= 2.0)]).
[INFO 07-22 15:04:18] ax.modelbridge.dispatch_utils: Using Bayesian optimization since there are more ordered parameters than there are categories for the unordered categorical parameters.
[INFO 07-22 15:04:18] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 12 trials, GPEI for subsequent trials]). Iterations after 12 will take longer to generate due to  model-fitting.
Detected sequential enforcement. Be sure to use a ConcurrencyLimiter.

== Status ==
Current time: 2022-07-22 15:04:35 (running for 00:00:16.56)
Memory usage on this node: 9.9/16.0 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/5.13 GiB heap, 0.0/2.0 GiB objects
Current best trial: 34b7abda with landscape=-1.6624439263544026 and parameters={'iterations': 100, 'x1': 0.26526361983269453, 'x2': 0.9248840995132923, 'x3': 0.15171580761671066, 'x4': 0.43602637108415365, 'x5': 0.8573104059323668, 'x6': 0.08981018699705601}
Result logdir: /Users/kai/ray_results/ax
Number of trials: 10/10 (10 TERMINATED)

Trial name	status	loc	iterations	x1	x2	x3	x4	x5	x6	iter	total time (s)	ts	landscape	l2norm
objective_2dfbe86a	TERMINATED	127.0.0.1:44721	100	0.0558336	0.0896192	0.958956	0.234474	0.174516	0.970311	100	2.57372	99	-0.805233	1.39917
objective_2fa776c0	TERMINATED	127.0.0.1:44726	100	0.744772	0.754537	0.0950125	0.273877	0.0966829	0.368943	100	2.6361	99	-0.11286	1.16341
objective_2fabaa1a	TERMINATED	127.0.0.1:44727	100	0.405704	0.374626	0.935628	0.222185	0.787212	0.00812439	100	2.62393	99	-0.11348	1.35995
objective_2faee7c0	TERMINATED	127.0.0.1:44728	100	0.664728	0.207519	0.359514	0.704578	0.755882	0.812402	100	2.62069	99	-0.0119837	1.53035
objective_313d3d3a	TERMINATED	127.0.0.1:44747	100	0.0418746	0.992783	0.906027	0.594429	0.825393	0.646362	100	3.16233	99	-0.00677976	1.80573
objective_32c9acd8	TERMINATED	127.0.0.1:44726	100	0.126064	0.703408	0.344681	0.337363	0.401396	0.679202	100	3.12119	99	-0.904622	1.16864
objective_32cf8ca2	TERMINATED	127.0.0.1:44756	100	0.0910936	0.304138	0.869848	0.405435	0.567922	0.228608	100	2.70791	99	-0.146532	1.18178
objective_32d8dd20	TERMINATED	127.0.0.1:44758	100	0.603178	0.409057	0.729056	0.0825984	0.572948	0.508304	100	2.64158	99	-0.247223	1.28691
objective_34adf04a	TERMINATED	127.0.0.1:44768	100	0.454189	0.271772	0.530871	0.991841	0.691843	0.472366	100	2.70327	99	-0.0132915	1.49917
objective_34b7abda	TERMINATED	127.0.0.1:44771	100	0.265264	0.924884	0.151716	0.436026	0.85731	0.0898102	100	2.68521	99	-1.66244	1.37185

[INFO 07-22 15:04:19] ax.service.ax_client: Generated new trial 0 with parameters {'x1': 0.055834, 'x2': 0.089619, 'x3': 0.958956, 'x4': 0.234474, 'x5': 0.174516, 'x6': 0.970311, 'iterations': 100}.
[INFO 07-22 15:04:22] ax.service.ax_client: Generated new trial 1 with parameters {'x1': 0.744772, 'x2': 0.754537, 'x3': 0.095012, 'x4': 0.273877, 'x5': 0.096683, 'x6': 0.368943, 'iterations': 100}.
[INFO 07-22 15:04:22] ax.service.ax_client: Generated new trial 2 with parameters {'x1': 0.405704, 'x2': 0.374626, 'x3': 0.935628, 'x4': 0.222185, 'x5': 0.787212, 'x6': 0.008124, 'iterations': 100}.
[INFO 07-22 15:04:22] ax.service.ax_client: Generated new trial 3 with parameters {'x1': 0.664728, 'x2': 0.207519, 'x3': 0.359514, 'x4': 0.704578, 'x5': 0.755882, 'x6': 0.812402, 'iterations': 100}.

Result for objective_2dfbe86a:
  date: 2022-07-22_15-04-22
  done: false
  experiment_id: 4ef8a12ac94a4f4fa483ec18e347967f
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 1
  l2norm: 1.3991721132671366
  landscape: -0.8052333562869153
  node_ip: 127.0.0.1
  pid: 44721
  time_since_restore: 0.00022912025451660156
  time_this_iter_s: 0.00022912025451660156
  time_total_s: 0.00022912025451660156
  timestamp: 1658498662
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 2dfbe86a
  warmup_time: 0.0035619735717773438
  

[INFO 07-22 15:04:24] ax.service.ax_client: Completed trial 0 with data: {'landscape': (-0.805233, None), 'l2norm': (1.399172, None)}.
[INFO 07-22 15:04:24] ax.service.ax_client: Generated new trial 4 with parameters {'x1': 0.041875, 'x2': 0.992783, 'x3': 0.906027, 'x4': 0.594429, 'x5': 0.825393, 'x6': 0.646362, 'iterations': 100}.

Result for objective_2faee7c0:
  date: 2022-07-22_15-04-24
  done: false
  experiment_id: 3699644e85ac439cb7c1a36ed0976307
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 1
  l2norm: 1.530347488145437
  landscape: -0.011983676977099367
  node_ip: 127.0.0.1
  pid: 44728
  time_since_restore: 0.00022292137145996094
  time_this_iter_s: 0.00022292137145996094
  time_total_s: 0.00022292137145996094
  timestamp: 1658498664
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 2faee7c0
  warmup_time: 0.0027179718017578125
  
Result for objective_2fa776c0:
  date: 2022-07-22_15-04-24
  done: false
  experiment_id: c555bfed13ac43e5b8c8e9f6d4b9b2f7
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 1
  l2norm: 1.1634068454629019
  landscape: -0.11285961764770336
  node_ip: 127.0.0.1
  pid: 44726
  time_since_restore: 0.000225067138671875
  time_this_iter_s: 0.000225067138671875
  time_total_s: 0.000225067138671875
  timestamp: 1658498664
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 2fa776c0
  warmup_time: 0.0026290416717529297
  
Result for objective_2dfbe86a:
  date: 2022-07-22_15-04-24
  done: true
  experiment_id: 4ef8a12ac94a4f4fa483ec18e347967f
  experiment_tag: 1_iterations=100,x1=0.0558,x2=0.0896,x3=0.9590,x4=0.2345,x5=0.1745,x6=0.9703
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 100
  l2norm: 1.3991721132671366
  landscape: -0.8052333562869153
  node_ip: 127.0.0.1
  pid: 44721
  time_since_restore: 2.573719024658203
  time_this_iter_s: 0.0251619815826416
  time_total_s: 2.573719024658203
  timestamp: 1658498664
  timesteps_since_restore: 0
  timesteps_total: 99
  training_iteration: 100
  trial_id: 2dfbe86a
  warmup_time: 0.0035619735717773438
  
Result for objective_2fabaa1a:
  date: 2022-07-22_15-04-24
  done: false
  experiment_id: eb9287e4fe5f44c7868dc943e2642312
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 1
  l2norm: 1.3599537840291782
  landscape: -0.11348012497414121
  node_ip: 127.0.0.1
  pid: 44727
  time_since_restore: 0.00022077560424804688
  time_this_iter_s: 0.00022077560424804688
  time_total_s: 0.00022077560424804688
  timestamp: 1658498664
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 2fabaa1a
  warmup_time: 0.0025510787963867188
  

[INFO 07-22 15:04:27] ax.service.ax_client: Completed trial 3 with data: {'landscape': (-0.011984, None), 'l2norm': (1.530347, None)}.
[INFO 07-22 15:04:27] ax.service.ax_client: Generated new trial 5 with parameters {'x1': 0.126064, 'x2': 0.703408, 'x3': 0.344681, 'x4': 0.337363, 'x5': 0.401396, 'x6': 0.679202, 'iterations': 100}.
[INFO 07-22 15:04:27] ax.service.ax_client: Completed trial 1 with data: {'landscape': (-0.11286, None), 'l2norm': (1.163407, None)}.
[INFO 07-22 15:04:27] ax.service.ax_client: Generated new trial 6 with parameters {'x1': 0.091094, 'x2': 0.304138, 'x3': 0.869848, 'x4': 0.405435, 'x5': 0.567922, 'x6': 0.228608, 'iterations': 100}.
[INFO 07-22 15:04:27] ax.service.ax_client: Completed trial 2 with data: {'landscape': (-0.11348, None), 'l2norm': (1.359954, None)}.
[INFO 07-22 15:04:27] ax.service.ax_client: Generated new trial 7 with parameters {'x1': 0.603178, 'x2': 0.409057, 'x3': 0.729056, 'x4': 0.082598, 'x5': 0.572948, 'x6': 0.508304, 'iterations': 100}.

Result for objective_313d3d3a:
  date: 2022-07-22_15-04-27
  done: false
  experiment_id: fa7afd557e154fbebe4f54d8eedb3573
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 1
  l2norm: 1.805729990121368
  landscape: -0.006779757704679272
  node_ip: 127.0.0.1
  pid: 44747
  time_since_restore: 0.00021076202392578125
  time_this_iter_s: 0.00021076202392578125
  time_total_s: 0.00021076202392578125
  timestamp: 1658498667
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 313d3d3a
  warmup_time: 0.0029790401458740234
  
Result for objective_2faee7c0:
  date: 2022-07-22_15-04-27
  done: true
  experiment_id: 3699644e85ac439cb7c1a36ed0976307
  experiment_tag: 4_iterations=100,x1=0.6647,x2=0.2075,x3=0.3595,x4=0.7046,x5=0.7559,x6=0.8124
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 100
  l2norm: 1.530347488145437
  landscape: -0.011983676977099367
  node_ip: 127.0.0.1
  pid: 44728
  time_since_restore: 2.6206929683685303
  time_this_iter_s: 0.027359962463378906
  time_total_s: 2.6206929683685303
  timestamp: 1658498667
  timesteps_since_restore: 0
  timesteps_total: 99
  training_iteration: 100
  trial_id: 2faee7c0
  warmup_time: 0.0027179718017578125
  
Result for objective_2fa776c0:
  date: 2022-07-22_15-04-27
  done: true
  experiment_id: c555bfed13ac43e5b8c8e9f6d4b9b2f7
  experiment_tag: 2_iterations=100,x1=0.7448,x2=0.7545,x3=0.0950,x4=0.2739,x5=0.0967,x6=0.3689
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 100
  l2norm: 1.1634068454629019
  landscape: -0.11285961764770336
  node_ip: 127.0.0.1
  pid: 44726
  time_since_restore: 2.6361019611358643
  time_this_iter_s: 0.0264589786529541
  time_total_s: 2.6361019611358643
  timestamp: 1658498667
  timesteps_since_restore: 0
  timesteps_total: 99
  training_iteration: 100
  trial_id: 2fa776c0
  warmup_time: 0.0026290416717529297
  
Result for objective_32c9acd8:
  date: 2022-07-22_15-04-27
  done: false
  experiment_id: c555bfed13ac43e5b8c8e9f6d4b9b2f7
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 1
  l2norm: 1.1686440476629836
  landscape: -0.9046216637367911
  node_ip: 127.0.0.1
  pid: 44726
  time_since_restore: 0.00020194053649902344
  time_this_iter_s: 0.00020194053649902344
  time_total_s: 0.00020194053649902344
  timestamp: 1658498667
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 32c9acd8
  warmup_time: 0.0026290416717529297
  
Result for objective_2fabaa1a:
  date: 2022-07-22_15-04-27
  done: true
  experiment_id: eb9287e4fe5f44c7868dc943e2642312
  experiment_tag: 3_iterations=100,x1=0.4057,x2=0.3746,x3=0.9356,x4=0.2222,x5=0.7872,x6=0.0081
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 100
  l2norm: 1.3599537840291782
  landscape: -0.11348012497414121
  node_ip: 127.0.0.1
  pid: 44727
  time_since_restore: 2.623929977416992
  time_this_iter_s: 0.032716989517211914
  time_total_s: 2.623929977416992
  timestamp: 1658498667
  timesteps_since_restore: 0
  timesteps_total: 99
  training_iteration: 100
  trial_id: 2fabaa1a
  warmup_time: 0.0025510787963867188
  
Result for objective_32d8dd20:
  date: 2022-07-22_15-04-30
  done: false
  experiment_id: 171527593b0f4cbf941c0a03faaf0953
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 1
  l2norm: 1.2869105702896437
  landscape: -0.24722262157458608
  node_ip: 127.0.0.1
  pid: 44758
  time_since_restore: 0.00021886825561523438
  time_this_iter_s: 0.00021886825561523438
  time_total_s: 0.00021886825561523438
  timestamp: 1658498670
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 32d8dd20
  warmup_time: 0.002732992172241211
  
Result for objective_32cf8ca2:
  date: 2022-07-22_15-04-29
  done: false
  experiment_id: 37610500f6df493aae4e7e46bb21bf09
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 1
  l2norm: 1.1817810425508524
  landscape: -0.14653248187442922
  node_ip: 127.0.0.1
  pid: 44756
  time_since_restore: 0.00025081634521484375
  time_this_iter_s: 0.00025081634521484375
  time_total_s: 0.00025081634521484375
  timestamp: 1658498669
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 32cf8ca2
  warmup_time: 0.0032138824462890625
  

[INFO 07-22 15:04:30] ax.service.ax_client: Completed trial 4 with data: {'landscape': (-0.00678, None), 'l2norm': (1.80573, None)}.
[INFO 07-22 15:04:30] ax.service.ax_client: Generated new trial 8 with parameters {'x1': 0.454189, 'x2': 0.271772, 'x3': 0.530871, 'x4': 0.991841, 'x5': 0.691843, 'x6': 0.472366, 'iterations': 100}.
[INFO 07-22 15:04:30] ax.service.ax_client: Completed trial 5 with data: {'landscape': (-0.904622, None), 'l2norm': (1.168644, None)}.
[INFO 07-22 15:04:30] ax.service.ax_client: Generated new trial 9 with parameters {'x1': 0.265264, 'x2': 0.924884, 'x3': 0.151716, 'x4': 0.436026, 'x5': 0.85731, 'x6': 0.08981, 'iterations': 100}.

Result for objective_313d3d3a:
  date: 2022-07-22_15-04-30
  done: true
  experiment_id: fa7afd557e154fbebe4f54d8eedb3573
  experiment_tag: 5_iterations=100,x1=0.0419,x2=0.9928,x3=0.9060,x4=0.5944,x5=0.8254,x6=0.6464
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 100
  l2norm: 1.805729990121368
  landscape: -0.006779757704679272
  node_ip: 127.0.0.1
  pid: 44747
  time_since_restore: 3.1623308658599854
  time_this_iter_s: 0.02911996841430664
  time_total_s: 3.1623308658599854
  timestamp: 1658498670
  timesteps_since_restore: 0
  timesteps_total: 99
  training_iteration: 100
  trial_id: 313d3d3a
  warmup_time: 0.0029790401458740234
  
Result for objective_32c9acd8:
  date: 2022-07-22_15-04-30
  done: true
  experiment_id: c555bfed13ac43e5b8c8e9f6d4b9b2f7
  experiment_tag: 6_iterations=100,x1=0.1261,x2=0.7034,x3=0.3447,x4=0.3374,x5=0.4014,x6=0.6792
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 100
  l2norm: 1.1686440476629836
  landscape: -0.9046216637367911
  node_ip: 127.0.0.1
  pid: 44726
  time_since_restore: 3.1211891174316406
  time_this_iter_s: 0.02954697608947754
  time_total_s: 3.1211891174316406
  timestamp: 1658498670
  timesteps_since_restore: 0
  timesteps_total: 99
  training_iteration: 100
  trial_id: 32c9acd8
  warmup_time: 0.0026290416717529297
  

[INFO 07-22 15:04:32] ax.service.ax_client: Completed trial 7 with data: {'landscape': (-0.247223, None), 'l2norm': (1.286911, None)}.
[INFO 07-22 15:04:32] ax.service.ax_client: Completed trial 6 with data: {'landscape': (-0.146532, None), 'l2norm': (1.181781, None)}.

Result for objective_32d8dd20:
  date: 2022-07-22_15-04-32
  done: true
  experiment_id: 171527593b0f4cbf941c0a03faaf0953
  experiment_tag: 8_iterations=100,x1=0.6032,x2=0.4091,x3=0.7291,x4=0.0826,x5=0.5729,x6=0.5083
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 100
  l2norm: 1.2869105702896437
  landscape: -0.24722262157458608
  node_ip: 127.0.0.1
  pid: 44758
  time_since_restore: 2.6415798664093018
  time_this_iter_s: 0.026781082153320312
  time_total_s: 2.6415798664093018
  timestamp: 1658498672
  timesteps_since_restore: 0
  timesteps_total: 99
  training_iteration: 100
  trial_id: 32d8dd20
  warmup_time: 0.002732992172241211
  
Result for objective_32cf8ca2:
  date: 2022-07-22_15-04-32
  done: true
  experiment_id: 37610500f6df493aae4e7e46bb21bf09
  experiment_tag: 7_iterations=100,x1=0.0911,x2=0.3041,x3=0.8698,x4=0.4054,x5=0.5679,x6=0.2286
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 100
  l2norm: 1.1817810425508524
  landscape: -0.14653248187442922
  node_ip: 127.0.0.1
  pid: 44756
  time_since_restore: 2.707913875579834
  time_this_iter_s: 0.027456998825073242
  time_total_s: 2.707913875579834
  timestamp: 1658498672
  timesteps_since_restore: 0
  timesteps_total: 99
  training_iteration: 100
  trial_id: 32cf8ca2
  warmup_time: 0.0032138824462890625
  
Result for objective_34adf04a:
  date: 2022-07-22_15-04-33
  done: false
  experiment_id: 4f65c5b68f5c49d98fda388e37c83deb
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 1
  l2norm: 1.4991655675380078
  landscape: -0.01329150870283869
  node_ip: 127.0.0.1
  pid: 44768
  time_since_restore: 0.00021600723266601562
  time_this_iter_s: 0.00021600723266601562
  time_total_s: 0.00021600723266601562
  timestamp: 1658498673
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 34adf04a
  warmup_time: 0.0027239322662353516
  
Result for objective_34b7abda:
  date: 2022-07-22_15-04-33
  done: false
  experiment_id: f135a2c40f5644ba9d2ae096a9dd10e0
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 1
  l2norm: 1.3718451333547932
  landscape: -1.6624439263544026
  node_ip: 127.0.0.1
  pid: 44771
  time_since_restore: 0.0002338886260986328
  time_this_iter_s: 0.0002338886260986328
  time_total_s: 0.0002338886260986328
  timestamp: 1658498673
  timesteps_since_restore: 0
  timesteps_total: 0
  training_iteration: 1
  trial_id: 34b7abda
  warmup_time: 0.002721071243286133
  

[INFO 07-22 15:04:35] ax.service.ax_client: Completed trial 8 with data: {'landscape': (-0.013292, None), 'l2norm': (1.499166, None)}.
[INFO 07-22 15:04:35] ax.service.ax_client: Completed trial 9 with data: {'landscape': (-1.662444, None), 'l2norm': (1.371845, None)}.

Result for objective_34adf04a:
  date: 2022-07-22_15-04-35
  done: true
  experiment_id: 4f65c5b68f5c49d98fda388e37c83deb
  experiment_tag: 9_iterations=100,x1=0.4542,x2=0.2718,x3=0.5309,x4=0.9918,x5=0.6918,x6=0.4724
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 100
  l2norm: 1.4991655675380078
  landscape: -0.01329150870283869
  node_ip: 127.0.0.1
  pid: 44768
  time_since_restore: 2.7032668590545654
  time_this_iter_s: 0.029300928115844727
  time_total_s: 2.7032668590545654
  timestamp: 1658498675
  timesteps_since_restore: 0
  timesteps_total: 99
  training_iteration: 100
  trial_id: 34adf04a
  warmup_time: 0.0027239322662353516
  
Result for objective_34b7abda:
  date: 2022-07-22_15-04-35
  done: true
  experiment_id: f135a2c40f5644ba9d2ae096a9dd10e0
  experiment_tag: 10_iterations=100,x1=0.2653,x2=0.9249,x3=0.1517,x4=0.4360,x5=0.8573,x6=0.0898
  hostname: Kais-MacBook-Pro.local
  iterations_since_restore: 100
  l2norm: 1.3718451333547932
  landscape: -1.6624439263544026
  node_ip: 127.0.0.1
  pid: 44771
  time_since_restore: 2.6852078437805176
  time_this_iter_s: 0.029579877853393555
  time_total_s: 2.6852078437805176
  timestamp: 1658498675
  timesteps_since_restore: 0
  timesteps_total: 99
  training_iteration: 100
  trial_id: 34b7abda
  warmup_time: 0.002721071243286133
  

And now we have the hyperparameters found to minimize the mean loss.

print("Best hyperparameters found were: ", results.get_best_result().config)

Best hyperparameters found were:  {'iterations': 100, 'x1': 0.26526361983269453, 'x2': 0.9248840995132923, 'x3': 0.15171580761671066, 'x4': 0.43602637108415365, 'x5': 0.8573104059323668, 'x6': 0.08981018699705601}

Ray 3.0.0.dev0

Running Tune experiments with AxSearch¶