Search Space API¶

Overview¶

Tune has a native interface for specifying search spaces. You can specify the search space via tune.run(config=...).

Thereby, you can either use the tune.grid_search primitive to specify an axis of a grid search…

tune.run(
    trainable,
    config={"bar": tune.grid_search([True, False])})

… or one of the random sampling primitives to specify distributions (Random Distributions API):

tune.run(
    trainable,
    config={
        "param1": tune.choice([True, False]),
        "bar": tune.uniform(0, 10),
        "alpha": tune.sample_from(lambda _: np.random.uniform(100) ** 2),
        "const": "hello"  # It is also ok to specify constant values.
    })

Caution

If you use a SearchAlgorithm, you may not be able to specify lambdas or grid search with this interface, as some search algorithms may not be compatible.

To sample multiple times/run multiple trials, specify tune.run(num_samples=N. If grid_search is provided as an argument, the same grid will be repeated N times.

# 13 different configs.
tune.run(trainable, num_samples=13, config={
    "x": tune.choice([0, 1, 2]),
    }
)

# 13 different configs.
tune.run(trainable, num_samples=13, config={
    "x": tune.choice([0, 1, 2]),
    "y": tune.randn([0, 1, 2]),
    }
)

# 4 different configs.
tune.run(trainable, config={"x": tune.grid_search([1, 2, 3, 4])}, num_samples=1)

# 3 different configs.
tune.run(trainable, config={"x": grid_search([1, 2, 3])}, num_samples=1)

# 6 different configs.
tune.run(trainable, config={"x": tune.grid_search([1, 2, 3])}, num_samples=2)

# 9 different configs.
tune.run(trainable, num_samples=1, config={
    "x": tune.grid_search([1, 2, 3]),
    "y": tune.grid_search([a, b, c])}
)

# 18 different configs.
tune.run(trainable, num_samples=2, config={
    "x": tune.grid_search([1, 2, 3]),
    "y": tune.grid_search([a, b, c])}
)

# 45 different configs.
tune.run(trainable, num_samples=5, config={
    "x": tune.grid_search([1, 2, 3]),
    "y": tune.grid_search([a, b, c])}
)

Note that grid search and random search primitives are inter-operable. Each can be used independently or in combination with each other.

# 6 different configs.
tune.run(trainable, num_samples=2, config={
    "x": tune.sample_from(...),
    "y": tune.grid_search([a, b, c])
    }
)

In the below example, num_samples=10 repeats the 3x3 grid search 10 times, for a total of 90 trials, each with randomly sampled values of alpha and beta.

 tune.run(
     my_trainable,
     name="my_trainable",
     # num_samples will repeat the entire config 10 times.
     num_samples=10
     config={
         # ``sample_from`` creates a generator to call the lambda once per trial.
         "alpha": tune.sample_from(lambda spec: np.random.uniform(100)),
         # ``sample_from`` also supports "conditional search spaces"
         "beta": tune.sample_from(lambda spec: spec.config.alpha * np.random.normal()),
         "nn_layers": [
             # tune.grid_search will make it so that all values are evaluated.
             tune.grid_search([16, 64, 256]),
             tune.grid_search([16, 64, 256]),
         ],
     },
 )

Custom/Conditional Search Spaces¶

You’ll often run into awkward search spaces (i.e., when one hyperparameter depends on another). Use tune.sample_from(func) to provide a custom callable function for generating a search space.

The parameter func should take in a spec object, which has a config namespace from which you can access other hyperparameters. This is useful for conditional distributions:

tune.run(
    ...,
    config={
        # A random function
        "alpha": tune.sample_from(lambda _: np.random.uniform(100)),
        # Use the `spec.config` namespace to access other hyperparameters
        "beta": tune.sample_from(lambda spec: spec.config.alpha * np.random.normal())
    }
)

Here’s an example showing a grid search over two nested parameters combined with random sampling from two lambda functions, generating 9 different trials. Note that the value of beta depends on the value of alpha, which is represented by referencing spec.config.alpha in the lambda function. This lets you specify conditional parameter distributions.

 tune.run(
     my_trainable,
     name="my_trainable",
     config={
         "alpha": tune.sample_from(lambda spec: np.random.uniform(100)),
         "beta": tune.sample_from(lambda spec: spec.config.alpha * np.random.normal()),
         "nn_layers": [
             tune.grid_search([16, 64, 256]),
             tune.grid_search([16, 64, 256]),
         ],
     }
 )

Note

This format is not supported by every SearchAlgorithm, and only some SearchAlgorithms, like HyperOpt and Optuna, handle conditional search spaces at all.

In order to use conditional search spaces with HyperOpt, a Hyperopt search space is necessary. Optuna supports conditional search spaces through its define-by-run interface (optuna_define_by_run_example).

Random Distributions API¶

This section covers the functions you can use to define your search spaces.

Caution

Not all SearchAlgorithms support all distributions. In particular, tune.sample_from and tune.grid_search are often unsupported. The default Random search and grid search (tune.suggest.basic_variant.BasicVariantGenerator) supports all distributions.

For a high-level overview, see this example:

config = {
    # Sample a float uniformly between -5.0 and -1.0
    "uniform": tune.uniform(-5, -1),

    # Sample a float uniformly between 3.2 and 5.4,
    # rounding to increments of 0.2
    "quniform": tune.quniform(3.2, 5.4, 0.2),

    # Sample a float uniformly between 0.0001 and 0.01, while
    # sampling in log space
    "loguniform": tune.loguniform(1e-4, 1e-2),

    # Sample a float uniformly between 0.0001 and 0.1, while
    # sampling in log space and rounding to increments of 0.00005
    "qloguniform": tune.qloguniform(1e-4, 1e-1, 5e-5),

    # Sample a random float from a normal distribution with
    # mean=10 and sd=2
    "randn": tune.randn(10, 2),

    # Sample a random float from a normal distribution with
    # mean=10 and sd=2, rounding to increments of 0.2
    "qrandn": tune.qrandn(10, 2, 0.2),

    # Sample a integer uniformly between -9 (inclusive) and 15 (exclusive)
    "randint": tune.randint(-9, 15),

    # Sample a random uniformly between -21 (inclusive) and 12 (inclusive (!))
    # rounding to increments of 3 (includes 12)
    "qrandint": tune.qrandint(-21, 12, 3),

    # Sample a integer uniformly between 1 (inclusive) and 10 (exclusive),
    # while sampling in log space
    "lograndint": tune.lograndint(1, 10),

    # Sample a integer uniformly between 1 (inclusive) and 10 (inclusive (!)),
    # while sampling in log space and rounding to increments of 2
    "qlograndint": tune.qlograndint(1, 10, 2),

    # Sample an option uniformly from the specified choices
    "choice": tune.choice(["a", "b", "c"]),

    # Sample from a random function, in this case one that
    # depends on another value from the search space
    "func": tune.sample_from(lambda spec: spec.config.uniform * 0.01),

    # Do a grid search over these values. Every value will be sampled
    # `num_samples` times (`num_samples` is the parameter you pass to `tune.run()`)
    "grid": tune.grid_search([32, 64, 128])
}

tune.uniform¶

ray.tune.uniform(lower: float, upper: float)[source]¶

Sample a float value uniformly between lower and upper.

Sampling from tune.uniform(1, 10) is equivalent to sampling from np.random.uniform(1, 10))

tune.quniform¶

ray.tune.quniform(lower: float, upper: float, q: float)[source]¶

Sample a quantized float value uniformly between lower and upper.

Sampling from tune.uniform(1, 10) is equivalent to sampling from np.random.uniform(1, 10))

The value will be quantized, i.e. rounded to an integer increment of q. Quantization makes the upper bound inclusive.

tune.loguniform¶

ray.tune.loguniform(lower: float, upper: float, base: float = 10)[source]¶

Sugar for sampling in different orders of magnitude.

Parameters

lower (float) – Lower boundary of the output interval (e.g. 1e-4)
upper (float) – Upper boundary of the output interval (e.g. 1e-2)
base (int) – Base of the log. Defaults to 10.

tune.qloguniform¶

ray.tune.qloguniform(lower: float, upper: float, q: float, base: float = 10)[source]¶

Sugar for sampling in different orders of magnitude.

The value will be quantized, i.e. rounded to an integer increment of q.

Quantization makes the upper bound inclusive.

Parameters

lower (float) – Lower boundary of the output interval (e.g. 1e-4)
upper (float) – Upper boundary of the output interval (e.g. 1e-2)
q (float) – Quantization number. The result will be rounded to an integer increment of this value.
base (int) – Base of the log. Defaults to 10.

tune.randn¶

ray.tune.randn(mean: float = 0.0, sd: float = 1.0)[source]¶

Sample a float value normally with mean and sd.

Parameters

mean (float) – Mean of the normal distribution. Defaults to 0.
sd (float) – SD of the normal distribution. Defaults to 1.

tune.qrandn¶

ray.tune.qrandn(mean: float, sd: float, q: float)[source]¶

Sample a float value normally with mean and sd.

The value will be quantized, i.e. rounded to an integer increment of q.

Parameters

mean (float) – Mean of the normal distribution.
sd (float) – SD of the normal distribution.
q (float) – Quantization number. The result will be rounded to an integer increment of this value.

tune.randint¶

ray.tune.randint(lower: int, upper: int)[source]¶

Sample an integer value uniformly between lower and upper.

lower is inclusive, upper is exclusive.

Sampling from tune.randint(10) is equivalent to sampling from np.random.randint(10)

Changed in version 1.5.0: When converting Ray Tune configs to searcher-specific search spaces, the lower and upper limits are adjusted to keep compatibility with the bounds stated in the docstring above.

tune.qrandint¶

ray.tune.qrandint(lower: int, upper: int, q: int = 1)[source]¶

Sample an integer value uniformly between lower and upper.

lower is inclusive, upper is also inclusive (!).

The value will be quantized, i.e. rounded to an integer increment of q. Quantization makes the upper bound inclusive.

Changed in version 1.5.0: When converting Ray Tune configs to searcher-specific search spaces, the lower and upper limits are adjusted to keep compatibility with the bounds stated in the docstring above.

tune.lograndint¶

ray.tune.lograndint(lower: int, upper: int, base: float = 10)[source]¶

Sample an integer value log-uniformly between lower and upper, with base being the base of logarithm.

lower is inclusive, upper is exclusive.

Changed in version 1.5.0: When converting Ray Tune configs to searcher-specific search spaces, the lower and upper limits are adjusted to keep compatibility with the bounds stated in the docstring above.

tune.qlograndint¶

ray.tune.qlograndint(lower: int, upper: int, q: int, base: float = 10)[source]¶

Sample an integer value log-uniformly between lower and upper, with base being the base of logarithm.

lower is inclusive, upper is also inclusive (!).

The value will be quantized, i.e. rounded to an integer increment of q. Quantization makes the upper bound inclusive.

Changed in version 1.5.0: When converting Ray Tune configs to searcher-specific search spaces, the lower and upper limits are adjusted to keep compatibility with the bounds stated in the docstring above.

tune.choice¶

ray.tune.choice(categories: Sequence)[source]¶

Sample a categorical value.

Sampling from tune.choice([1, 2]) is equivalent to sampling from np.random.choice([1, 2])

tune.sample_from¶

ray.tune.sample_from(func: Callable[[Dict], Any])[source]¶

Specify that tune should sample configuration values from this function.

Parameters: func – An callable function to draw a sample from.

Grid Search API¶

ray.tune.grid_search(values: List) → Dict[str, List][source]¶

Convenience method for specifying grid search over a value.

Parameters: values – An iterable whose parameters will be gridded.

Ray v1.9.2

Search Space API¶

Overview¶

Custom/Conditional Search Spaces¶

Random Distributions API¶

tune.uniform¶

tune.quniform¶

tune.loguniform¶

tune.qloguniform¶

tune.randn¶

tune.qrandn¶

tune.randint¶

tune.qrandint¶

tune.lograndint¶

tune.qlograndint¶

tune.choice¶

tune.sample_from¶

Grid Search API¶

References¶