ray.tune.search.sigopt.SigOptSearch#

class ray.tune.search.sigopt.SigOptSearch(space: Optional[List[Dict]] = None, name: str = 'Default Tune Experiment', max_concurrent: int = 1, connection: None = None, experiment_id: Optional[str] = None, observation_budget: Optional[int] = None, project: Optional[str] = None, metric: Optional[Union[str, List[str]]] = 'episode_reward_mean', mode: Optional[Union[str, List[str]]] = 'max', points_to_evaluate: Optional[List[Dict]] = None, **kwargs)[source]#

Bases: ray.tune.search.searcher.Searcher

A wrapper around SigOpt to provide trial suggestions.

You must install SigOpt and have a SigOpt API key to use this module. Store the API token as an environment variable SIGOPT_KEY as follows:

pip install -U sigopt
export SIGOPT_KEY= ...

You will need to use the SigOpt experiment and space specification.

This searcher manages its own concurrency. If this Searcher is used in a ConcurrencyLimiter, the max_concurrent value passed to it will override the value passed here.

Parameters

  • space – SigOpt configuration. Parameters will be sampled from this configuration and will be used to override parameters generated in the variant generation process. Not used if existing experiment_id is given

  • name – Name of experiment. Required by SigOpt.

  • max_concurrent – Number of maximum concurrent trials supported based on the user’s SigOpt plan. Defaults to 1. If this Searcher is used in a ConcurrencyLimiter, the max_concurrent value passed to it will override the value passed here.

  • connection – An existing connection to SigOpt.

  • experiment_id – Optional, if given will connect to an existing experiment. This allows for a more interactive experience with SigOpt, such as prior beliefs and constraints.

  • observation_budget – Optional, can improve SigOpt performance.

  • project – Optional, Project name to assign this experiment to. SigOpt can group experiments by project

  • metric (str or list(str)) – If str then the training result objective value attribute. If list(str) then a list of metrics that can be optimized together. SigOpt currently supports up to 2 metrics.

  • mode – If experiment_id is given then this field is ignored, If str then must be one of {min, max}. If list then must be comprised of {min, max, obs}. Determines whether objective is minimizing or maximizing the metric attribute. If metrics is a list then mode must be a list of the same length as metric.

Example:

space = [
    {
        'name': 'width',
        'type': 'int',
        'bounds': {
            'min': 0,
            'max': 20
        },
    },
    {
        'name': 'height',
        'type': 'int',
        'bounds': {
            'min': -100,
            'max': 100
        },
    },
]
algo = SigOptSearch(
    space, name="SigOpt Example Experiment",
    metric="mean_loss", mode="min")


Example:

space = [
    {
        'name': 'width',
        'type': 'int',
        'bounds': {
            'min': 0,
            'max': 20
        },
    },
    {
        'name': 'height',
        'type': 'int',
        'bounds': {
            'min': -100,
            'max': 100
        },
    },
]
algo = SigOptSearch(
    space, name="SigOpt Multi Objective Example Experiment",
    metric=["average", "std"], mode=["max", "min"])

Methods

add_evaluated_point(parameters, value[, ...])

Pass results from a point that has been evaluated separately.

add_evaluated_trials(trials_or_analysis, metric)

Pass results from trials that have been evaluated separately.

on_trial_complete(trial_id[, result, error])

Notification for the completion of trial.

on_trial_result(trial_id, result)

Optional notification for result during training.

restore_from_dir(checkpoint_dir)

Restores the state of a searcher from a given checkpoint_dir.

save_to_dir(checkpoint_dir[, session_str])

Automatically saves the given searcher to the checkpoint_dir.

serialize_metric(metrics, modes)

Converts metrics to https://app.sigopt.com/docs/objects/metric

serialize_result(result)

Converts experiments results to https://app.sigopt.com/docs/objects/metric_evaluation

Attributes

CKPT_FILE_TMPL

FINISHED

OBJECTIVE_MAP

metric

The training result objective value attribute.

mode

Specifies if minimizing or maximizing the metric.