Execution (tune.run, tune.Experiment)¶

tune.run¶

ray.tune.run(run_or_experiment: Union[str, Callable, Type], name: Optional[str] = None, metric: Optional[str] = None, mode: Optional[str] = None, stop: Union[None, Mapping, ray.tune.stopper.Stopper, Callable[[str, Mapping], bool]] = None, time_budget_s: Union[None, int, float, datetime.timedelta] = None, config: Optional[Dict[str, Any]] = None, resources_per_trial: Union[None, Mapping[str, Union[float, int, Mapping]], ray.tune.utils.placement_groups.PlacementGroupFactory] = None, num_samples: int = 1, local_dir: Optional[str] = None, search_alg: Optional[Union[ray.tune.suggest.suggestion.Searcher, ray.tune.suggest.search.SearchAlgorithm, str]] = None, scheduler: Optional[Union[ray.tune.schedulers.trial_scheduler.TrialScheduler, str]] = None, keep_checkpoints_num: Optional[int] = None, checkpoint_score_attr: Optional[str] = None, checkpoint_freq: int = 0, checkpoint_at_end: bool = False, verbose: Union[int, ray.tune.utils.log.Verbosity] = Verbosity.V3_TRIAL_DETAILS, progress_reporter: Optional[ray.tune.progress_reporter.ProgressReporter] = None, log_to_file: bool = False, trial_name_creator: Optional[Callable[[ray.tune.trial.Trial], str]] = None, trial_dirname_creator: Optional[Callable[[ray.tune.trial.Trial], str]] = None, sync_config: Optional[ray.tune.syncer.SyncConfig] = None, export_formats: Optional[Sequence] = None, max_failures: int = 0, fail_fast: bool = False, restore: Optional[str] = None, server_port: Optional[int] = None, resume: bool = False, reuse_actors: bool = False, trial_executor: Optional[ray.tune.ray_trial_executor.RayTrialExecutor] = None, raise_on_failed_trial: bool = True, callbacks: Optional[Sequence[ray.tune.callback.Callback]] = None, max_concurrent_trials: Optional[int] = None, _experiment_checkpoint_dir: Optional[str] = None, queue_trials: Optional[bool] = None, loggers: Optional[Sequence[Type[ray.tune.logger.Logger]]] = None, _remote: Optional[bool] = None) → ray.tune.analysis.experiment_analysis.ExperimentAnalysis[source]¶

Executes training.

When a SIGINT signal is received (e.g. through Ctrl+C), the tuning run will gracefully shut down and checkpoint the latest experiment state. Sending SIGINT again (or SIGKILL/SIGTERM instead) will skip this step.

Many aspects of Tune, such as the frequency of global checkpointing, maximum pending placement group trials and the path of the result directory be configured through environment variables. Refer to Environment variables for a list of environment variables available.

Examples:

# Run 10 trials (each trial is one instance of a Trainable). Tune runs
# in parallel and automatically determines concurrency.
tune.run(trainable, num_samples=10)

# Run 1 trial, stop when trial has reached 10 iterations
tune.run(my_trainable, stop={"training_iteration": 10})

# automatically retry failed trials up to 3 times
tune.run(my_trainable, stop={"training_iteration": 10}, max_failures=3)

# Run 1 trial, search over hyperparameters, stop after 10 iterations.
space = {"lr": tune.uniform(0, 1), "momentum": tune.uniform(0, 1)}
tune.run(my_trainable, config=space, stop={"training_iteration": 10})

# Resumes training if a previous machine crashed
tune.run(my_trainable, config=space,
         local_dir=<path/to/dir>, resume=True)

# Rerun ONLY failed trials after an experiment is finished.
tune.run(my_trainable, config=space,
         local_dir=<path/to/dir>, resume="ERRORED_ONLY")

Parameters

run_or_experiment (function | class | str | Experiment) – If function|class|str, this is the algorithm or model to train. This may refer to the name of a built-on algorithm (e.g. RLLib’s DQN or PPO), a user-defined trainable function or class, or the string identifier of a trainable function or class registered in the tune registry. If Experiment, then Tune will execute training based on Experiment.spec. If you want to pass in a Python lambda, you will need to first register the function: tune.register_trainable("lambda_id", lambda x: ...). You can then use tune.run("lambda_id").
metric (str) – Metric to optimize. This metric should be reported with tune.report(). If set, will be passed to the search algorithm and scheduler.
mode (str) – Must be one of [min, max]. Determines whether objective is minimizing or maximizing the metric attribute. If set, will be passed to the search algorithm and scheduler.
name (str) – Name of experiment.
stop (dict | callable | Stopper) – Stopping criteria. If dict, the keys may be any field in the return result of ‘train()’, whichever is reached first. If function, it must take (trial_id, result) as arguments and return a boolean (True if trial should be stopped, False otherwise). This can also be a subclass of ray.tune.Stopper, which allows users to implement custom experiment-wide stopping (i.e., stopping an entire Tune run based on some time constraint).
time_budget_s (int|float|datetime.timedelta) – Global time budget in seconds after which all trials are stopped. Can also be a datetime.timedelta object.
config (dict) – Algorithm-specific configuration for Tune variant generation (e.g. env, hyperparams). Defaults to empty dict. Custom search algorithms may ignore this.
resources_per_trial (dict|PlacementGroupFactory) – Machine resources to allocate per trial, e.g. {"cpu": 64, "gpu": 8}. Note that GPUs will not be assigned unless you specify them here. Defaults to 1 CPU and 0 GPUs in Trainable.default_resource_request(). This can also be a PlacementGroupFactory object wrapping arguments to create a per-trial placement group.
num_samples (int) – Number of times to sample from the hyperparameter space. Defaults to 1. If grid_search is provided as an argument, the grid will be repeated num_samples of times. If this is -1, (virtually) infinite samples are generated until a stopping condition is met.
local_dir (str) – Local dir to save training results to. Defaults to ~/ray_results.
search_alg (Searcher|SearchAlgorithm|str) – Search algorithm for optimization. You can also use the name of the algorithm.
scheduler (TrialScheduler|str) – Scheduler for executing the experiment. Choose among FIFO (default), MedianStopping, AsyncHyperBand, HyperBand and PopulationBasedTraining. Refer to ray.tune.schedulers for more options. You can also use the name of the scheduler.
keep_checkpoints_num (int) – Number of checkpoints to keep. A value of None keeps all checkpoints. Defaults to None. If set, need to provide checkpoint_score_attr.
checkpoint_score_attr (str) – Specifies by which attribute to rank the best checkpoint. Default is increasing order. If attribute starts with min- it will rank attribute in decreasing order, i.e. min-validation_loss.
checkpoint_freq (int) – How many training iterations between checkpoints. A value of 0 (default) disables checkpointing. This has no effect when using the Functional Training API.
checkpoint_at_end (bool) – Whether to checkpoint at the end of the experiment regardless of the checkpoint_freq. Default is False. This has no effect when using the Functional Training API.
verbose (Union[int, Verbosity]) – 0, 1, 2, or 3. Verbosity mode. 0 = silent, 1 = only status updates, 2 = status and brief trial results, 3 = status and detailed trial results. Defaults to 3.
progress_reporter (ProgressReporter) – Progress reporter for reporting intermediate experiment progress. Defaults to CLIReporter if running in command-line, or JupyterNotebookReporter if running in a Jupyter notebook.
log_to_file (bool|str|Sequence) – Log stdout and stderr to files in Tune’s trial directories. If this is False (default), no files are written. If true, outputs are written to trialdir/stdout and trialdir/stderr, respectively. If this is a single string, this is interpreted as a file relative to the trialdir, to which both streams are written. If this is a Sequence (e.g. a Tuple), it has to have length 2 and the elements indicate the files to which stdout and stderr are written, respectively.
trial_name_creator (Callable[[Trial], str]) – Optional function for generating the trial string representation.
trial_dirname_creator (Callable[[Trial], str]) – Function for generating the trial dirname. This function should take in a Trial object and return a string representing the name of the directory. The return value cannot be a path.
sync_config (SyncConfig) – Configuration object for syncing. See tune.SyncConfig.
export_formats (list) – List of formats that exported at the end of the experiment. Default is None.
max_failures (int) – Try to recover a trial at least this many times. Ray will recover from the latest checkpoint if present. Setting to -1 will lead to infinite recovery retries. Setting to 0 will disable retries. Defaults to 0.
fail_fast (bool | str) – Whether to fail upon the first error. If fail_fast=’raise’ provided, Tune will automatically raise the exception received by the Trainable. fail_fast=’raise’ can easily leak resources and should be used with caution (it is best used with ray.init(local_mode=True)).
restore (str) – Path to checkpoint. Only makes sense to set if running 1 trial. Defaults to None.
server_port (int) – Port number for launching TuneServer.
resume (str|bool) – One of “LOCAL”, “REMOTE”, “PROMPT”, “ERRORED_ONLY”, “AUTO”, or bool. “LOCAL”/True restores the checkpoint from the local experiment directory, determined by name and local_dir. “REMOTE” restores the checkpoint from upload_dir (as passed to sync_config). “PROMPT” provides the CLI feedback. False forces a new experiment. “ERRORED_ONLY” resets and reruns errored trials upon resume - previous trial artifacts will be left untouched. “AUTO” will attempt to resume from a checkpoint and otherwise start a new experiment. If resume is set but checkpoint does not exist, ValueError will be thrown.
reuse_actors (bool) – Whether to reuse actors between different trials when possible. This can drastically speed up experiments that start and stop actors often (e.g., PBT in time-multiplexing mode). This requires trials to have the same resource requirements.
trial_executor (TrialExecutor) – Manage the execution of trials.
raise_on_failed_trial (bool) – Raise TuneError if there exists failed trial (of ERROR state) when the experiments complete.
callbacks (list) – List of callbacks that will be called at different times in the training loop. Must be instances of the ray.tune.callback.Callback class. If not passed, LoggerCallback and SyncerCallback callbacks are automatically added.
max_concurrent_trials (int) – Maximum number of trials to run concurrently. Must be non-negative. If None or 0, no limit will be applied. This is achieved by wrapping the search_alg in a ConcurrencyLimiter, and thus setting this argument will raise an exception if the search_alg is already a ConcurrencyLimiter. Defaults to None.
_remote (bool) – Whether to run the Tune driver in a remote function. This is disabled automatically if a custom trial executor is passed in. This is enabled by default in Ray client mode.

Returns

Object for experiment analysis.

Return type

ExperimentAnalysis

Raises

TuneError – Any trials failed and raise_on_failed_trial is True.

PublicAPI: This API is stable across Ray releases.

tune.run_experiments¶

ray.tune.run_experiments(experiments: Union[ray.tune.experiment.Experiment, Mapping, Sequence[Union[ray.tune.experiment.Experiment, Mapping]]], scheduler: Optional[ray.tune.schedulers.trial_scheduler.TrialScheduler] = None, server_port: Optional[int] = None, verbose: Union[int, ray.tune.utils.log.Verbosity] = Verbosity.V3_TRIAL_DETAILS, progress_reporter: Optional[ray.tune.progress_reporter.ProgressReporter] = None, resume: bool = False, reuse_actors: bool = False, trial_executor: Optional[ray.tune.ray_trial_executor.RayTrialExecutor] = None, raise_on_failed_trial: bool = True, concurrent: bool = True, queue_trials: Optional[bool] = None, callbacks: Optional[Sequence[ray.tune.callback.Callback]] = None, _remote: Optional[bool] = None)[source]¶

Runs and blocks until all trials finish.

Examples

>>> experiment_spec = Experiment("experiment", my_func)
>>> run_experiments(experiments=experiment_spec)

>>> experiment_spec = {"experiment": {"run": my_func}}
>>> run_experiments(experiments=experiment_spec)

Returns: List of Trial objects, holding data for each executed trial.

PublicAPI: This API is stable across Ray releases.

tune.Experiment¶

ray.tune.Experiment(name, run, stop=None, time_budget_s=None, config=None, resources_per_trial=None, num_samples=1, local_dir=None, _experiment_checkpoint_dir: Optional[str] = None, sync_config=None, trial_name_creator=None, trial_dirname_creator=None, log_to_file=False, checkpoint_freq=0, checkpoint_at_end=False, keep_checkpoints_num=None, checkpoint_score_attr=None, export_formats=None, max_failures=0, restore=None)[source]¶

Tracks experiment specifications.

Implicitly registers the Trainable if needed. The args here take the same meaning as the arguments defined tune.py:run.

experiment_spec = Experiment(
    "my_experiment_name",
    my_func,
    stop={"mean_accuracy": 100},
    config={
        "alpha": tune.grid_search([0.2, 0.4, 0.6]),
        "beta": tune.grid_search([1, 2]),
    },
    resources_per_trial={
        "cpu": 1,
        "gpu": 0
    },
    num_samples=10,
    local_dir="~/ray_results",
    checkpoint_freq=10,
    max_failures=2)

Parameters

TODO (xwjiang) – Add the whole list.
_experiment_checkpoint_dir – Internal use only. If present, use this as the root directory for experiment checkpoint. If not present, the directory path will be deduced from trainable name instead.

DeveloperAPI: This API may change across minor Ray releases.

tune.SyncConfig¶

ray.tune.SyncConfig(upload_dir: Optional[str] = None, syncer: Union[None, str] = 'auto', sync_on_checkpoint: bool = True, sync_period: int = 300, sync_to_cloud: Any = None, sync_to_driver: Any = None, node_sync_period: int = - 1, cloud_sync_period: int = - 1) → None[source]¶

Configuration object for syncing.

If an upload_dir is specified, both experiment and trial checkpoints will be stored on remote (cloud) storage. Synchronization then only happens via this remote storage.

Parameters

upload_dir (str) – Optional URI to sync training results and checkpoints to (e.g. s3://bucket, gs://bucket or hdfs://path). Specifying this will enable cloud-based checkpointing.
syncer (None|func|str) – Function for syncing the local_dir to and from remote storage. If string, then it must be a string template that includes {source} and {target} for the syncer to run. If not provided, it defaults to rsync for non cloud-based storage, and to standard S3, gsutil or HDFS sync commands for cloud-based storage. If set to None, no syncing will take place. Defaults to "auto" (auto detect).
sync_on_checkpoint (bool) – Force sync-down of trial checkpoint to driver (only non cloud-storage). If set to False, checkpoint syncing from worker to driver is asynchronous and best-effort. This does not affect persistent storage syncing. Defaults to True.
sync_period (int) – Syncing period for syncing between nodes.

PublicAPI: This API is stable across Ray releases.

Ray 1.12.1

Execution (tune.run, tune.Experiment)¶

tune.run¶

tune.run_experiments¶

tune.Experiment¶

tune.SyncConfig¶