Contents

Analysis (tune.analysis)

You can use the ExperimentAnalysis object for analyzing results. It is returned automatically when calling tune.run.

analysis = tune.run(
    trainable,
    name="example-experiment",
    num_samples=10,
)

Here are some example operations for obtaining a summary of your experiment:

# Get a dataframe for the last reported results of all of the trials
df = analysis.results_df

# Get a dataframe for the max accuracy seen for each trial
df = analysis.dataframe(metric="mean_accuracy", mode="max")

# Get a dict mapping {trial logdir -> dataframes} for all trials in the experiment.
all_dataframes = analysis.trial_dataframes

# Get a list of trials
trials = analysis.trials

You may want to get a summary of multiple experiments that point to the same local_dir. This is also supported by the ExperimentAnalysis class.

from ray.tune import ExperimentAnalysis
analysis = ExperimentAnalysis("~/ray_results/example-experiment")

ExperimentAnalysis (tune.ExperimentAnalysis)

class ray.tune.ExperimentAnalysis(experiment_checkpoint_path: str, trials: Optional[List[ray.tune.trial.Trial]] = None, default_metric: Optional[str] = None, default_mode: Optional[str] = None, sync_config: Optional[ray.tune.syncer.SyncConfig] = None)[source]

Analyze results from a Tune experiment.

To use this class, the experiment must be executed with the JsonLogger.

Parameters

  • experiment_checkpoint_path (str) – Path to a json file or directory representing an experiment state, or a directory containing multiple experiment states (a run’s local_dir). Corresponds to Experiment.local_dir/Experiment.name/ experiment_state.json

  • trials (list|None) – List of trials that can be accessed via analysis.trials.

  • default_metric (str) – Default metric for comparing results. Can be overwritten with the metric parameter in the respective functions.

  • default_mode (str) – Default mode for comparing results. Has to be one of [min, max]. Can be overwritten with the mode parameter in the respective functions.

Example

>>> tune.run(my_trainable, name="my_exp", local_dir="~/tune_results")
>>> analysis = ExperimentAnalysis(
>>>     experiment_checkpoint_path="~/tune_results/my_exp/state.json")

PublicAPI (beta): This API is in beta and may change before becoming stable.

property best_trial: ray.tune.trial.Trial

Get the best trial of the experiment

The best trial is determined by comparing the last trial results using the metric and mode parameters passed to tune.run().

If you didn’t pass these parameters, use get_best_trial(metric, mode, scope) instead.

property best_config: Dict

Get the config of the best trial of the experiment

The best trial is determined by comparing the last trial results using the metric and mode parameters passed to tune.run().

If you didn’t pass these parameters, use get_best_config(metric, mode, scope) instead.

property best_checkpoint: ray.ml.checkpoint.Checkpoint

Get the checkpoint path of the best trial of the experiment

The best trial is determined by comparing the last trial results using the metric and mode parameters passed to tune.run().

If you didn’t pass these parameters, use get_best_checkpoint(trial, metric, mode) instead.

Returns

Checkpoint object.

property best_logdir: str

Get the logdir of the best trial of the experiment

The best trial is determined by comparing the last trial results using the metric and mode parameters passed to tune.run().

If you didn’t pass these parameters, use get_best_logdir(metric, mode) instead.

property best_dataframe: pandas.core.frame.DataFrame

Get the full result dataframe of the best trial of the experiment

The best trial is determined by comparing the last trial results using the metric and mode parameters passed to tune.run().

If you didn’t pass these parameters, use get_best_logdir(metric, mode) and use it to look for the dataframe in the self.trial_dataframes dict.

property best_result: Dict

Get the last result of the best trial of the experiment

The best trial is determined by comparing the last trial results using the metric and mode parameters passed to tune.run().

If you didn’t pass these parameters, use get_best_trial(metric, mode, scope).last_result instead.

property best_result_df: pandas.core.frame.DataFrame

Get the best result of the experiment as a pandas dataframe.

The best trial is determined by comparing the last trial results using the metric and mode parameters passed to tune.run().

If you didn’t pass these parameters, use get_best_trial(metric, mode, scope).last_result instead.

property results: Dict[str, Dict]

Get the last result of the all trials of the experiment

property results_df: pandas.core.frame.DataFrame

Get all the last results as a pandas dataframe.

property trial_dataframes: Dict[str, pandas.core.frame.DataFrame]

List of all dataframes of the trials.

dataframe(metric: Optional[str] = None, mode: Optional[str] = None) → pandas.core.frame.DataFrame[source]

Returns a pandas.DataFrame object constructed from the trials.

This function will look through all observed results of each trial and return the one corresponding to the passed metric and mode: If mode=min, it returns the result with the lowest ever observed metric for this trial (this is not necessarily the last)! For mode=max, it’s the highest, respectively. If metric=None or mode=None, the last result will be returned.

Parameters

  • metric (str) – Key for trial info to order on. If None, uses last result.

  • mode (None|str) – One of [None, “min”, “max”].

Returns

Constructed from a result dict of each trial.

Return type

pd.DataFrame

get_trial_checkpoints_paths(trial: ray.tune.trial.Trial, metric: Optional[str] = None) → List[Tuple[str, numbers.Number]][source]

Gets paths and metrics of all persistent checkpoints of a trial.

Parameters

  • trial (Trial) – The log directory of a trial, or a trial instance.

  • metric (str) – key for trial info to return, e.g. “mean_accuracy”. “training_iteration” is used by default if no value was passed to self.default_metric.

Returns

List of [path, metric] for all persistent checkpoints of the trial.

get_best_checkpoint(trial: ray.tune.trial.Trial, metric: Optional[str] = None, mode: Optional[str] = None) → Optional[ray.ml.checkpoint.Checkpoint][source]

Gets best persistent checkpoint path of provided trial.

Parameters

  • trial (Trial) – The log directory of a trial, or a trial instance.

  • metric (str) – key of trial info to return, e.g. “mean_accuracy”. “training_iteration” is used by default if no value was passed to self.default_metric.

  • mode (str) – One of [min, max]. Defaults to self.default_mode.

Returns

Checkpoint object.

get_all_configs(prefix: bool = False) → Dict[str, Dict][source]

Returns a list of all configurations.

Parameters

prefix (bool) – If True, flattens the config dict and prepends config/.

Returns

Dict of all configurations of trials, indexed by

their trial dir.

Return type

Dict[str, Dict]

get_best_trial(metric: Optional[str] = None, mode: Optional[str] = None, scope: str = 'last', filter_nan_and_inf: bool = True) → Optional[ray.tune.trial.Trial][source]

Retrieve the best trial object.

Compares all trials’ scores on metric. If metric is not specified, self.default_metric will be used. If mode is not specified, self.default_mode will be used. These values are usually initialized by passing the metric and mode parameters to tune.run().

Parameters

  • metric (str) – Key for trial info to order on. Defaults to self.default_metric.

  • mode (str) – One of [min, max]. Defaults to self.default_mode.

  • scope (str) – One of [all, last, avg, last-5-avg, last-10-avg]. If scope=last, only look at each trial’s final step for metric, and compare across trials based on mode=[min,max]. If scope=avg, consider the simple average over all steps for metric and compare across trials based on mode=[min,max]. If scope=last-5-avg or scope=last-10-avg, consider the simple average over the last 5 or 10 steps for metric and compare across trials based on mode=[min,max]. If scope=all, find each trial’s min/max score for metric based on mode, and compare trials based on mode=[min,max].

  • filter_nan_and_inf (bool) – If True (default), NaN or infinite values are disregarded and these trials are never selected as the best trial.

get_best_config(metric: Optional[str] = None, mode: Optional[str] = None, scope: str = 'last') → Optional[Dict][source]

Retrieve the best config corresponding to the trial.

Compares all trials’ scores on metric. If metric is not specified, self.default_metric will be used. If mode is not specified, self.default_mode will be used. These values are usually initialized by passing the metric and mode parameters to tune.run().

Parameters

  • metric (str) – Key for trial info to order on. Defaults to self.default_metric.

  • mode (str) – One of [min, max]. Defaults to self.default_mode.

  • scope (str) – One of [all, last, avg, last-5-avg, last-10-avg]. If scope=last, only look at each trial’s final step for metric, and compare across trials based on mode=[min,max]. If scope=avg, consider the simple average over all steps for metric and compare across trials based on mode=[min,max]. If scope=last-5-avg or scope=last-10-avg, consider the simple average over the last 5 or 10 steps for metric and compare across trials based on mode=[min,max]. If scope=all, find each trial’s min/max score for metric based on mode, and compare trials based on mode=[min,max].

get_best_logdir(metric: Optional[str] = None, mode: Optional[str] = None, scope: str = 'last') → Optional[str][source]

Retrieve the logdir corresponding to the best trial.

Compares all trials’ scores on metric. If metric is not specified, self.default_metric will be used. If mode is not specified, self.default_mode will be used. These values are usually initialized by passing the metric and mode parameters to tune.run().

Parameters

  • metric (str) – Key for trial info to order on. Defaults to self.default_metric.

  • mode (str) – One of [min, max]. Defaults to self.default_mode.

  • scope (str) – One of [all, last, avg, last-5-avg, last-10-avg]. If scope=last, only look at each trial’s final step for metric, and compare across trials based on mode=[min,max]. If scope=avg, consider the simple average over all steps for metric and compare across trials based on mode=[min,max]. If scope=last-5-avg or scope=last-10-avg, consider the simple average over the last 5 or 10 steps for metric and compare across trials based on mode=[min,max]. If scope=all, find each trial’s min/max score for metric based on mode, and compare trials based on mode=[min,max].

get_last_checkpoint(trial=None, metric='training_iteration', mode='max')[source]

Gets the last persistent checkpoint path of the provided trial, i.e., with the highest “training_iteration”.

If no trial is specified, it loads the best trial according to the provided metric and mode (defaults to max. training iteration).

Parameters

  • trial (Trial) – The log directory or an instance of a trial.

  • None (If) –

  • automatically. (load the latest trial) –

  • metric (str) – If no trial is specified, use this metric to identify

  • trial. (the best trial and load the last checkpoint from this) –

  • mode (str) – If no trial is specified, use the metric and this mode

  • it. (to identify the best trial and load the last checkpoint from) –

Returns

Path for last checkpoint of trial

fetch_trial_dataframes() → Dict[str, pandas.core.frame.DataFrame][source]

Fetches trial dataframes from files.

Returns

A dictionary containing “trial dir” to Dataframe.

stats() → Dict[source]

Returns a dictionary of the statistics of the experiment.

If experiment_checkpoint_path pointed to a directory of experiments, the dict will be in the format of {experiment_session_id: stats}.

set_filetype(file_type: Optional[str] = None)[source]

Overrides the existing file type.

Parameters

file_type (str) – Read results from json or csv files. Has to be one of [None, json, csv]. Defaults to csv.

runner_data() → Dict[source]

Returns a dictionary of the TrialRunner data.

If experiment_checkpoint_path pointed to a directory of experiments, the dict will be in the format of {experiment_session_id: TrialRunner_data}.

TrialCheckpoint (tune.cloud.TrialCheckpoint)

class ray.tune.cloud.TrialCheckpoint(local_path: Optional[str] = None, cloud_path: Optional[str] = None)[source]

DEPRECATED: This API is deprecated and may be removed in future Ray releases.

download(cloud_path: Optional[str] = None, local_path: Optional[str] = None, overwrite: bool = False) → str[source]

Download checkpoint from cloud.

This will fetch the checkpoint directory from cloud storage and save it to local_path.

If a local_path argument is provided and self.local_path is unset, it will be set to local_path.

Parameters

  • cloud_path (Optional[str]) – Cloud path to load checkpoint from. Defaults to self.cloud_path.

  • local_path (Optional[str]) – Local path to save checkpoint at. Defaults to self.local_path.

  • overwrite (bool) – If True, overwrites potential existing local checkpoint. If False, exits if self.local_dir already exists and has files in it.

upload(cloud_path: Optional[str] = None, local_path: Optional[str] = None, clean_before: bool = False)[source]

Upload checkpoint to cloud.

This will push the checkpoint directory from local storage to cloud_path.

If a cloud_path argument is provided and self.cloud_path is unset, it will be set to cloud_path.

Parameters

  • cloud_path (Optional[str]) – Cloud path to load checkpoint from. Defaults to self.cloud_path.

  • local_path (Optional[str]) – Local path to save checkpoint at. Defaults to self.local_path.

  • clean_before (bool) – If True, deletes potentially existing cloud bucket before storing new data.

save(path: Optional[str] = None, force_download: bool = False)[source]

Save trial checkpoint to directory or cloud storage.

If the path is a local target and the checkpoint already exists on local storage, the local directory is copied. Else, the checkpoint is downloaded from cloud storage.

If the path is a cloud target and the checkpoint does not already exist on local storage, it is downloaded from cloud storage before. That way checkpoints can be transferred across cloud storage providers.

Parameters

  • path (Optional[str]) – Path to save checkpoint at. If empty, the default cloud storage path is saved to the default local directory.

  • force_download (bool) – If True, forces (re-)download of the checkpoint. Defaults to False.

previous

Stopping mechanisms (tune.stopper)

next

Console Output (Reporters)