Analysis (tune.analysis)¶
You can use the ExperimentAnalysis
object for analyzing results. It is returned automatically when calling tune.run
.
analysis = tune.run(
trainable,
name="example-experiment",
num_samples=10,
)
Here are some example operations for obtaining a summary of your experiment:
# Get a dataframe for the last reported results of all of the trials
df = analysis.results_df
# Get a dataframe for the max accuracy seen for each trial
df = analysis.dataframe(metric="mean_accuracy", mode="max")
# Get a dict mapping {trial logdir -> dataframes} for all trials in the experiment.
all_dataframes = analysis.trial_dataframes
# Get a list of trials
trials = analysis.trials
You may want to get a summary of multiple experiments that point to the same local_dir
. For this, you can use the Analysis
class.
from ray.tune import Analysis
analysis = Analysis("~/ray_results/example-experiment")
ExperimentAnalysis (tune.ExperimentAnalysis)¶
-
class
ray.tune.
ExperimentAnalysis
(experiment_checkpoint_path: str, trials: Optional[List[ray.tune.trial.Trial]] = None, default_metric: Optional[str] = None, default_mode: Optional[str] = None)[source]¶ Bases:
ray.tune.analysis.experiment_analysis.Analysis
Analyze results from a Tune experiment.
To use this class, the experiment must be executed with the JsonLogger.
- Parameters
experiment_checkpoint_path (str) – Path to a json file representing an experiment state. Corresponds to Experiment.local_dir/Experiment.name/experiment_state.json
trials (list|None) – List of trials that can be accessed via analysis.trials.
default_metric (str) – Default metric for comparing results. Can be overwritten with the
metric
parameter in the respective functions.default_mode (str) – Default mode for comparing results. Has to be one of [min, max]. Can be overwritten with the
mode
parameter in the respective functions.
Example
>>> tune.run(my_trainable, name="my_exp", local_dir="~/tune_results") >>> analysis = ExperimentAnalysis( >>> experiment_checkpoint_path="~/tune_results/my_exp/state.json")
-
property
best_trial
¶ Get the best trial of the experiment
The best trial is determined by comparing the last trial results using the metric and mode parameters passed to tune.run().
If you didn’t pass these parameters, use get_best_trial(metric, mode, scope) instead.
-
property
best_config
¶ Get the config of the best trial of the experiment
The best trial is determined by comparing the last trial results using the metric and mode parameters passed to tune.run().
If you didn’t pass these parameters, use get_best_config(metric, mode, scope) instead.
-
property
best_checkpoint
¶ Get the checkpoint path of the best trial of the experiment
The best trial is determined by comparing the last trial results using the metric and mode parameters passed to tune.run().
If you didn’t pass these parameters, use get_best_checkpoint(trial, metric, mode) instead.
-
property
best_logdir
¶ Get the logdir of the best trial of the experiment
The best trial is determined by comparing the last trial results using the metric and mode parameters passed to tune.run().
If you didn’t pass these parameters, use get_best_logdir(metric, mode) instead.
-
property
best_dataframe
¶ Get the full result dataframe of the best trial of the experiment
The best trial is determined by comparing the last trial results using the metric and mode parameters passed to tune.run().
If you didn’t pass these parameters, use get_best_logdir(metric, mode) and use it to look for the dataframe in the self.trial_dataframes dict.
-
property
best_result
¶ Get the last result of the best trial of the experiment
The best trial is determined by comparing the last trial results using the metric and mode parameters passed to tune.run().
If you didn’t pass these parameters, use get_best_trial(metric, mode, scope).last_result instead.
-
property
best_result_df
¶ Get the best result of the experiment as a pandas dataframe.
The best trial is determined by comparing the last trial results using the metric and mode parameters passed to tune.run().
If you didn’t pass these parameters, use get_best_trial(metric, mode, scope).last_result instead.
-
property
results
¶ Get the last result of the all trials of the experiment
-
get_best_trial
(metric: Optional[str] = None, mode: Optional[str] = None, scope: str = 'last', filter_nan_and_inf: bool = True) → Optional[ray.tune.trial.Trial][source]¶ Retrieve the best trial object.
Compares all trials’ scores on
metric
. Ifmetric
is not specified,self.default_metric
will be used. If mode is not specified,self.default_mode
will be used. These values are usually initialized by passing themetric
andmode
parameters totune.run()
.- Parameters
metric (str) – Key for trial info to order on. Defaults to
self.default_metric
.mode (str) – One of [min, max]. Defaults to
self.default_mode
.scope (str) – One of [all, last, avg, last-5-avg, last-10-avg]. If scope=last, only look at each trial’s final step for metric, and compare across trials based on mode=[min,max]. If scope=avg, consider the simple average over all steps for metric and compare across trials based on mode=[min,max]. If scope=last-5-avg or scope=last-10-avg, consider the simple average over the last 5 or 10 steps for metric and compare across trials based on mode=[min,max]. If scope=all, find each trial’s min/max score for metric based on mode, and compare trials based on mode=[min,max].
filter_nan_and_inf (bool) – If True (default), NaN or infinite values are disregarded and these trials are never selected as the best trial.
-
get_best_config
(metric: Optional[str] = None, mode: Optional[str] = None, scope: str = 'last') → Optional[Dict][source]¶ Retrieve the best config corresponding to the trial.
Compares all trials’ scores on metric. If
metric
is not specified,self.default_metric
will be used. If mode is not specified,self.default_mode
will be used. These values are usually initialized by passing themetric
andmode
parameters totune.run()
.- Parameters
metric (str) – Key for trial info to order on. Defaults to
self.default_metric
.mode (str) – One of [min, max]. Defaults to
self.default_mode
.scope (str) – One of [all, last, avg, last-5-avg, last-10-avg]. If scope=last, only look at each trial’s final step for metric, and compare across trials based on mode=[min,max]. If scope=avg, consider the simple average over all steps for metric and compare across trials based on mode=[min,max]. If scope=last-5-avg or scope=last-10-avg, consider the simple average over the last 5 or 10 steps for metric and compare across trials based on mode=[min,max]. If scope=all, find each trial’s min/max score for metric based on mode, and compare trials based on mode=[min,max].
-
get_best_logdir
(metric: Optional[str] = None, mode: Optional[str] = None, scope: str = 'last') → Optional[str][source]¶ Retrieve the logdir corresponding to the best trial.
Compares all trials’ scores on metric. If
metric
is not specified,self.default_metric
will be used. If mode is not specified,self.default_mode
will be used. These values are usually initialized by passing themetric
andmode
parameters totune.run()
.- Parameters
metric (str) – Key for trial info to order on. Defaults to
self.default_metric
.mode (str) – One of [min, max]. Defaults to
self.default_mode
.scope (str) – One of [all, last, avg, last-5-avg, last-10-avg]. If scope=last, only look at each trial’s final step for metric, and compare across trials based on mode=[min,max]. If scope=avg, consider the simple average over all steps for metric and compare across trials based on mode=[min,max]. If scope=last-5-avg or scope=last-10-avg, consider the simple average over the last 5 or 10 steps for metric and compare across trials based on mode=[min,max]. If scope=all, find each trial’s min/max score for metric based on mode, and compare trials based on mode=[min,max].
Analysis (tune.Analysis)¶
-
class
ray.tune.
Analysis
(experiment_dir: str, default_metric: Optional[str] = None, default_mode: Optional[str] = None)[source]¶ Analyze all results from a directory of experiments.
To use this class, the experiment must be executed with the JsonLogger.
- Parameters
experiment_dir (str) – Directory of the experiment to load.
default_metric (str) – Default metric for comparing results. Can be overwritten with the
metric
parameter in the respective functions. If None but a mode was passed, the anonymous metric ray.tune.result.DEFAULT_METRIC will be used per default.default_mode (str) – Default mode for comparing results. Has to be one of [min, max]. Can be overwritten with the
mode
parameter in the respective functions.
-
dataframe
(metric: Optional[str] = None, mode: Optional[str] = None) → pandas.core.frame.DataFrame[source]¶ Returns a pandas.DataFrame object constructed from the trials.
- Parameters
metric (str) – Key for trial info to order on. If None, uses last result.
mode (str) – One of [min, max].
- Returns
Constructed from a result dict of each trial.
- Return type
pd.DataFrame
-
get_best_config
(metric: Optional[str] = None, mode: Optional[str] = None) → Optional[Dict][source]¶ Retrieve the best config corresponding to the trial.
- Parameters
metric (str) – Key for trial info to order on. Defaults to
self.default_metric
.mode (str) – One of [min, max]. Defaults to
self.default_mode
.
-
get_best_logdir
(metric: Optional[str] = None, mode: Optional[str] = None) → Optional[str][source]¶ Retrieve the logdir corresponding to the best trial.
- Parameters
metric (str) – Key for trial info to order on. Defaults to
self.default_metric
.mode (str) – One of [min, max]. Defaults to
self.default_mode
.
-
get_all_configs
(prefix: bool = False) → Dict[str, Dict][source]¶ Returns a list of all configurations.
- Parameters
prefix (bool) – If True, flattens the config dict and prepends config/.
- Returns
- Dict of all configurations of trials, indexed by
their trial dir.
- Return type
Dict[str, Dict]
-
get_trial_checkpoints_paths
(trial: ray.tune.trial.Trial, metric: Optional[str] = None) → List[Tuple[str, numbers.Number]][source]¶ Gets paths and metrics of all persistent checkpoints of a trial.
- Parameters
trial (Trial) – The log directory of a trial, or a trial instance.
metric (str) – key for trial info to return, e.g. “mean_accuracy”. “training_iteration” is used by default if no value was passed to
self.default_metric
.
- Returns
List of [path, metric] for all persistent checkpoints of the trial.
-
get_best_checkpoint
(trial: ray.tune.trial.Trial, metric: Optional[str] = None, mode: Optional[str] = None) → Optional[str][source]¶ Gets best persistent checkpoint path of provided trial.
- Parameters
trial (Trial) – The log directory of a trial, or a trial instance.
metric (str) – key of trial info to return, e.g. “mean_accuracy”. “training_iteration” is used by default if no value was passed to
self.default_metric
.mode (str) – One of [min, max]. Defaults to
self.default_mode
.
- Returns
Path for best checkpoint of trial determined by metric
-
get_last_checkpoint
(trial=None, metric='training_iteration', mode='max')[source]¶ Helper function that wraps Analysis.get_best_checkpoint(). Gets the last persistent checkpoint path of the provided trial, i.e., with the highest “training_iteration”. If no trial is specified, it loads the best trial according to the provided metric and mode (defaults to max. training iteration).
- Parameters
trial (Trial) – The log directory or an instance of a trial.
None, load the latest trial automatically. (If) –
metric (str) – If no trial is specified, use this metric to identify
best trial and load the last checkpoint from this trial. (the) –
mode (str) – If no trial is specified, use the metric and this mode
identify the best trial and load the last checkpoint from it. (to) –
- Returns
Path for last checkpoint of trial
-
property
trial_dataframes
¶ List of all dataframes of the trials.