Optimizer Analyzer¶
-
class
deepobs.analyzer.analyze_utils.
OptimizerAnalyzer
(path, opt, metric, testproblem, conv_perf)[source]¶ DeepOBS analyzer class for an optimizer (and a specific test problem).
This class will give access to all relevant information regarding this optimizer such as the best performing hyperparameter setting or the number of settings.
Parameters: - path (str) -- Path to the parent folder of the optimizer folder (i.e. the test problem folder).
- opt (str) -- Name of the optimizer (folder).
- metric (str) -- Metric to use for this test problem. If available this
will be
test_accuracies
, otherwisetest_losses
. - testproblem (str) -- Name of the test problem this optimizer (folder) belongs to.
- conv_perf (float) -- Convergence performance of the test problem this optimizer (folder) belongs to.
-
name
¶ Name of the optimizer (folder).
-
metric
¶ Metric to use for this test problem. If available this will be
test_accuracies
, otherwisetest_losses
.
-
testproblem
¶ Name of the test problem this optimizer (folder) belongs to.
-
conv_perf
¶ Convergence performance for this test problem.
-
settings
¶ Dictionary of hyperparameter settings for this optimizer (on this test problem) where the key is the name of the setting (folder) and the value is an instance of the SettingAnalyzer class (see below).
-
num_settings
¶ Total number of settings for this optimizer (and test problem)
-
get_best_setting_best
()[source]¶ Returns the setting for this optimizer that has the best overall performance using the metric (
test_losses
ortest_accuracies
) defined for this test problem. In contrast toget_best_setting_final
in not only looks at the final performance per setting, but the best performance per setting.Returns: Instance of the SettingAnalyzer class with the best overall performance Return type: SettingAnalyzer
-
get_best_setting_final
()[source]¶ Returns the setting for this optimizer that has the best final performance using the metric (
test_losses
ortest_accuracies
) defined for this test problem.Returns: Instance of the SettingAnalyzer class with the best final performance Return type: SettingAnalyzer
-
get_bm_table
(perf_table, mode='most')[source]¶ Generates the overall performance table for this optimizer.
This includes metrics for the performance, speed and tuneability of this optimizer (on this test problem).
Parameters: - perf_table (dict) -- A dictionary with three keys:
Performance
,Speed
andTuneability
. - mode (str) -- Whether to use the setting with the best final
(
final
) performance, the best overall (best
) performance or the setting with the most runs (most
). Defaults tomost
.
Returns: Dictionary with holding the performance, speed and tuneability measure for this optimizer.
Return type: dict
- perf_table (dict) -- A dictionary with three keys:
-
get_setting_most_runs
()[source]¶ Returns the setting with the most repeated runs (with the same setting, but probably different seeds).
Returns: Instance of the SettingAnalyzer class with the most repeated runs. Return type: SettingAnalyzer
-
plot_lr_sensitivity
(ax, mode='final')[source]¶ Generates the
learning rate
sensitivity plot for this optimizer. This plots the relative performance (relative to the best setting for this optimizer) against thelearning rate
used in this setting.This assumes that all settings or otherwise equal and only different in the
learning rate
.Parameters: - ax (matplotlib.axes) -- Handle to a matplotlib axis to plot the
learning rate
sensitivity onto. - mode (str) -- Whether to use the final (
final
) performance or the best (best
) when evaluating each setting. Defaults tofinal
.
- ax (matplotlib.axes) -- Handle to a matplotlib axis to plot the
-
plot_performance
(ax, mode='most')[source]¶ Generates a performance plot for this optimzer using one hyperparameter setting.
Can either use the setting with the best final performance, the best overall performance or the setting with the most runs.
This function will plot all four possible performance metrics (
test_losses
,train_losses
,test_accuracies
andtrain_accuracies
).Parameters: - ax (list) -- List of four matplotlib axis to plot the performancs metrics onto.
- mode (str) -- Whether to use the setting with the best final
(
final
) performance, the best overall (best
) performance or the setting with the most runs (most
) when plotting. Defaults tomost
.