nip.utils.experiments.HyperparameterExperiment

nip.utils.experiments.HyperparameterExperiment#

class nip.utils.experiments.HyperparameterExperiment(experiment_fn: Callable[[ExperimentFunctionArguments], None], param_grid: dict | None = None, run_id_fn: Callable[[RunIDFunctionArguments], str] | None = None, experiment_name: str = 'EXPERIMENT', run_preparer_fn: Callable[[dict, Namespace], PreparedExperimentInfo] | None = None, arg_parser_description: str = 'Run hyperparameter experiments', default_config_filename: str | None = None, config_file_base_path: Path | str | None = None, default_wandb_project: str | None = None, allow_resuming_wandb_run: bool = False, add_run_infix_argument: bool = True)[source]#

A base class to run an experiment over a grid of hyperparameters.

Runs each combination of hyperparameters in the grid as a separate experiment.

Methods Summary

__init__(experiment_fn[, param_grid, ...])

_load_param_grid(config_filename)

Load the hyperparameter grid from a config file.

_run()

Run the experiment.

_setup_logger(combo_index, num_combos)

Set up the logger for a single run.

check_no_extant_runs()

Make sure there are no runs with the same ID as any run in this experiment.

run()

Run the experiment.

Attributes

combinations

An iterator over the combinations of hyperparameters.

common_run_name

A name for the experiment that is common to all runs.

enumerated_combinations

An iterator over the combinations of hyperparameters plus an enumeration.

Methods

__init__(experiment_fn: Callable[[ExperimentFunctionArguments], None], param_grid: dict | None = None, run_id_fn: Callable[[RunIDFunctionArguments], str] | None = None, experiment_name: str = 'EXPERIMENT', run_preparer_fn: Callable[[dict, Namespace], PreparedExperimentInfo] | None = None, arg_parser_description: str = 'Run hyperparameter experiments', default_config_filename: str | None = None, config_file_base_path: Path | str | None = None, default_wandb_project: str | None = None, allow_resuming_wandb_run: bool = False, add_run_infix_argument: bool = True)[source]#
_load_param_grid(config_filename: str)[source]#

Load the hyperparameter grid from a config file.

Parameters:

config_filename (str) – The path to the config file given as a command line argument. If this is an absolute path, it will be used as is. If it is a relative path, it will be resolved relative to the config_file_base_path given in the constructor.

Raises:

ValueError – If the config file format is not supported, or the content of the file does not match the expected format.

abstract _run()[source]#

Run the experiment.

This is the function that actually runs the experiment, and should be implemented by subclasses.

_setup_logger(combo_index: int, num_combos: int)[source]#

Set up the logger for a single run.

Parameters:

  • combo_index (int) – The index of the current combination in the list of combinations.

  • num_combos (int) – The total number of combinations.

check_no_extant_runs()[source]#

Make sure there are no runs with the same ID as any run in this experiment.

Raises:

ValueError – If there is a run with the same ID as any run in this experiment.

run()[source]#

Run the experiment.