nip.utils.experiments.HyperparameterExperiment#
- class nip.utils.experiments.HyperparameterExperiment(experiment_fn: Callable[[ExperimentFunctionArguments], None], param_grid: dict | None = None, run_id_fn: Callable[[RunIDFunctionArguments], str] | None = None, experiment_name: str = 'EXPERIMENT', run_preparer_fn: Callable[[dict, Namespace], PreparedExperimentInfo] | None = None, arg_parser_description: str = 'Run hyperparameter experiments', default_config_filename: str | None = None, config_file_base_path: Path | str | None = None, default_wandb_project: str | None = None, allow_resuming_wandb_run: bool = False, add_run_infix_argument: bool = True)[source]#
A base class to run an experiment over a grid of hyperparameters.
Runs each combination of hyperparameters in the grid as a separate experiment.
Methods Summary
__init__
(experiment_fn[, param_grid, ...])_load_param_grid
(config_filename)Load the hyperparameter grid from a config file.
_run
()Run the experiment.
_setup_logger
(combo_index, num_combos)Set up the logger for a single run.
Make sure there are no runs with the same ID as any run in this experiment.
run
()Run the experiment.
Attributes
combinations
An iterator over the combinations of hyperparameters.
common_run_name
A name for the experiment that is common to all runs.
enumerated_combinations
An iterator over the combinations of hyperparameters plus an enumeration.
Methods
- __init__(experiment_fn: Callable[[ExperimentFunctionArguments], None], param_grid: dict | None = None, run_id_fn: Callable[[RunIDFunctionArguments], str] | None = None, experiment_name: str = 'EXPERIMENT', run_preparer_fn: Callable[[dict, Namespace], PreparedExperimentInfo] | None = None, arg_parser_description: str = 'Run hyperparameter experiments', default_config_filename: str | None = None, config_file_base_path: Path | str | None = None, default_wandb_project: str | None = None, allow_resuming_wandb_run: bool = False, add_run_infix_argument: bool = True)[source]#
- _load_param_grid(config_filename: str)[source]#
Load the hyperparameter grid from a config file.
- Parameters:
config_filename (str) – The path to the config file given as a command line argument. If this is an absolute path, it will be used as is. If it is a relative path, it will be resolved relative to the config_file_base_path given in the constructor.
- Raises:
ValueError – If the config file format is not supported, or the content of the file does not match the expected format.
- abstract _run()[source]#
Run the experiment.
This is the function that actually runs the experiment, and should be implemented by subclasses.
- check_no_extant_runs()[source]#
Make sure there are no runs with the same ID as any run in this experiment.
- Raises:
ValueError – If there is a run with the same ID as any run in this experiment.