nip.utils.experiments.SequentialHyperparameterExperiment#
- class nip.utils.experiments.SequentialHyperparameterExperiment(param_grid: dict, experiment_fn: Callable[[dict, str, Namespace, Callable, LoggerAdapter, str], None], run_id_fn: Callable[[int | None, Namespace], str] | None = None, run_preparer_fn: Callable[[dict, Namespace], PreparedExperimentInfo] | None = None, experiment_name: str = 'EXPERIMENT', arg_parser_description: str = 'Run hyperparameter experiments sequentially', default_wandb_project: str | None = None, allow_resuming_wandb_run: bool = False, add_run_infix_argument: bool = True, output_width: int = 70)[source]#
A class to run an experiment over a grid of hyperparameters in sequence.
Runs each combination of hyperparameters in the grid as a separate experiment. If there is an error in one of the experiments, all subsequent experiments are skipped.
A summary of the results is printed at the end.
The workflow is as follows:
Call the constructor with the hyperparameter grid and the experiment function.
(Optional) Add any additional arguments to the arg parser using
self.parser.add_argument
.Call
self.run()
to run the experiment.
- Parameters:
param_grid (dict) – A dictionary mapping hyperparameter names to lists of values to try.
experiment_fn (Callable[[ExperimentFunctionArguments], None]) – A function that takes a single hyperparameter combination and runs the experiment. The arguments are specified in the
ExperimentFunctionArguments
dataclass.run_id_fn (Callable[[int, Namespace], str], optional) –
A function that takes a single hyperparameter combination and returns a unique identifier for the run. If None, the default is to use the experiment name and the combination index. It should take the form:
run_id_fn(combo_index, cmd_args)
where
combo_index
is the index of the combination in the ParameterGrid andcmd_args
is the command line arguments.run_preparer_fn (Callable[[dict, Namespace], PreparedExperimentInfo], optional) –
A function that takes a single hyperparameter combination and prepares the run for it. It should return a
PreparedExperimentInfo
instance. This is optional. It should take the form:run_preparer_fn(combo, cmd_args)
where
combo
is a single combination of hyperparameters andcmd_args
is the command line arguments.experiment_name (str, default="EXPERIMENT") – The name of the experiment.
arg_parser_description (str, default="Run hyperparameter experiments sequentially") – The description of the argument parser.
default_wandb_project (Optional[str], default=None) – The default W&B project to use. If None, the default is to use the WANDB_PROJECT environment variable.
allow_resuming_wandb_run (bool, default=False) – Whether to allow resuming a W&B run with the same ID as a run in this experiment.
add_run_infix_argument (bool, default=True) – Whether to add an argument to the parser for adding an infix to the run ID.
output_width (int, default=70) – The width of the output to print (after the logging prefix).
Methods Summary
__init__
(param_grid, experiment_fn[, ...])_run
(base_logger)Run the experiment.
_run_single_experiment
(combinations, combo, ...)Run an experiment for a single combination of hyperparameters.
Make sure there are no runs with the same ID as any run in this experiment.
run
()Run the experiment.
Attributes
combinations
An iterator over the combinations of hyperparameters.
common_run_name
A name for the experiment that is common to all runs.
enumerated_combinations
An iterator over the combinations of hyperparameters plus an enumeration.
Methods
- __init__(param_grid: dict, experiment_fn: Callable[[dict, str, Namespace, Callable, LoggerAdapter, str], None], run_id_fn: Callable[[int | None, Namespace], str] | None = None, run_preparer_fn: Callable[[dict, Namespace], PreparedExperimentInfo] | None = None, experiment_name: str = 'EXPERIMENT', arg_parser_description: str = 'Run hyperparameter experiments sequentially', default_wandb_project: str | None = None, allow_resuming_wandb_run: bool = False, add_run_infix_argument: bool = True, output_width: int = 70)[source]#
- _run(base_logger: Logger)[source]#
Run the experiment.
This is the function that actually runs the experiment, and should be implemented by subclasses.
- _run_single_experiment(combinations: list[tuple[int, dict]], combo: dict, combo_index: int, cmd_args: Namespace, base_logger: Logger) bool [source]#
Run an experiment for a single combination of hyperparameters.
- check_no_extant_runs()[source]#
Make sure there are no runs with the same ID as any run in this experiment.
- Raises:
ValueError – If there is a run with the same ID as any run in this experiment.