nip.utils.experiments.SequentialHyperparameterExperiment#

class nip.utils.experiments.SequentialHyperparameterExperiment(param_grid: dict, experiment_fn: Callable[[dict, str, Namespace, Callable, LoggerAdapter, str], None], run_id_fn: Callable[[int | None, Namespace], str] | None = None, run_preparer_fn: Callable[[dict, Namespace], PreparedExperimentInfo] | None = None, experiment_name: str = 'EXPERIMENT', arg_parser_description: str = 'Run hyperparameter experiments sequentially', default_wandb_project: str | None = None, allow_resuming_wandb_run: bool = False, add_run_infix_argument: bool = True, output_width: int = 70)[source]#

A class to run an experiment over a grid of hyperparameters in sequence.

Runs each combination of hyperparameters in the grid as a separate experiment. If there is an error in one of the experiments, all subsequent experiments are skipped.

A summary of the results is printed at the end.

The workflow is as follows:

  1. Call the constructor with the hyperparameter grid and the experiment function.

  2. (Optional) Add any additional arguments to the arg parser using self.parser.add_argument.

  3. Call self.run() to run the experiment.

Parameters:
  • param_grid (dict) – A dictionary mapping hyperparameter names to lists of values to try.

  • experiment_fn (Callable[[ExperimentFunctionArguments], None]) – A function that takes a single hyperparameter combination and runs the experiment. The arguments are specified in the ExperimentFunctionArguments dataclass.

  • run_id_fn (Callable[[int, Namespace], str], optional) –

    A function that takes a single hyperparameter combination and returns a unique identifier for the run. If None, the default is to use the experiment name and the combination index. It should take the form:

    run_id_fn(combo_index, cmd_args)
    

    where combo_index is the index of the combination in the ParameterGrid and cmd_args is the command line arguments.

  • run_preparer_fn (Callable[[dict, Namespace], PreparedExperimentInfo], optional) –

    A function that takes a single hyperparameter combination and prepares the run for it. It should return a PreparedExperimentInfo instance. This is optional. It should take the form:

    run_preparer_fn(combo, cmd_args)
    

    where combo is a single combination of hyperparameters and cmd_args is the command line arguments.

  • experiment_name (str, default="EXPERIMENT") – The name of the experiment.

  • arg_parser_description (str, default="Run hyperparameter experiments sequentially") – The description of the argument parser.

  • default_wandb_project (Optional[str], default=None) – The default W&B project to use. If None, the default is to use the WANDB_PROJECT environment variable.

  • allow_resuming_wandb_run (bool, default=False) – Whether to allow resuming a W&B run with the same ID as a run in this experiment.

  • add_run_infix_argument (bool, default=True) – Whether to add an argument to the parser for adding an infix to the run ID.

  • output_width (int, default=70) – The width of the output to print (after the logging prefix).

Methods Summary

__init__(param_grid, experiment_fn[, ...])

_run(base_logger)

Run the experiment.

_run_single_experiment(combinations, combo, ...)

Run an experiment for a single combination of hyperparameters.

check_no_extant_runs()

Make sure there are no runs with the same ID as any run in this experiment.

run()

Run the experiment.

Attributes

combinations

An iterator over the combinations of hyperparameters.

common_run_name

A name for the experiment that is common to all runs.

enumerated_combinations

An iterator over the combinations of hyperparameters plus an enumeration.

Methods

__init__(param_grid: dict, experiment_fn: Callable[[dict, str, Namespace, Callable, LoggerAdapter, str], None], run_id_fn: Callable[[int | None, Namespace], str] | None = None, run_preparer_fn: Callable[[dict, Namespace], PreparedExperimentInfo] | None = None, experiment_name: str = 'EXPERIMENT', arg_parser_description: str = 'Run hyperparameter experiments sequentially', default_wandb_project: str | None = None, allow_resuming_wandb_run: bool = False, add_run_infix_argument: bool = True, output_width: int = 70)[source]#
_run(base_logger: Logger)[source]#

Run the experiment.

This is the function that actually runs the experiment, and should be implemented by subclasses.

_run_single_experiment(combinations: list[tuple[int, dict]], combo: dict, combo_index: int, cmd_args: Namespace, base_logger: Logger) bool[source]#

Run an experiment for a single combination of hyperparameters.

check_no_extant_runs()[source]#

Make sure there are no runs with the same ID as any run in this experiment.

Raises:

ValueError – If there is a run with the same ID as any run in this experiment.

run()[source]#

Run the experiment.