nip.utils.experiments.MultiprocessHyperparameterExperiment#
- class nip.utils.experiments.MultiprocessHyperparameterExperiment(param_grid: dict, experiment_fn: Callable[[ExperimentFunctionArguments], None], run_id_fn: Callable[[int | None, Namespace], str] | None = None, run_preparer_fn: Callable[[dict, Namespace], PreparedExperimentInfo] | None = None, experiment_name: str = 'EXPERIMENT', arg_parser_description: str = 'Run hyperparameter experiments in parallel', default_wandb_project: str | None = None, allow_resuming_wandb_run: bool = False, add_run_infix_argument: bool = True, default_num_workers: int = 1)[source]#
A class to run an experiment over a grid of hyperparameters in parallel.
Runs each combination of hyperparameters in the grid as a separate experiment using a pool of workers.
The workflow is as follows:
Call the constructor with the hyperparameter grid and the experiment function.
(Optional) Add any additional arguments to the arg parser using
self.parser.add_argument
.Call
self.run()
to run the experiment.
- Parameters:
param_grid (dict) – A dictionary mapping hyperparameter names to lists of values to try.
experiment_fn (Callable[[ExperimentFunctionArguments], None]) – A function that takes a single hyperparameter combination and runs the experiment. The arguments are specified in the
ExperimentFunctionArguments
dataclass.run_id_fn (Callable[[int, Namespace], str], optional) –
A function that takes a single hyperparameter combination and returns a unique identifier for the run. If None, the default is to use the experiment name and the combination index. It should take the form:
run_id_fn(combo_index, cmd_args)
where
combo_index
is the index of the combination in the ParameterGrid andcmd_args
is the command line arguments.run_preparer_fn (Callable[[dict, Namespace], PreparedExperimentInfo], optional) –
A function that takes a single hyperparameter combination and prepares the run for it. It should return a
PreparedExperimentInfo
instance. This is optional. It should take the form:run_preparer_fn(combo, cmd_args)
where
combo
is a single combination of hyperparameters andcmd_args
is the command line arguments.experiment_name (str, default="EXPERIMENT") – The name of the experiment.
arg_parser_description (str, default="Run hyperparameter experiments in parallel") – The description of the argument parser.
default_wandb_project (Optional[str], default=None) – The default W&B project to use. If None, the default is to use the WANDB_PROJECT environment variable.
allow_resuming_wandb_run (bool, default=False) – Whether to allow resuming a W&B run with the same ID as a run in this experiment.
add_run_infix_argument (bool, default=True) – Whether to add an argument to the parser for adding an infix to the run ID.
default_num_workers (int, default=1) – The default number of workers to use for multiprocessing.
Methods Summary
__init__
(param_grid, experiment_fn[, ...])_run
(base_logger)Run the experiment.
_task_fn
(combinations, combo_index, ...)Run a task on a single worker.
Make sure there are no runs with the same ID as any run in this experiment.
run
()Run the experiment.
Attributes
combinations
An iterator over the combinations of hyperparameters.
common_run_name
A name for the experiment that is common to all runs.
enumerated_combinations
An iterator over the combinations of hyperparameters plus an enumeration.
Methods
- __init__(param_grid: dict, experiment_fn: Callable[[ExperimentFunctionArguments], None], run_id_fn: Callable[[int | None, Namespace], str] | None = None, run_preparer_fn: Callable[[dict, Namespace], PreparedExperimentInfo] | None = None, experiment_name: str = 'EXPERIMENT', arg_parser_description: str = 'Run hyperparameter experiments in parallel', default_wandb_project: str | None = None, allow_resuming_wandb_run: bool = False, add_run_infix_argument: bool = True, default_num_workers: int = 1)[source]#
- _run(base_logger: Logger)[source]#
Run the experiment.
This is the function that actually runs the experiment, and should be implemented by subclasses.
- _task_fn(combinations: list[dict], combo_index: int, cmd_args: Namespace, base_logger: Logger, fine_grained_global_tqdm: bool, tqdm_func: Callable, global_tqdm: tqdm) bool [source]#
Run a task on a single worker.
- Parameters:
combinations (list[dict]) – The list of combinations of hyperparameters.
combo_index (int) – The index of the current combination.
cmd_args (Namespace) – The command line arguments.
base_logger (logging.Logger) – The base logger.
fine_grained_global_tqdm (bool) – Whether to update the global progress bar after each iteration. If False, the global progress bar is only updated after each experiment is finished.
tqdm_func (Callable) – The tqdm function to use in the experiment to create new progress bars. This argument is provided by
tqdm_multiprocess
.global_tqdm (tqdm) – The global progress bar. This argument is provided by
tqdm_multiprocess
.
- check_no_extant_runs()[source]#
Make sure there are no runs with the same ID as any run in this experiment.
- Raises:
ValueError – If there is a run with the same ID as any run in this experiment.