nip.utils.experiments.MultiprocessHyperparameterExperiment

nip.utils.experiments.MultiprocessHyperparameterExperiment#

class nip.utils.experiments.MultiprocessHyperparameterExperiment(experiment_fn: Callable[[ExperimentFunctionArguments], None], param_grid: dict | None = None, run_id_fn: Callable[[RunIDFunctionArguments], str] | None = None, run_preparer_fn: Callable[[dict, Namespace], PreparedExperimentInfo] | None = None, experiment_name: str = 'EXPERIMENT', arg_parser_description: str = 'Run hyperparameter experiments in parallel', default_config_filename: str | None = None, config_file_base_path: Path | str | None = None, default_wandb_project: str | None = None, allow_resuming_wandb_run: bool = False, add_run_infix_argument: bool = True, default_num_workers: int = 1)[source]#

A class to run an experiment over a grid of hyperparameters in parallel.

Runs each combination of hyperparameters in the grid as a separate experiment using a pool of workers.

The workflow is as follows:

  1. Call the constructor with the hyperparameter grid and the experiment function.

  2. (Optional) Add any additional arguments to the arg parser using self.parser.add_argument.

  3. Call self.run() to run the experiment.

Parameters:

  • experiment_fn (Callable[[ExperimentFunctionArguments], None]) – A function that takes a single hyperparameter combination and runs the experiment. The arguments are specified in the ExperimentFunctionArguments dataclass.

  • param_grid (dict, optional) – A dictionary mapping hyperparameter names to lists of values to try. If not given, a positional argument “config_file” will be added to the parser, and this will be used to load the hyperparameter grid from a config file.

  • run_id_fn (Callable[[RunIDFunctionArguments], str], optional) – A function that takes a single hyperparameter combination and returns a unique identifier for the run. If None, the default is to use the experiment name and the combination index. The arguments are specified in the RunIDFunctionArguments dataclass.

  • run_preparer_fn (Callable[[dict, Namespace], PreparedExperimentInfo], optional) –

    A function that takes a single hyperparameter combination and prepares the run for it. It should return a PreparedExperimentInfo instance. This is optional. It should take the form:

    run_preparer_fn(combo, cmd_args)

    where combo is a single combination of hyperparameters and cmd_args is the command line arguments.

  • experiment_name (str, default="EXPERIMENT") – The name of the experiment.

  • arg_parser_description (str, default="Run hyperparameter experiments in parallel") – The description of the argument parser.

  • default_config_filename (Optional[str], default=None) – The default config filename to use if the param_grid is not given. If None, no default is set, and if the user does not provide a config file, an empty grid is used.

  • config_file_base_path (Optional[str | Path], default=None) – The base path to use for the config file. If None, the current working directory is used. If the config file is specified as a relative path, it will be resolved relative to this base path. If this is an absolute path, it will be used as is.

  • default_wandb_project (Optional[str], default=None) – The default W&B project to use. If None, the default is to use the WANDB_PROJECT environment variable.

  • allow_resuming_wandb_run (bool, default=False) – Whether to allow resuming a W&B run with the same ID as a run in this experiment.

  • add_run_infix_argument (bool, default=True) – Whether to add an argument to the parser for adding an infix to the run ID.

  • default_num_workers (int, default=1) – The default number of workers to use for multiprocessing.

Methods Summary

__init__(experiment_fn[, param_grid, ...])

_load_param_grid(config_filename)

Load the hyperparameter grid from a config file.

_run()

Run the experiment.

_setup_logger(combo_index, num_combos)

Set up the logger for a single run.

_task_fn(combinations, combo_index, ...)

Run a task on a single worker.

check_no_extant_runs()

Make sure there are no runs with the same ID as any run in this experiment.

run()

Run the experiment.

Attributes

combinations

An iterator over the combinations of hyperparameters.

common_run_name

A name for the experiment that is common to all runs.

enumerated_combinations

An iterator over the combinations of hyperparameters plus an enumeration.

Methods

__init__(experiment_fn: Callable[[ExperimentFunctionArguments], None], param_grid: dict | None = None, run_id_fn: Callable[[RunIDFunctionArguments], str] | None = None, run_preparer_fn: Callable[[dict, Namespace], PreparedExperimentInfo] | None = None, experiment_name: str = 'EXPERIMENT', arg_parser_description: str = 'Run hyperparameter experiments in parallel', default_config_filename: str | None = None, config_file_base_path: Path | str | None = None, default_wandb_project: str | None = None, allow_resuming_wandb_run: bool = False, add_run_infix_argument: bool = True, default_num_workers: int = 1)[source]#
_load_param_grid(config_filename: str)[source]#

Load the hyperparameter grid from a config file.

Parameters:

config_filename (str) – The path to the config file given as a command line argument. If this is an absolute path, it will be used as is. If it is a relative path, it will be resolved relative to the config_file_base_path given in the constructor.

Raises:

ValueError – If the config file format is not supported, or the content of the file does not match the expected format.

_run()[source]#

Run the experiment.

This is the function that actually runs the experiment, and should be implemented by subclasses.

_setup_logger(combo_index: int, num_combos: int)[source]#

Set up the logger for a single run.

Parameters:

  • combo_index (int) – The index of the current combination in the list of combinations.

  • num_combos (int) – The total number of combinations.

_task_fn(combinations: list[dict], combo_index: int, cmd_args: Namespace, fine_grained_global_tqdm: bool, tqdm_func: Callable, global_tqdm: tqdm) → bool[source]#

Run a task on a single worker.

Parameters:

  • combinations (list[dict]) – The list of combinations of hyperparameters.

  • combo_index (int) – The index of the current combination.

  • cmd_args (Namespace) – The command line arguments.

  • fine_grained_global_tqdm (bool) – Whether to update the global progress bar after each iteration. If False, the global progress bar is only updated after each experiment is finished.

  • tqdm_func (Callable) – The tqdm function to use in the experiment to create new progress bars. This argument is provided by tqdm_multiprocess.

  • global_tqdm (tqdm) – The global progress bar. This argument is provided by tqdm_multiprocess.

check_no_extant_runs()[source]#

Make sure there are no runs with the same ID as any run in this experiment.

Raises:

ValueError – If there is a run with the same ID as any run in this experiment.

run()[source]#

Run the experiment.