cv_experiment.py

Script for running code validation experiments.

This script runs through a grid of hyperparameters, specified in the param_grid dict, and runs experiments for the code validation task for each. The grid specifies the trainer to use, the interaction protocol and many other parameters.

Additional settings, like whether to log to W&B, the number of rollout workers to use, and whether to use the dummy API, can be set via command line arguments. Run the script with the --help flag to see all available arguments.

scripts/cv_experiment.py

Run Code Validation experiments, running from a hyperparameter grid in sequence.

usage: scripts/cv_experiment.py [-h] [--config-file CONFIG_FILE] [-d] [-q]
                                [--use-wandb] [--wandb-project WANDB_PROJECT]
                                [--wandb-entity WANDB_ENTITY] [--tag TAG]
                                [--gpu-num GPU_NUM] [--ignore-cache]
                                [--no-pretrain] [--combo-groups COMBO_GROUPS]
                                [--combo-num COMBO_NUM] [--num-skip NUM_SKIP]
                                [--dummy]
                                [run_infix]

run_infix

Infix to add to the run ID to distinguish between different runs. Defaults to ‘test_{time_now}’ when using dummy API; otherwise raises an error.

-h, --help

show this help message and exit

--config-file <config_file>, -c <config_file>

The path to the file containing the hyperparameter grid (relative to ‘/home/runner/work/neural-interactive-proofs/neural-interactive-proofs/scripts/config/cv_experiment’)

-d, --debug

Print debug messages

-q, --quiet

Print less output (set log level to WARNING)

--use-wandb

Whether to use W&B to log the experiment

--wandb-project <wandb_project>

The name of the W&B project to use

--wandb-entity <wandb_entity>

The name of the W&B entity to use

--tag <tag>

An optional tag for the W&B run

--gpu-num <gpu_num>

The (0-based) number of the GPU to use

--ignore-cache

Ignore the dataset and model cache and rebuild from scratch.

--no-pretrain

Don’t pretrain the agents, regardless of the hyperparameters

--combo-groups <combo_groups>

Into how many groups to split the experiment combinations

--combo-num <combo_num>

Which combo group to run this time

--num-skip <num_skip>

The number of initial combos to skip. Useful to resume a group

--dummy

Whether to use the dummy API for the agents. Useful for testing.

The config file should be a JSON, JSON5, or YAML file containing a dictionary with keys ‘kind’ and ‘parameters’. If ‘kind’ is ‘single_experiment’, then ‘parameters’ should be a dictionary with the hyperparameters to use. If ‘kind’ is ‘grid’, then ‘parameters’ should be a dictionary with keys as hyperparameter names and values as lists of values to try.