Glossary

code validation scenario

A scenario where the task is to determine whether a given piece of code is a correct solution to a problem statement. Each problem instance consists of a problem statement and a candidate solution.

deterministic interaction protocol

An interaction protocol where the order of play is fixed and does not vary between trajectories. Deterministic protocols with a single verifier are implemented by subclassing DeterministicSingleVerifierProtocolHandler.

experiment settings

An ExperimentSettings object, which contains various settings for the experiment not relevant to reproducibility (e.g. the GPU device number and whether to use Weights & Biases).

graph isomorphism scenario

A scenario where the task is to determine whether two graphs are isomorphic.

hyper-parameter

One of the hyper-parameters.

hyper-parameters

The parameters that define the configuration of an experiment. These are implemented by the HyperParameters class and its sub-parameters classes.

An experiment should be completely reproducible from its hyper-parameters, up to hardware quirks and model API non-reproducibility.

image classification scenario

A scenario where the task is to a binary classification of images. This is available with a number of image datasets.

interaction protocol

The set of rules that govern how agents interact in the environment. This includes the names of the agents involved, the communication channels between agents, the order of play and the reward signal for each agent in each turn. Protocols are implemented by subclassing ProtocolHandler. See the guide to Creating a new protocol.

scenario

A scenario contains the specification and implementation of all of the elements of a classification task in which we want to train agents. The main components of a scenario are the dataset, the environment and the agents.

An example of such a task is the code validation scenario, where a problem statement together with a purported solution is given, and the task is to determine whether the solution is correct.

Scenarios are implemented by subclassing the base classes in the nip.scenario_base module.

scenario instance

A ScenarioInstance object, which contains all the components of the experiment. These components are instances of the classes defined in the scenario.

TensorDict

A nested dictionary of PyTorch tensors. Read the documentation.

trainer

The class that performs the optimisation steps on the agents, using the environment to generate the training data. Trainers are implemented by subclassing Trainer.