nip.scenario_base.rollout_analysis.RolloutAnalyser#

class nip.scenario_base.rollout_analysis.RolloutAnalyser(hyper_params: HyperParameters, settings: ExperimentSettings, protocol_handler: ProtocolHandler)[source]#

Base class for analysing rollouts.

Parameters:

Methods Summary

__init__(hyper_params, settings, ...)

forward(rollouts[, use_tqdm])

Evaluate the rollouts.

relevant_agents_and_channels()

Return an iterator over agent names and channel names to be analysed.

Attributes

system_prompt_template_filename

The filename of the system prompt template.

name

Methods

__init__(hyper_params: HyperParameters, settings: ExperimentSettings, protocol_handler: ProtocolHandler)[source]#
abstract forward(rollouts: NestedArrayDict | TensorDictBase, use_tqdm: bool = False) dict[tuple[str, str], Any][source]#

Evaluate the rollouts.

Parameters:
  • rollouts (NestedArrayDict | TensorDictBase) – The rollouts to evaluate.

  • use_tqdm (bool) – Whether to use tqdm for progress bars.

Returns:

evaluations (dict[tuple[str, str], Any]) – The evaluations. A dictionary indexed by agent name and channel name, where evaluations[agent_name, channel_name] is the evaluations.

abstract relevant_agents_and_channels() Iterator[tuple[str, str]][source]#

Return an iterator over agent names and channel names to be analysed.

Yields:
  • agent_name (str) – The name of the agent.

  • channel_name (str) – The name of the channel.