nip.utils.plotting.decision_agreement.analyse_decision_agreement

nip.utils.plotting.decision_agreement.analyse_decision_agreement#

nip.utils.plotting.decision_agreement.analyse_decision_agreement(rollouts: list[NestedArrayDict], use_tqdm: bool = True) → DecisionAgreementAnalysis[source]#

Analyse to what extent the datapoint decisions of rollouts agree.

This function computes various statistics about to what extent the verifiers in the rollouts agree on the decisions they make for the datapoints.

Notes

  • This function currently only works for rollouts generated by pure-text trainers, where rollouts are stored in a NestedArrayDict.

  • The function assumes that the rollouts all have the same datapoints (but not necessarily in the same order). It will raise an error if the datapoint IDs are not the same across all rollouts.

Parameters:

  • rollouts (list[NestedArrayDict]) – The rollouts to be analysed. Each rollout is a NestedArrayDict containing the datapoints and their corresponding decisions.

  • use_tqdm (bool, default=True) – Whether to use tqdm to show a progress bar.

Returns:

DecisionAgreementAnalysis – An object containing the results of the analysis.

Raises:

ValueError – If the rollouts do not have the same datapoint IDs in the same order.