nip.utils.plotting.decision_agreement.DecisionAgreementAnalysis

nip.utils.plotting.decision_agreement.DecisionAgreementAnalysis#

class nip.utils.plotting.decision_agreement.DecisionAgreementAnalysis(agreement_matrix: ndarray[Any, dtype[_ScalarType_co]], positive_agreement_matrix: ndarray[Any, dtype[_ScalarType_co]], negative_agreement_matrix: ndarray[Any, dtype[_ScalarType_co]], cohen_kappa_matrix: ndarray[Any, dtype[_ScalarType_co]])[source]#

Class for storing the results of a decision agreement analysis.

Parameters:

  • agreement_matrix (NDArray) – A matrix of shape (n_rollouts, n_rollouts), which for each pair of rollouts contains the proportion of the datapoints for which the verifier makes the same decision.

  • positive_agreement_matrix (NDArray) – A matrix of shape (n_rollouts, n_rollouts), which for each pair of rollouts contains the proportion of the positively labelled datapoints for which the verifier makes the same decision.

  • negative_agreement_matrix (NDArray) – A matrix of shape (n_rollouts, n_rollouts), which for each pair of rollouts contains the proportion of the negatively labelled datapoints for which the verifier makes the same decision.

  • cohen_kappa_matrix (NDArray) – A matrix of shape (n_rollouts, n_rollouts), which for each pair of rollouts contains the Cohen’s kappa score [Coh60] for the decisions made by the verifiers. This is a measure of the agreement between two raters, corrected for chance agreement.

Methods Summary

__eq__(other)

Return self==value.

__init__(agreement_matrix, ...)

__repr__()

Return repr(self).

Attributes

agreement_matrix

positive_agreement_matrix

negative_agreement_matrix

cohen_kappa_matrix

Methods

__eq__(other)#

Return self==value.

__init__(agreement_matrix: ndarray[Any, dtype[_ScalarType_co]], positive_agreement_matrix: ndarray[Any, dtype[_ScalarType_co]], negative_agreement_matrix: ndarray[Any, dtype[_ScalarType_co]], cohen_kappa_matrix: ndarray[Any, dtype[_ScalarType_co]]) → None#
__repr__()#

Return repr(self).