Multiple Testing

evaluatio.inference.multiple_testing¶

Functions¶

holm_correction¶

holm_correction(pvalues: Iterable[float], alpha: float=0.05) -> MultipleTestingResult

Apply Holm-Bonferroni correction to a set of p-values.

Parameters

• pvalues : array-like of float
  P-values to correct. Order is preserved in the output arrays. Must all be in the range (0, 1).

• alpha : float
  Familywise error rate threshold. Default is 0.05.

Returns

• MultipleTestingResult
  Results dataclass. Fields: rejected (ndarray of bool, same order as input), adjusted_pvalues (ndarray of float), method (str, in this case always “holm”), alpha (float). See MultipleTestingResult for full documentation.

Raises

• ValueError
  If any p-value is outside (0, 1) or if alpha is not in [0, 1].

See-Also

bonferroni_correction : More conservative alternative.

Note

Holm correction¹ controls the familywise error rate (FWER) under any dependence structure between tests. It is uniformly more powerful than Bonferroni correction and should be preferred in almost all cases.

References

1. Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65-70.

Examples

>>> from evaluatio.inference.multiple_testing import holm_correction
>>> pvalues = [0.03, 0.04, 0.001, 0.8, 0.02]
>>> result = holm_correction(pvalues, alpha=0.05)
>>> result.rejected
array([ True,  True,  True, False,  True])
>>> result.adjusted_pvalues
array([0.09, 0.09, 0.005, 0.8, 0.09])

bonferroni_correction¶

bonferroni_correction(pvalues: Iterable[float], alpha: float=0.05) -> MultipleTestingResult

Apply Bonferroni correction to a set of p-values.

Included for completeness. Holm correction is preferred in almost all cases as it is uniformly more powerful while providing the same familywise error rate control.

Parameters

• pvalues : array-like of float
  P-values to correct. Order is preserved in the output arrays. Must all be in the range (0, 1).

• alpha : float
  Familywise error rate threshold. Default is 0.05.

Returns

• MultipleTestingResult
  Results dataclass. Fields: rejected (ndarray of bool, same order as input), adjusted_pvalues (ndarray of float), method (str, in this case always “bonferroni”), alpha (float). See MultipleTestingResult for full documentation.

Raises

• ValueError
  If any p-value is outside (0, 1) or if alpha is not in (0, 1).

See-Also

holm_correction : Less conservative alternative.

References

1. Bonferroni, C. (1936). Teoria statistica delle classi e calcolo delle probabilita. Pubblicazioni del R. Istituto superiore di scienze economiche e commericiali di Firenze, 8, 3-62.

Examples

>>> from evaluatio.inference.multiple_testing import bonferroni_correction
>>> pvalues = [0.03, 0.04, 0.001, 0.8, 0.02]
>>> result = bonferroni_correction(pvalues, alpha=0.05)
>>> result.rejected
array([False, False,  True, False, False])
>>> result.adjusted_pvalues
array([0.15, 0.2, 0.005, 1.0, 0.1])

Classes¶

MultipleTestingResult¶

Result of a multiple testing correction procedure.

Attributes

• rejected : ndarray of bool
  Boolean array in the same order as the input p-values. True indicates the null hypothesis is rejected at the corrected threshold.

• adjusted_pvalues : ndarray of float
  Corrected p-values in the same order as the input p-values.

• method : str
  Name of the correction method applied, e.g. 'holm'.

• alpha : float
  The familywise error rate used.