Skip to content

Multiple Testing

evaluatio.inference.multiple_testing

MultipleTestingResult dataclass

Result of a multiple testing correction procedure.

Attributes:

Name Type Description
rejected ndarray of bool

Boolean array in the same order as the input p-values. True indicates the null hypothesis is rejected at the corrected threshold.
adjusted_pvalues ndarray of float

Corrected p-values in the same order as the input p-values.
method str

Name of the correction method applied, e.g. 'holm'.
alpha float

The familywise error rate used.

bonferroni_correction

Apply Bonferroni correction to a set of p-values.

Included for completeness. Holm correction is preferred in almost all cases as it is uniformly more powerful while providing the same familywise error rate control.

Parameters:

Name Type Description Default
pvalues array-like of float

P-values to correct. Order is preserved in the output arrays. Must all be in the range (0, 1).

 required
alpha float

Familywise error rate threshold. Default is 0.05.

 0.05

Returns:

Type Description
MultipleTestingResult

Results dataclass. Fields: rejected (ndarray of bool, same order as input), adjusted_pvalues (ndarray of float), method (str, in this case always "bonferroni"), alpha (float). See MultipleTestingResult for full documentation.

Raises:

Type Description
ValueError

If any p-value is outside (0, 1) or if alpha is not in (0, 1).
See Also

holm_correction : Less conservative alternative.

References
  • Bonferroni, C. (1936). Teoria statistica delle classi e calcolo delle probabilita. Pubblicazioni del R. Istituto superiore di scienze economiche e commericiali di Firenze, 8, 3-62.

Examples:

>>> from evaluatio.inference.multiple_testing import bonferroni_correction
>>> pvalues = [0.03, 0.04, 0.001, 0.8, 0.02]
>>> result = bonferroni_correction(pvalues, alpha=0.05)
>>> result.rejected
array([False, False,  True, False, False])
>>> result.adjusted_pvalues
array([0.15, 0.2, 0.005, 1.0, 0.1])

holm_correction

Apply Holm-Bonferroni correction to a set of p-values.

Parameters:

Name Type Description Default
pvalues array-like of float

P-values to correct. Order is preserved in the output arrays. Must all be in the range (0, 1).

 required
alpha float

Familywise error rate threshold. Default is 0.05.

 0.05

Returns:

Type Description
MultipleTestingResult

Results dataclass. Fields: rejected (ndarray of bool, same order as input), adjusted_pvalues (ndarray of float), method (str, in this case always "holm"), alpha (float). See MultipleTestingResult for full documentation.

Raises:

Type Description
ValueError

If any p-value is outside (0, 1) or if alpha is not in [0, 1].
See Also

bonferroni_correction : More conservative alternative.

Notes

Holm correction1 controls the familywise error rate (FWER) under any dependence structure between tests. It is uniformly more powerful than Bonferroni correction and should be preferred in almost all cases.

  1. Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65-70. ↩

References
  • Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65-70.

Examples:

>>> from evaluatio.inference.multiple_testing import holm_correction
>>> pvalues = [0.03, 0.04, 0.001, 0.8, 0.02]
>>> result = holm_correction(pvalues, alpha=0.05)
>>> result.rejected
array([ True,  True,  True, False,  True])
>>> result.adjusted_pvalues
array([0.09, 0.09, 0.005, 0.8, 0.09])