Skip to content

Confidence Intervals

evaluatio.inference.ci

Bootstrap-based confidence interval estimation utilities.

This module provides functionality to estimate empirical confidence intervals using bootstrap resampling. It also defines a lightweight container for returning confidence interval results.

The implementation relies on optimized native bindings for performance.

ConfidenceInterval dataclass

Container for confidence interval results.

Attributes:

Name Type Description
mean float

The mean of the input sample.
lower float

The lower bound of the confidence interval.
upper float

The upper bound of the confidence interval.

bootstrap_confidence_interval

Estimate a confidence interval using bootstrap resampling

Parameters:

Name Type Description Default
x Iterable[float]

Input sample values.

 required
iterations int

Number of bootstrap resampling iterations.

 required
alpha float

Significance level for the confidence interval. For example, alpha=0.05 corresponds to a 95% confidence interval.

 required

Returns:

Type Description
ConfidenceInterval

Results dataclass. Fields: mean (float with the mean of x), lower (float, lower CI bound), upper (float, upper CI bound). See ConfidenceInterval for full documentation.

Raises:

Type Description
ValueError

If iterations < 1, if x is empty, or if alpha is not in (0, 1).
Notes
  • The confidence interval is computed empirically via bootstrap sampling.
  • The returned mean is computed from the original sample, not the bootstrap distribution.

Examples:

>>> from evaluatio.inference.ci import bootstrap_confidence_interval
>>> values = [0.4, 0.7, 0.5, 0.9]
>>> result = bootstrap_confidence_interval(values, 1000, 0.05)
>>> result
ConfidenceInterval(mean=0.625, lower=0.45, upper=0.8)

error_rate_ci

Estimate a confidence interval using bootstrap directly on error counts

Parameters:

Name Type Description Default
counts Iterable[int]

The error counts

 required
exposure Iterable[int]

A list of lengths to normalise by

 required
iterations int

Number of bootstrap resampling iterations.

 required
alpha float

Significance level for the confidence interval. For example, alpha=0.05 corresponds to a 95% confidence interval.

 required

Returns:

Type Description
ConfidenceInterval

Results dataclass. Fields: mean (float with the mean of x), lower (float, lower CI bound), upper (float, upper CI bound). See ConfidenceInterval for full documentation.

Raises:

Type Description
ValueError

If iterations < 1, if x is empty, or if alpha is not in (0, 1).
Notes
  • The confidence interval is computed empirically via bootstrap sampling of the counts and exposure.
  • The returned mean is computed from the original sample, not the bootstrap distribution.
See Also
  • word_error_rate_ci, character_error_rate_ci, and universal_error_rate_ci are all wrappers around this function.

Examples:

>>> from evaluatio.inference.ci import error_rate_confidence_interval
>>> values = [0.4, 0.7, 0.5, 0.9]
>>> result = error_rate_confidence_interval(values, 1000, 0.05)
>>> result
ConfidenceInterval(mean=0.625, lower=0.45, upper=0.8)