Skip to content

ChrF (Character F-Score)

API reference for the chrf module. For metric and evaluation documentation, see: ChrF

evaluatio.metrics.chrf

ChrF metric wrappers providing bootstrap confidence intervals and permutation tests.

This module provides thin wrappers around sacrebleu.CHRF that add statistical inference capabilities: bootstrap confidence intervals (chrf_ci) and paired permutation significance tests (chrf_permutation_test).

Both functions operate at the corpus level but delegate the underlying ChrF arithmetic entirely to sacrebleu, ensuring scoring is identical to the reference implementation.

Notes

Sentence-level scores used in chrf_permutation_test are obtained via sacrebleu.CHRF.sentence_score; corpus-level scores and statistics used in chrf_ci are obtained via the internal _extract_corpus_statistics and _compute_score_from_stats helpers, which may change across sacrebleu versions.

chrf_ci

Compute a bootstrap confidence interval for corpus ChrF score.

Draws iterations bootstrap resamples (with replacement) from the per-sentence sufficient statistics, recomputes the corpus ChrF score for each resample, and returns the percentile-based confidence interval.

Parameters:

Name Type Description Default
references Iterable[Iterable[str]]

Reference translations. Each element is an iterable of strings representing row in a test set. Has to be the same length as hypotheses. The length of all the elements has to be the same.

 required
hypotheses Sequence[str]

Model hypotheses, one string per sentence.

 required
iterations int

Number of bootstrap resamples. Default is 1000. For publication-quality intervals, 10_000 or more is recommended.

 required
alpha float

Significance level. The returned interval covers 1 - alpha of the bootstrap distribution. E.g. 0.05 is 95 % CI.

 required
chrf CHRF or None

A pre-configured :class:sacrebleu.CHRF instance. If None (default), a default instance is created with sacrebleu's standard parameters (char_order=6, word_order=0, beta=2).

 None
seed int

Seed for the NumPy random number generator used during resampling, ensuring reproducibility. Default is 0.

 0

Returns:

Type Description
ConfidenceInterval

Raises:

Type Description
ValueError

If the reference sets do not all have the same number of sentences.

Examples:

>>> ci = chrf_ci([["the cat sat"]], ["the cat sat"])
>>> ci.score
100.0

chrf_permutation_test

Run a paired approximate permutation test comparing two sets of hypotheses.

For each sentence the ChrF score is computed independently for each hypothesis set. The resulting paired score vectors are passed to inference.hypothesis.paired_permutation_test, which estimates the probability that the observed difference in means (or its absolute value, when two_tailed=True) could arise by chance under the null hypothesis that the two systems are equivalent.

Parameters:

Name Type Description Default
references Iterable[Iterable[str]]

Reference translations. Each element is an iterable of strings representing one reference set. All reference sets must contain one string per sentence and have the same length as the hypothesis sequences.

 required
hypotheses1 Sequence[str]

Hypothesis strings for system 1, one per sentence.

 required
hypotheses2 Sequence[str]

Hypothesis strings for system 2, one per sentence.

 required
iterations int

Number of permutation iterations.

 required
two_tailed bool

If True (default), conduct a two-tailed test (H₁: systems differ). If False, conduct a one-tailed test (H₁: system 1 is better).

 True
chrf CHRF or None

A pre-configured sacrebleu.CHRF instance. If None (default), a default instance is created.

 None

Returns:

Type Description
float

Estimated p-value. Small values (e.g. < 0.05) indicate that the observed score difference is unlikely under the null hypothesis.

Raises:

Type Description
ValueError

If the reference sets do not all have the same number of sentences.

Examples:

>>> p = chrf_permutation_test(
...     [["the cat sat", "a dog ran"]],
...     ["the cat sat", "a dog ran"],
...     ["a cat sat", "the dog ran"],
... )
>>> 0.0 <= p <= 1.0
True