Getting Started¶

Installation¶

Installing Evaluatio is quite straight forward:

pipuvPoetry

pip install evaluatio

uv add evaluatio

poetry add evaluatio

If you want to use the translation metrics (BLEU, ChrF, etc.) you need to install sacrebleu as an optional dependency. This can be done by using the following commands:

pipuvPoetry

pip install evaluatio[translation]

uv add evaluatio[translation]

poetry add evaluatio[translation]

Example usage¶

The usage of the library depends quite heavily on what tasks you are doing and what systems you are evaluating. The best place to start is therefore the tasks guides:

ASR Evaluation

Calculating WER with confidence interval¶

Evaluatio is basically a drop-in replacement for evaluate, especially for ASR related metrics:

import pandas as pd
df = pd.read_csv("inferences.csv")

from evaluatio.metrics.wer import word_error_rate, word_error_rate_ci
model_wer = word_error_rate(df["references"], df["predictions"])
model_ci = word_error_rate_ci(df["references"], df["predictions"], 5000, 0.05)

print(f"Model WER: {model_wer}")
print(f"Model CI:  {model_ci.mean:.3f} ± {model_ci.upper-model_ci.mean:.3f}")