Skip to content

Evaluatio Changelog

Welcome to the changelog of the library. The Captain's log if you will! 🧑‍✈️

Upcoming release

  • Changed all documentation over from MyST to Zensical
    • API reference pages should now be better, and internal links to the API should work.
    • Docstrings had to be changed to accommodate the change
    • References now has to be added manually rather than using Bibtex.

"Evaluation is creation: hear it, you creators! Evaluating is itself the most valuable treasure of all that we value. It is only through evaluation that value exists: and without evaluation the nut of existence would be hollow. Hear it, you creators!"

-- Friedrich Nietzsche, Thus Spoke Zarathustra (1883)

Release v0.5.2

  • Exposed the optimal_alignment function through the Python API and fixed attribute errors.
  • Minor documentation fixes

Release v0.5.1

  • Exposed the error_rate_ci function through the Python API

Release v0.5.0 (babel-fish)

This release is one of the most substantial yet. While the release is notable for introducing machine translation metrics and documentation, but there are significant other improvements, especially to evaluatio-docs and the documentation more generally.

"All explicit knowledge is translated knowledge, and all translation is imperfect."

-- Patrick Rothfuss, The Wise Man's Fear

Changelog

  • Added a bootstrap test for BLEU and ChrF.
  • Added a confidence interval function for BLEU and ChrF.
  • Added a general purpose permutation test.
  • Paired bootstrap test was moved to evaluatio.inference.hypothesis.
  • Significant improvements to evaluatio-docs including:
    • More extensive task guides.
    • Automatically generated API pages.
    • Pages for metrics not implemented by the library like BLEURT.
  • Added bootstrap_confidence_interval.
  • All Python modules now have extensive Numpy style docstrings.

Release v0.4.0

  • Added Bonferroni and Holm-Bonferroni corrections.
  • Added article about multiple testing.
  • Split the WER article into a separate ASR task article.

Release v0.3.0

  • Added multiprocessing for certain functions using rayon.
  • Fixed type annotations to be less restrictive.
  • Minor improvements to documentation.
  • Minor improvements to CI.

Release v0.2.0

  • evaluatio-docs now exists and is hosted on GitHub pages!
  • Typing for iterables is now more forgiving (changed List to Iterable).
  • Bootstrap based confidence intervals have been added:
    • One generic mean based one.
    • One specifically for WER due to the way corpus-level means are handled.
  • Added an independent and paired version of Cohen's d.

Release v0.1.0

Bonjour! This was the first version of the library and was basically a re-organised version of the original library universal-edit-distance.