Evaluation & Testing

LLM-Evals-Catalogue

Catalog of evaluation metrics for large language models

Status
UnmaintainedNo commits in 1014 days.
Last checked
Checked · Oct 12, 2026
License
Open source
Pricing
Free tier · from $0
Visit LLM-Evals-Catalogue Compare with alternatives

What it is

A comprehensive catalogue of evaluation metrics and benchmarks for assessing the performance of large language models, aiding in model selection and improvement.

LLM-Evals-Catalogue stands out by providing a comprehensive and standardized set of evaluation metrics specifically tailored for large language models, enabling more informed model selection and improvement decisions.

When it fits — and when it doesn’t

A good fit for

  • Data science teams looking for a standardized way to evaluate and compare large language models

  • Researchers who need a comprehensive set of benchmarks for their studies

  • Machine learning practitioners aiming to improve model performance through systematic evaluation

Look elsewhere if

  • Teams needing real-time streaming evaluations (batch-only architecture)

  • Projects with limited computational resources, as some evaluations may be resource-intensive

Known weaknesses

Each one links to the public issue, changelog or docs page it came from, where we have one.

  1. 01

    Documentation lacks depth and examples

    Core documentation focuses on API reference but lacks practical use cases and tutorials

What it does well

  • Comprehensive catalogue of evaluation metrics and benchmarks

  • Facilitates model comparison and improvement through standardized evaluations

  • Open-source, allowing for community contributions and improvements

Pricing

Free tier
YesOpen source — free to use
Paid from
$0
Model
Flat rate
Enterprise plan
No

Pricing not independently confirmed — check the vendor’s page.

Performance benchmarks

How Fast Is It?

Set up LLM-Evals-Catalogue

Setup steps, common gotchas and what to check after 30 days.

Open the setup guide →