LLM-Evals-Catalogue
Catalog of evaluation metrics for large language models
Pricing
Free tier
Flat rate
Adoption
↘CoolingLicense
Open Source
Data freshness
Aging · Jun 8, 2026Overview
What is LLM-Evals-Catalogue?
A comprehensive catalogue of evaluation metrics and benchmarks for assessing the performance of large language models, aiding in model selection and improvement.
Key differentiator
“LLM-Evals-Catalogue stands out by providing a comprehensive and standardized set of evaluation metrics specifically tailored for large language models, enabling more informed model selection and improvement decisions.”
Capability profile
Capability Radar
Honest assessment
Strengths & Weaknesses
↑ Strengths
↓ Weaknesses
API requires Python-specific patterns, TypeScript SDK is community-maintained
v0.1 to v0.2 migration required rewriting chain definitions
Primary support is for Python, limited official SDKs for other languages
Core documentation focuses on API reference but lacks practical use cases and tutorials
Fit analysis
Who is it for?
✓ Best for
Data science teams looking for a standardized way to evaluate and compare large language models
Researchers who need a comprehensive set of benchmarks for their studies
Machine learning practitioners aiming to improve model performance through systematic evaluation
✕ Not a fit for
Teams needing real-time streaming evaluations (batch-only architecture)
Projects with limited computational resources, as some evaluations may be resource-intensive
Cost structure
Pricing
Free Tier
Available
Open source — free to use
Starts at
$0
Model
Flat rate
Enterprise
None
Performance benchmarks
How Fast Is It?
Ecosystem
Relationships
Works well with
Next step
Get Started with LLM-Evals-Catalogue
Step-by-step setup guide with code examples and common gotchas.