Evaluation & TestingHallucination Leaderboard
LLM Performance Comparison for Document Summarization Hallucinations
- Status
- Generally available
- Last checked
- Aging · Sep 5, 2026
- License
- Open source
- Pricing
- Free tier · from $0
What it is
The Hallucination Leaderboard evaluates and compares the performance of Large Language Models in producing hallucinations when summarizing short documents, providing insights into model reliability.
The Hallucination Leaderboard is the only open-source tool dedicated to evaluating and comparing LLMs specifically on their tendency to produce hallucinations when summarizing short documents.
When it fits — and when it doesn’t
A good fit for
Researchers studying the reliability and accuracy of LLMs in document summarization tasks
Teams needing a tool to benchmark different language models for hallucination tendencies
Developers who want an open-source solution to evaluate model performance
Look elsewhere if
Projects requiring real-time evaluation or monitoring of hallucinations
Users looking for a cloud-based service without self-hosting capabilities
Known weaknesses
Each one links to the public issue, changelog or docs page it came from, where we have one.
- 01
Limited language support beyond Python
Primary development and documentation focus on Python, limited SDKs for other languages
- 02
Complex setup process for self-hosting
Requires setting up multiple dependencies and services, detailed configuration steps
What it does well
Evaluates LLM performance in document summarization hallucinations
Provides a leaderboard for comparing different models
Open-source and self-hosted
Pricing
- Free tier
- YesOpen source — free to use
- Paid from
- $0
- Model
- Flat rate
- Enterprise plan
- No
Pricing not independently confirmed — check the vendor’s page.
Performance benchmarks
How Fast Is It?
Set up Hallucination Leaderboard
Setup steps, common gotchas and what to check after 30 days.