Evaluation & Testing

Jury

Evaluate NLP model outputs with automated text-to-text metrics.

Status
UnmaintainedNo commits in 748 days.
Last checked
Checked · Oct 11, 2026
License
Open source
Pricing
Free tier · from $0
Visit Jury Compare with alternatives

What it is

Jury is an easy-to-use tool for evaluating Natural Language Generation (NLG) models, providing various automated text-to-text metrics to assess the quality of generated texts. It simplifies the process of measuring and comparing NLG model performance.

Jury stands out as an open-source, Python-based tool specifically tailored for evaluating NLG models using automated text-to-text metrics, offering easy integration and community-driven support.

When it fits — and when it doesn’t

A good fit for

  • Research teams needing a comprehensive set of evaluation metrics for NLG models

  • Developers integrating automated text-to-text metrics into their NLP pipelines

  • Data scientists who require easy integration and community support

Look elsewhere if

  • Projects requiring real-time performance evaluation (Jury is designed for batch processing)

  • Teams looking for a cloud-based service with managed backend operations

Known weaknesses

Each one links to the public issue, changelog or docs page it came from, where we have one.

  1. 01

    Limited language support beyond English

    Documentation and examples focus on English, with no clear support for other languages

  2. 02

    Performance issues with large datasets

    Evaluation of NLG models on large text corpora can lead to significant slowdowns or memory errors

What it does well

  • Automated text-to-text metrics for NLG evaluation

  • Easy integration into existing NLP pipelines

  • Comprehensive set of evaluation metrics

  • Community-driven and open-source development

Pricing

Free tier
YesOpen source — free to use
Paid from
$0
Model
Flat rate
Enterprise plan
No

Pricing not independently confirmed — check the vendor’s page.

Performance benchmarks

How Fast Is It?

Set up Jury

Setup steps, common gotchas and what to check after 30 days.

Open the setup guide →
Get the badge

A small badge for Jury’s README or site: its name, “Tracked on AI Navigator”, and the availability status and check month from this page. It updates as the catalog does.

Jury — tracked on AI Navigator

Markdown

[![Jury — tracked on AI Navigator](https://navigateaitools.com/badge/jury.svg)](https://navigateaitools.com/explorer/tool/jury)

HTML

<a href="https://navigateaitools.com/explorer/tool/jury"><img src="https://navigateaitools.com/badge/jury.svg" alt="Jury — tracked on AI Navigator" height="58"></a>