Evaluation & Testing

Leeky

Black box language model leakage/contamination testing tool

Status
UnmaintainedNo commits in 910 days.
Last checked
Checked · Oct 11, 2026
License
Open source
Pricing
Free tier · from $0
Visit Leeky Compare with alternatives

What it is

Leeky is an open-source tool designed to test for data leakage and contamination in black-box language models, ensuring the integrity of AI systems.

Leeky stands out as an open-source tool specifically designed for testing data leakage and contamination in black-box language models, offering detailed insights into potential risks.

When it fits — and when it doesn’t

A good fit for

  • Teams developing black-box language models who need to ensure no data leakage occurs

  • Organizations that require rigorous testing for AI safety and compliance

  • Researchers studying the security aspects of machine learning systems

Look elsewhere if

  • Projects where real-time performance is critical (Leeky may be resource-intensive)

  • Teams without access to local compute resources (requires a local setup)

Known weaknesses

Each one links to the public issue, changelog or docs page it came from, where we have one.

  1. 01

    Limited language support beyond Python

    Primary development and maintenance focus on Python, other language bindings are not officially supported

  2. 02

    Small community and limited third-party integrations

    GitHub issues and pull requests show low activity; few documented use cases outside of core features

What it does well

  • Detects data leakage in black-box language models

  • Provides detailed reports on contamination risks

  • Flexible configuration for various testing scenarios

Pricing

Free tier
YesOpen source — free to use
Paid from
$0
Model
Flat rate
Enterprise plan
No

Pricing not independently confirmed — check the vendor’s page.

Performance benchmarks

How Fast Is It?

Set up Leeky

Setup steps, common gotchas and what to check after 30 days.

Open the setup guide →