Leeky
Black box language model leakage/contamination testing tool
Pricing
Free tier
Flat rate
Adoption
↘CoolingLicense
Open Source
Data freshness
Aging · Jun 8, 2026Overview
What is Leeky?
Leeky is an open-source tool designed to test for data leakage and contamination in black-box language models, ensuring the integrity of AI systems.
Key differentiator
“Leeky stands out as an open-source tool specifically designed for testing data leakage and contamination in black-box language models, offering detailed insights into potential risks.”
Capability profile
Capability Radar
Honest assessment
Strengths & Weaknesses
↑ Strengths
↓ Weaknesses
API requires Python-specific patterns, TypeScript SDK is community-maintained
v0.1 to v0.2 migration required rewriting chain definitions
Primary development and maintenance focus on Python, other language bindings are not officially supported
GitHub issues and pull requests show low activity; few documented use cases outside of core features
Fit analysis
Who is it for?
✓ Best for
Teams developing black-box language models who need to ensure no data leakage occurs
Organizations that require rigorous testing for AI safety and compliance
Researchers studying the security aspects of machine learning systems
✕ Not a fit for
Projects where real-time performance is critical (Leeky may be resource-intensive)
Teams without access to local compute resources (requires a local setup)
Cost structure
Pricing
Free Tier
Available
Open source — free to use
Starts at
$0
Model
Flat rate
Enterprise
None
Performance benchmarks
How Fast Is It?
Ecosystem
Relationships
Works well with
Next step
Get Started with Leeky
Step-by-step setup guide with code examples and common gotchas.