Developer Tools●●●Banger
Cheddar-bench – unsupervised benchmark for coding agents
Unsupervised bug benchmark using agents as both attackers and defenders—novel scoring methodology.
Big BrainWizardryShip It
przadka
903mo ago
Back to browse
Black-box API bug detection across 7 AI systems
by riyajoshi·Jun 4, 2026·10 points·4 comments
AI Analysis
●MidSlick
Execution-based scoring with live APIs beats LLM-graded benchmarks, but they evaluated themselves.
Strengths
- •Execution-based scoring with live APIs and planted bugs is verifiable, not subjective.
- •Tests three complexity tiers across seven application domains with repeated runs.
Weaknesses
- •KushoAI created the benchmark and evaluated themselves — obvious conflict of interest.
- •AI agent benchmark space is crowded with Cognition, AI2, and countless others.
Category
Target Audience
Engineering teams evaluating AI testing tools
Similar To
Cognition Labs · AI2 · LangSmith
Similar Projects
Developer Tools●●Solid
Bbt – Black Box Testing Directly from Your Documentation
Tests live in README as plain English; clever partial parsing eliminates Gherkin boilerplate overhead.
Big BrainNiche Gem
LionelDraghi
303mo ago
Developer Tools●Mid
Assay – Found 250 bugs in LiteLLM, LobeChat via AI code verification
AI finds 250 bugs in LiteLLM, LobeChat, but no demo or accessible entry point.
Big Brain
tywellshn
313mo ago
Developer Tools●●Solid
I built a bug reporter that opens a GitHub PR to fix the bug
AI PR generation for typos and copy, but bug reporting tools already exist elsewhere.
Solve My ProblemSlickShip It
kosbay
103mo ago
Developer Tools●●Solid
We built an AI Agent to reproduce bugs
Open-sourced agent that actually attempts to reproduce GitHub bugs in your CI.
Big BrainShip It
nvoxland
1221mo ago
AI/ML●●●Banger
Agonora – Character benchmarking for the post-AI job market
Measures epistemic humility and paradox tolerance using AI-evaluated scenario responses.
Big BrainBold Bet
mw67
102mo ago