Benchmark harness measuring AI coding tool+workflow performance, not just model capability. 100 tasks, sigmoid scoring, 12 capability dimensions, gap analysis.
Tests workflow + tool + model together, not just model capability like SWE-bench.
Engineering teams evaluating AI coding tools and workflows
SWE-bench · Aider benchmarks · Claude Code eval tools
Deterministic agent benchmarking with strict validation—unlike SWE-Bench, measures whether agents actually operate.
263k config search space benchmarked across robot fleets—nothing like this exists for robotics AI.
Opposite-narrator test catches models agreeing with both sides of same dispute.
Expands corpus to 16 CVE-anchored scenarios to break model ties.
First benchmark testing structured requirements on complex greenfield agent tasks.
AI benchmarking for jj CLI when LMSys and HuggingFace already dominate the space.
