ACE – A dynamic benchmark measuring the cost to break AI agents

Name: ACE – A dynamic benchmark measuring the cost to break AI agents
Availability: InStock
Author: zachdotai

by zachdotai·Apr 5, 2026·9 points·3 comments

Visit Project View on HN

AI Analysis

●●SolidBig Brain

Measures AI agent security in dollars to exploit, not just binary pass or fail rates.

Strengths

•Economic framing of security vulnerabilities is smarter than static pass or fail benchmarks.
•Tests six major budget-tier models under identical agent configurations.

Weaknesses

•Methodology is early and evolving rather than a stable standard.
•Company blog post format feels like lead generation content marketing.

Post Description

We built Adversarial Cost to Exploit (ACE), a benchmark that measures the token expenditure an autonomous adversary must invest to breach an LLM agent. Instead of binary pass/fail, ACE quantifies adversarial effort in dollars, enabling game-theoretic analysis of when an attack is economically rational.

We tested six budget-tier models (Gemini Flash-Lite, DeepSeek v3.2, Mistral Small 4, Grok 4.1 Fast, GPT-5.4 Nano, Claude Haiku 4.5) with identical agent configs and an autonomous red-teaming attacker.

Haiku 4.5 was an order of magnitude harder to break than every other model; $10.21 mean adversarial cost versus $1.15 for the next most resistant (GPT-5.4 Nano). The remaining four all fell below $1.

This is early work and we know the methodology is still going to evolve. We would love nothing more than feedback from the community as we iterate on this.