HN Showcase
Back to browse
GitHub Repository

Runtime constraint verification for AI outputs. 278 lines. Zero dependencies. Check → Score → Retry.

6 starsPython

AI-assert – Constraint verification for LLM outputs (278 lines, Python)

by kaantahti·Feb 26, 2026·1 point·0 comments
Visit ProjectView on HN

AI Analysis

●●●BangerSolve My ProblemShip It

Lightweight retry loop that improves IFEval instruction-following from 69% to 76% accuracy.

Strengths
  • •Genuinely solves a painful universal problem: LLMs routinely violate instructions (format, length, presence).
  • •278 lines, zero dependencies—small enough to audit, easy to vendor; multiplicative gate prevents averaging away failures.
  • •Measurable impact on real benchmark (IFEval): concrete +7% improvement shows this isn't theoretical.
  • •Works with any LLM (OpenAI, Anthropic, local)—no vendor lock-in, immediate utility.
Weaknesses
  • •Retries increase latency and token cost; no analysis of retry overhead vs. one-shot complexity tradeoffs provided.
  • •Constraints are procedural; no adaptive learning—feedback stays fixed even if the same constraint fails repeatedly.
Target Audience

LLM application developers, prompt engineers, AI systems requiring guaranteed output compliance

Similar To

Guardrails AI · Outlines · Pydantic BaseModel validation + retry loops

Post Description

I built a tiny library that verifies LLM outputs against constraints and retries on failure.

The core insight: LLMs don't reliably follow instructions, but you can catch failures cheaply and retry with targeted feedback. This is essentially a lightweight "process reward model" that requires zero training.

How it works: 1. Your LLM generates output 2. ai-assert checks it against constraints (length, word count, sentence count, regex, custom predicates) 3. Each constraint returns a score in [0,1] -- composite is multiplicative (zero in any = zero overall) 4. If score < threshold, retry with feedback ("Constraint X failed because Y -- regenerate") 5. Return the best-scoring attempt

On IFEval (25 instruction-following constraint types): 69.3% -> 76.2% accuracy.

278 lines. Zero dependencies. Works with any callable that takes a string and returns a string.

pip install ai-assert

https://github.com/kaantahti/ai-assert

Similar Projects