Developer Tools●●Solid
We built Cobalt, Open source unit testing for AI Agents
Testing framework for AI agents with LLM judges and SQLite result tracking.
Solve My ProblemShip It
fdefitte
314mo ago
Back to browse
GitHub Repository
Independent framework to test, benchmark, and evaluate LLMs & AI agents locally.
10 starsPython
Rubric – test what your LLM agent did, not just what it said
by kareemrashed·Jun 12, 2026·1 point·0 comments
AI Analysis
●●SolidSolve My Problem
Tests tool calls and trace quality when LangSmith only checks output strings.
Strengths
- •Zero required dependencies means fully local evaluation without API calls.
- •LangGraph integration extracts tool calls from existing message structures automatically.
Weaknesses
- •Agent evaluation is crowded with LangSmith, Arize Phoenix, and others.
- •Limited to LangGraph; no native support for AutoGen or CrewAI agents.
Category
Target Audience
LLM agent developers, AI engineering teams
Similar To
LangSmith · Arize Phoenix · Braintrust
Similar Projects
Security●●●●Gem
Edictum – Runtime governance for LLM agent tool calls
Proves text safety ≠ tool-call safety; catches hidden harmful executions deterministically.
Zero to OneBig BrainWizardry
acartag7
203mo ago
Developer Tools●Mid
FC-Eval – CLI to Benchmark Local or Cloud LLMs on Function Calling
AST-based validation for function calling tests, but BFCL already covers this ground.
Ship ItNiche Gem
gauravvij137
303mo ago
Developer Tools●●●Banger
Evalcraft – cassette-based testing for AI agents (pytest, $0/run)
VCR for LLM calls—eliminates API costs and non-determinism in agent testing.
Solve My ProblemShip ItSlick
beyhang
103mo ago
Developer Tools●●●Banger
CheckAgent The open-source pytest testing framework for AI agents
pytest-native testing for AI agents with 101 built-in safety attack probes.
Solve My ProblemSlick
xydac
301mo ago
Developer Tools●●●Banger
Cobalt – Unit tests for AI agents, like Jest but for LLMs
Jest for LLMs—CI-native eval that fails builds on quality drops, not dashboards.
Ship ItSolve My ProblemBig Brain
fdefitte
303mo ago