AI/ML●●●Banger
OpenCastor Agent Harness Evaluator Leaderboard
263k config search space benchmarked across robot fleets—nothing like this exists for robotics AI.
Zero to OneBig BrainNiche Gem
craigm26
313mo ago
Back to browse
GitHub Repository
An Open Benchmark for AI in Cybersecurity Operations.
23 starsPython
SOCBench – an open benchmark for AI on SoC tasks
by krmayankb·Jul 7, 2026·4 points·0 comments
AI Analysis
●●SolidBig BrainNiche Gem
Ablation testing isolates tool vs playbook contribution in AI security evals.
Strengths
- •Four personas with persona-scoped read-only tools for realistic SOC scenarios
- •Deterministic content-addressed NetFlow indexes enable reproducible benchmarks
- •tools_off and playbooks_off ablation shows what actually drives performance
Weaknesses
- •Alpha status with limited real-world validation on production SOC data
- •NetFlow-only focus misses endpoint logs, alerts, and other SOC data sources
Category
Target Audience
Security teams, AI researchers, SOC analysts
Similar To
MLCommons · HELM · Security AI benchmarks
Post Description
Similar Projects
AI/ML●●Solid
Benchmarking the Keep memory system with LoCoMo
Local-only memory benchmarking—rigorous, but LoCoMo is a standard test everyone runs.
Niche GemShip It
inguz
104mo ago
Developer Tools●●●Banger
Tracecore: Benchmark AI Agents on Deterministic Coding Tasks
Deterministic agent benchmarking with strict validation—unlike SWE-Bench, measures whether agents actually operate.
Solve My ProblemWizardryNiche Gem
extra_cookin
104mo ago
AI/ML●●Solid
RepoGauge – save token costs and compare agents on your own repos
Mining your own bugfix history for evals beats public benchmarks that don't match your codebase.
Solve My ProblemBig Brain
siliconc0w
102mo ago
Developer Tools●●●Banger
Cheddar-bench – unsupervised benchmark for coding agents
Unsupervised bug benchmark using agents as both attackers and defenders—novel scoring methodology.
Big BrainWizardryShip It
przadka
904mo ago
AI/ML●Mid
Tested 12 LLMs with few-shot examples
Research article revealing few-shot collapse patterns, not a usable tool or product.
Dark Horse
shuntaro-okuma
203mo ago