AI/ML●Mid
Proposal for a real long-term AI memory benchmark
Audited LoCoMo and found 6.4% of answer keys are wrong—benchmarks are broken.
Bold Bet
dial481
402mo ago
Back to browse
Tiny long-memory benchmark with Harbor running across Islo sandboxes
by zozo123-IB·May 12, 2026·2 points·0 comments
AI Analysis
●●SolidNiche GemBig Brain
Compresses long-memory evaluation into three questions testing recall, updates, and abstention.
Strengths
- •Tests critical failure mode where keyword retrieval returns stale corrected facts.
- •Harbor task wrapper makes toy benchmark reproducible as formal eval with verifier.
- •Islo sandboxes enable parallel execution with shareable public result pages.
Weaknesses
- •Intentionally small scope limits applicability to complex real-world memory scenarios.
- •Docker dependency for Harbor tasks creates friction for local development testing.
Category
Target Audience
AI researchers evaluating long-term memory systems
Similar To
LongMemEval · AgentBench · BIG-bench
Similar Projects
Other●●Solid
BiomeSyn – a sandbox for long-term artificial evolution
Rigorous perturbation analysis finds encoding behavior is regime-dependent, not universally optimal.
Big BrainNiche Gem
yangkecoy
413mo ago
AI/ML●●Solid
Claude tournaments a Unity WebGL scene across parallel islo sandboxes
Clever use of parallel islo.dev sandboxes to let Claude vision models judge UI iterations.
WizardryRabbit Hole
zozo123-IB
301mo ago
Developer Tools●●Solid
Tiny agentic loop with Docker sandbox
Thirty-line agent loop with Docker sandboxing contains blast radius safely.
CozyBig Brain
everlier
101mo ago
AI/ML●●●Banger
Remoroo. trying to fix memory in long-running coding agents
Demand-paging memory for agents beats context window limits that break Cursor and Devin.
Big BrainWizardry
adhamghazali
302mo ago
AI/ML●●Solid
MemReader: From Passive to Active Extraction for Long-Term Agent Memory
Active memory extraction with GRPO beats passive transcription on LOCOMO benchmarks.
Big BrainNiche Gem
MemTensor
401mo ago