AI/ML●●●Banger
AgentOS- a memory system for AI agents that learns what it doesn't know
82% token reduction with regex state machine, no vector DB required.
Big BrainDark HorseZero to One
ajstars
204mo ago
Back to browse
GitHub Repository
Tiered optimizer state allocation for memory-efficient MoE training. Cuts optimizer memory by 97.4%, outperforming AdamW/Muon/Lion while fitting a 6.78B MoE on a single 40GB GPU.
1 starsPython
SkewAdam – A tiered optimizer that cuts MoE state memory by 97%
by anonnumaan·Jul 22, 2026·2 points·0 comments
AI Analysis
●●●BangerWizardryBig Brain
Fits a 6.78B MoE on a single 40GB GPU by skewing optimizer state.
Strengths
- •Cuts optimizer state from 50GB to 1.29GB by dropping momentum on experts.
- •Achieves lower validation perplexity than AdamW and Muon on identical batches.
- •Tiered allocation treats backbone, experts, and router parameters differently.
Weaknesses
- •Implementation currently limited to PyTorch 2.x and specific MoE architectures.
- •Requires manual integration into existing training loops rather than drop-in.
Category
Target Audience
ML researchers and engineers training Mixture-of-Experts models
Similar To
DeepSpeed · FSDP · bitsandbytes
Similar Projects
Developer Tools●Mid
I built my own custom memory allocator
Educational malloc clone—solid fundamentals but competitive with textbook examples.
Niche Gem
sangramnavale
105mo ago
Productivity●Mid
Cutlistor – Instant cut list optimizer with 3D Model and PDF Import
3D model import is nice, but CutList Optimizer and MaxCut already do this.
Slick
xiyan
2028d ago
Infrastructure●●●Banger
Aethalloc – lock-free Rust memory allocator for Linux
O(1) anti-hoarding with atomic CAS beats glibc 26% on packet processing, 9x less fragmentation.
WizardryBig Brain
section_me
324mo ago
AI/ML●●●Banger
Running Gemma-4 26B at 124 tokens/SEC on a CPU, no GPU
26B model at 124 tok/s on CPU by compressing the output head, not the experts.
WizardryBig BrainZero to One
arun-prasath
10123d ago
AI/ML●●Solid
35B MoE LLM and other models locally on an old AMD crypto APU (BC250)
Kernel ttm.pages_limit workaround unlocks 16GB UMA for Vulkan inference on repurposed crypto hardware.
Dark HorseNiche Gem
akandr
204mo ago