AI/ML●●Solid
Best setup local LLM found for a 5090 (llama.cpp fork + turboquant)
450k context on 32GB VRAM using turboquant KV cache compression.
Big BrainNiche Gem
utopman
221mo ago
Back to browse
GitHub Repository
Run a 120B-parameter MoE (60 GB) on a 12 GB phone. CPU-only, lossless, on stock llama.cpp
22 starsC++
Run a 120B-parameter MoE on Android mid-range phone CPU-only llama.cpp
by Helldez·Jul 19, 2026·1 point·0 comments
AI Analysis
●●●●GemWizardryZero to OneBig Brain
Runs 60GB models on 12GB phones by streaming experts from flash, not RAM.
Strengths
- •5x memory overcommit achieved while maintaining byte-identical output to full RAM loading
- •Custom expert caching and read/compute overlap engineered for UFS storage constraints
- •Rigorous moe_gates tests verify lossless streaming against resident model baseline
Weaknesses
- •1.3 tok/s is usable but still too slow for conversational applications
- •Requires specific MoE architecture; dense models cannot use this technique
Category
Target Audience
Mobile AI developers, edge computing engineers
Similar To
llama.cpp · MLC LLM · Mediapipe Tasks
Similar Projects
AI/ML●●●Banger
Running LLM on smartwatch – found llama.cpp loading model twice in RAM
Found llama.cpp loading models twice in RAM — fixed with host_ptr, 74% reduction.
WizardryBig Brain
perinban
103mo ago
AI/ML●Mid
Loqi, a "local-first" translation tool using Ollama/llama.cpp
Local LLM translation when Argos Translate and Ollama wrappers already exist.
CozyNiche Gem
danterolle
3027d ago
AI/ML●●●Banger
TurboPrefill – Multi-GPU prefill acceleration for llama.cpp
2x prefill speedup on 12k+ token contexts by treating GPUs like a production line.
Big BrainWizardry
trykhlieb
201mo ago
Developer Tools●●Solid
Run Llama.cpp In-Process from Java with Project Panama FFM
Panama FFM beats JNI for in-process llama.cpp - no sidecar, no HTTP, no native install.
Big BrainNiche Gem
deemwar
601mo ago
Developer Tools●●●Banger
A single CLI to manage llama.cpp/vLLM/Ollama models
Finally one CLI for Ollama, llama.cpp, and vLLM instead of three separate tools.
Solve My ProblemSlick
everlier
214mo ago