AI/ML●●●Banger
2-bit Qwen3.6-35B-A3B with ~100% FP8 quality retention
Runs a 35B MoE model on 24GB VRAM with 2-bit quantization and minimal quality loss.
WizardryBig BrainSolve My Problem
phe2019
201d ago
Back to browse
We quantized Qwen3.6-35B-A3B to 2-bit: 12.3 GB, 225 tok/s on a 4090
by yzh·Jul 30, 2026·2 points·2 comments
AI Analysis
●●●BangerWizardryDark HorseZero to One
2-bit quantization squeezes 35B MoE model onto single 24GB GPU at 225 tok/s.
Strengths
- •Mixed 2/3-bit per projection quantization preserves quality while halving VRAM needs.
- •ZML engine offers single-binary deployment with no Python or CUDA toolkit required.
- •OpenAI-compatible API means drop-in replacement for existing LLM applications.
Weaknesses
- •Concurrency trade-offs on 16GB cards may limit multi-user or batch scenarios.
- •Linux-only support excludes Windows and macOS users from this optimization.
Category
Target Audience
ML engineers and hobbyists running large models on consumer hardware
Similar To
llama.cpp · vLLM · Ollama
Similar Projects
AI/ML●Mid
Open Access Qwen3.6-35B-A3B-UD-Q5_K_M with TurboQuant
Temporary public endpoint for Qwen3.6-35B quant on a spot instance.
Ship It
freakynit
423mo ago
AI/ML●●●Banger
Samosa Chat - Run Qwen3.6-35B-A3B Locally on a 16 GB Mac
Fits a 35B MoE model into 16GB RAM by running entirely on CPU without GPU acceleration.
WizardryBig Brain
dwa3592
6916d ago
AI/ML●●Solid
Qwen3.6-35B-A3B on a 16 GB M1 Pro with SSD-streamed MoE
SSD-streamed MoE lets 16GB M1s run 35B models, but it's a specialized fork of antirez's ds4.
Big BrainNiche Gem
andreaborio
24413d ago
AI/ML●Mid
Qwen Lens Studio – multimodal app on Qwen3.6-35B-A3B, runs on Ollama
Yet another multimodal wrapper when Cursor and Continue already dominate this space.
Ship It
vijgaurav
303mo ago
AI/ML●●Solid
I ran Qwen3.5 35B on my iPhone at 5.6 tok/SEC
Runs 19.5GB Qwen3.5 on 12GB RAM iPhone via memory swapping.
WizardryBold Bet
alexintosh
424mo ago