AI/ML●●Solid
TurboQuant for vector search – 2-4 bit compression
Data-oblivious quantization beats Product Quantization on online updates.
Big BrainNiche Gem
justsomeguy1996
8962mo ago
Back to browse
GitHub Repository
Standalone TurboQuant KV Cache Inference for https://huggingface.co/g023/Qwen3-1.77B-g023
4 starsPython
Standalone TurboQuant KV Cache Inference
by g023·Apr 3, 2026·3 points·4 comments
AI Analysis
●MidBig BrainShip It
Standalone KV cache compression script implementing TurboQuant with 1.55x ratio.
Strengths
- •Self-contained implementation of complex quantization math like Lloyd-Max and QJL.
- •Minimal dependencies make it easy to audit the actual inference logic.
- •Demonstrates specific memory savings on a custom Qwen3 model variant.
Weaknesses
- •Manual file management ("throw in folder") creates friction versus pip installable tools.
- •1.55x compression ratio trails industry standards like INT4 or FP4 quantization.
Category
Target Audience
ML researchers, LLM inference engineers
Similar To
vLLM · TensorRT-LLM · SGLang
Post Description
Implements TurboQuant (ICLR 2026, arXiv:2504.19874) KV cache compression
directly inside a Transformers inference script. All algorithms are self-contained. Minimal dependencies.
- uses https://huggingface.co/g023/Qwen3-1.77B-g023 as the demonstration model (throw model files in Qwen3-BEST folder)
Similar Projects
AI/ML●●●Banger
TurboQuant-WASM – Google's vector quantization in the browser
Google's ICLR 2026 quantization paper running client-side with SIMD-accelerated dot products.
WizardryZero to One
teamchong
16572mo ago
Developer Tools●●Solid
simple_ans – Asymmetric Numeral Systems Compression in Python/C++
Single-file C++ ANS kernel beats wrestling with zstandard for quantized data.
Niche GemCozy
jmagland
311mo ago
Open Source●Mid
Algorithms 1.0.0 – Minimal and clean implementations of algorithms
Files are single-purpose and readable: each algorithm comes with docstrings, type hints, complexity notes and runnable examples so you can read, test, or pip-install bits immediately. It isn't breaking new ground — algorithm collections are common — but the focus on clarity, tests, and a tiny surface API (merge_sort, BinaryHeap, dijkstra, etc.) makes this a reliable reference and teaching aid.
Niche GemCrowd Pleaser
kwk236
703mo ago
AI/ML●●Solid
Mamba SSM in Rust – training and inference with custom CUDA kernels
Custom CUDA kernels for SSM recurrence with zero framework dependencies.
WizardryNiche Gem
silvermpx
102mo ago
AI/ML●●Solid
NeuroFlow 55.8x video inference speedup for Vision Transformers PyTorch
Training-free dual-memory protocol cuts 1792p SigLIP inference from 678ms to 11.9ms.
Big BrainWizardry
ynnk
8220d ago