TurboQuant for mlx-lm (Apple Silicon)

Name: TurboQuant for mlx-lm (Apple Silicon)
Availability: InStock
Author: pythongiant

by pythongiant·Jul 7, 2026·1 point·1 comment

Visit Project View on HN

AI Analysis

●●●BangerWizardrySolve My Problem

Custom Metal kernels bring Google's TurboQuant KV-cache compression to Apple Silicon.

Strengths

•Custom Metal kernel for non-uniform quantization avoids slow generic Python fallbacks.
•Modular implementation allows studying individual components like PolarQuant and QJL separately.
•Drop-in KV cache replacement preserves attention scores via random Hadamard rotation.

Weaknesses

•Apple Silicon only, excluding Nvidia CUDA users from accessing these benefits.
•Benchmarks rely on specific models, may not generalize to all architectures.

Post Description

Hi HN,

I built mlx-turboquant, an implementation of Google's TurboQuant KV-cache compression algorithm for Apple's MLX framework.

The repository includes quality benchmarks, memory benchmarks, and a modular implementation so individual pieces (PolarQuant, QJL, packing, codebooks) can be studied independently.