AI/ML●●●Banger
Wordchipper – Rust BPE tokenizer, 9x faster than tiktoken
Nine times faster than tiktoken-rs with swappable lexer backends for benchmarking.
WizardryBig Brain
antimora
203mo ago
Back to browse
GitHub Repository
Fast exact BPE tokenizer. Byte-identical to tiktoken, 7x faster
6 starsC++
Quicktok, an exact BPE tokenizer 7x faster than tiktoken
by dmatth1·Jul 3, 2026·2 points·1 comment
Similar Projects
AI/ML●●Solid
GreedyPhrase – 1.21x better compression than GPT-4o tiktoken, 6x faster
Phrase-mining beats tiktoken compression 1.21x with 1/3 vocab size, but niche for token optimization.
Big BrainWizardry
bazlightyear
104mo ago
AI/ML●●Solid
UBPE – a universal BPE tokenizer, optimized and rethought
Novel BPE variant using tf-idf scoring produces shorter encodings than classic.
Big BrainNiche Gem
Scurrra
103mo ago
AI/ML●Mid
Id-agent – Token efficient UUID alternative for AI agents
Token-efficient word IDs for LLMs, but it's a narrow utility library.
Big BrainShip It
pranshuchittora
42551mo ago
Developer Tools●●Solid
PyNear – exact and approximate KNN, faster than Faiss
VP-Tree + SIMD beats Faiss 39× on exact L2, 257× on binary search.
Big BrainNiche Gem
pcael
203mo ago
Developer Tools●●Solid
Actual Claude Tokenizer
Shows actual token boundaries visually, not just a count like other tools.
Niche GemBig Brain
robkop
342mo ago