Wordchipper – Rust BPE tokenizer, 9x faster than tiktoken
Nine times faster than tiktoken-rs with swappable lexer backends for benchmarking.

They stopped pretending chunking at arbitrary byte offsets was fine and instead scan once to build message boundaries, then binary-search for clean split points — that simple change eliminates the OOM-by-design scenario. Couple that with SIMD-aware prefetch tuning (different distances for AVX2 vs AVX-512) and you get practical microarch-aware engineering, not just benchmark stunts; I want this shipped as a library or tool so other firms can stop reinventing the same footguns.
Low-latency engineers, quant developers, market-data infrastructure teams
Nine times faster than tiktoken-rs with swappable lexer backends for benchmarking.
Rust-powered BeautifulSoup with 10x speed and full API compatibility.
600x speedup over Node.js version, but Cursor support is currently broken.
Beats simd-csv with pclmulqdq trick, but CSV parsing is a solved category.
Reverse-engineered undocumented MOBI format — builds dictionaries in 6 seconds vs 12 hours.
2.7ms vs 151ms startup—pure speed optimization, Python thefuck already works.