HN Showcase
Back to browse
Actual Claude Tokenizer

Actual Claude Tokenizer

by robkop·Apr 20, 2026·3 points·4 comments
Visit ProjectView on HN

AI Analysis

●●SolidNiche GemBig Brain

Shows actual token boundaries visually, not just a count like other tools.

Strengths
  • •Faithful token boundary visualization instead of just hitting count_tokens endpoint
  • •API key stored locally in browser, never sent to the tokenizer server
  • •Open source with GitHub repo for transparency and contributions
Weaknesses
  • •n^2 cost makes long sequences expensive to tokenize faithfully
  • •Several tokenizer viewers already exist in the prompt engineering space
Target Audience

Prompt engineers, AI researchers, Claude API developers

Similar To

tiktoken · BPE Visualizer · Prompt Tokenizers

Post Description

I've seen a few "Claude tokenizers" floating around lately with all the 4.7 chatter, but most of them just hit the count_tokens endpoint and hand you back a number. You don't actually see how your text gets split or understand the changes from 4.6 to 4.7.

I built this a while back for doing some mech interp research. It faithfully represents Claude token splitting - showing hidden tokens, real boundaries and so on. It is not cheap to run - essentially n^2 cost - you could optimise for longer sequences but you are not guaranteed a faithful representation if so.

Open Source: https://github.com/R0bk/claude-tokenizer

Feedback welcome, let me know if there are any edge cases that look wrong.

P.S. I'd expect this to face a similar fate as streaming chunk and prefill based token extraction methods did. I do worry about the ability to do independent research once it's fully closed off and would love it if there was more public frontier tokenizers.

Similar Projects

Developer Tools●Mid

Claude Rank – See your Claude usage and compete with others

The UI pairs a live 'tokens shipped' counter with per-user leaderboards and cache-efficiency stats — exactly the kind of telemetry a team would want to monitor cost and behavior. Code hints (redis.zrevrank, OTEL_RESOURCE_ATTRIBUTES, db.execute and a mix of Rust + JS) show it's built from real infra primitives rather than a mock. It's a tidy, pragmatic tool for Claude users, but the idea is familiar and it needs clearer privacy/consent handling before I'd recommend it broadly.

Niche GemSlick
AkshayS96
124mo ago
Developer Tools●●Solid

Npx Claude-traces, visualizer for Claude Code/Agent SDK traces

Runs with one npx command and immediately surfaces a helpful timeline view with token counts, tool I/O panes and subagent nesting — exactly the sort of visibility you want when an agent goes off the rails. Cleverly reads the local ~/.claude/projects traces so setup is trivial, but its usefulness is limited by being Claude-only and local; add search/aggregation or a team-sharing mode and this jumps up a tier.

Niche GemSolve My ProblemSlick
hahawhatsgood
204mo ago