TokenDagger
AI
tokenizer library
SHOW HN2025
How HN took it
6 positive, 16 neutral or asking, 0 mixed, 5 critical, out of 27 comments about the project. Thread score +0.10.
Performance: 4 for, 1 againstExecution: 1 for, 3 againstIdea: 1 for, 1 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Drop-in replacement with substantial speedup is appealing.
From 2 comments
There’s something beautiful about creating a drop in replacement for something that improves performance substantially. ScyllaDB comes to mind
chrismustcode on HNVery cool. We use Tiktoken and I'd love to see the performance impact. Pretty great decision to make it drop-in compatible.
isjustintime on HN
Objections
Tokenization is rarely the bottleneck in LLM inference or training.
From 3 comments
Tokenizing text is ridiculously small part of the overall computation that goes into serving a request. With that said if you’re doing this on petabytes of data, never hurts to…
serjester on HNTokenization is typically done on CPU and is rarely (if ever) a bottleneck for training or inference. GPU kernels typically dominate in terms of wall clock time, the only…
refibrillator on HNTo echo the other replies, the tokenizer is definitely not the bottleneck. It just happens to be the first step in inference, so it's what I did first.
matthewolfe on HN
Dropping special-token regex may lose correctness checks for unspecified special tokens.
From 1 comment
Isn't this incorrect? If the user doesn't specify what to do with almost all of the special tokens, you still must detect them so you can raise an error.
anonymoushn on HN
Requests
Benchmark against Hugging Face tokenizers.
From 1 comment
Can you also compare the performance with https://github.com/huggingface/tokenizers/? Would be helpful, since the benchmark in the tiktoken readme seems to be very outdated.
Tiberium on HN
Push speedups upstream into tiktoken.
From 1 comment
Just curious whether it's possible to push any of your performance improvements to tiktoken itself?
pamelafox on HN
Questions
Does skipping special-token regex change output?
From 2 comments
> simplifying the algorithm to forego regex matching special tokens at all Does that mean there could be cases with less quality in terms of tokenization?
konsalexee on HNis is possible for your tokenizer to give different tokenization ever then openai tokenizer? i am asking because there are multiple ways to tokenize the same string?? sry if i am…
matrix2596 on HN
Alternatives named
Tiktoken
From 1 comment
It looks like TikToken is written in Rust (https://github.com/openai/tiktoken/tree/main/src), are the gains here actually from porting to C++?
notatallshaw on HN
Hugging Face tokenizers
From 1 comment
Can you also compare the performance with https://github.com/huggingface/tokenizers/? Would be helpful, since the benchmark in the tiktoken readme seems to be very outdated.
Tiberium on HN
bpe crate
From 1 comment
How does this compare to the BPE crate [1]? Its main selling point is support for incrementally re-tokenising text, but it's also faster than tiktoken. [1]…
p0 on HN
tokie
From 1 comment
Nice work! I tried something similar a while back ago: https://github.com/kevmo314/tokie The takeaway I also found was that the running cost was really dominated by…
kevmo314 on HN