HuggingFace
Languages and libraries
nlp tokenization library
SHOW HN2020
168points
42comments
#354of 3,028 in 2020
Top 12%of 2020 by points
How HN took it
CuriousMostly questions and neutral comments
3 positive, 10 neutral or asking, 0 mixed, 0 critical, out of 13 comments about the project. Thread score +0.21.
Performance: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Very fast tokenization, a 6-7x speedup in one quick test
From 1 comment
Great! Just did a quick test and got a 6-7x speedup on tokenization.
hnaccy on HN
Objections
Repos lack tests
From 1 comment
Question for HuggingFace folks. Your repos do not contain any tests. Why is that? How do you ensure your stuff actually works after you make a change?
m0zg on HN
Requests
Comparison with spaCy's tokenizer
From 1 comment
We use both SpaCy and HuggingFace at work. Is there a comparison of this vs SpaCy's tokenizer[1]? 1. https://spacy.io/usage/linguistic-features#tokenization
ZeroCool2u on HN
Questions
What does tokenization do?
From 1 comment
What does tokenization (of strings, I guess) do?
rsp1984 on HN
Why are repos without tests?
From 1 comment
Question for HuggingFace folks. Your repos do not contain any tests. Why is that? How do you ensure your stuff actually works after you make a change?
m0zg on HN
Alternatives named
spaCy tokenizer
From 1 comment
We use both SpaCy and HuggingFace at work. Is there a comparison of this vs SpaCy's tokenizer[1]? 1. https://spacy.io/usage/linguistic-features#tokenization
ZeroCool2u on HN