Chonky
AI
text chunking model
SHOW HN2025
How HN took it
4 positive, 8 neutral or asking, 0 mixed, 4 critical, out of 16 comments about the project. Thread score -0.03.
Docs: 0 for, 3 againstIdea: 2 for, 0 againstPerformance: 1 for, 1 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Interesting and impressive approach
From 3 comments
Really amazing and impressive work!
rekovacs on HNVery cool! The training objective is clever. The 50+ filters at Ecodash.ai for 90,000 plants came from a custom RAG model on top of 800,000 raw web pages. Because LLM’s are…
legel on HNInteresting! I worked previously for a company that did automatic generation of short video clips from long videos. I fine-tuned a t5 model by taking many Wikipedia articles and…
kamranjon on HN
Useful for noisy transcripts and OCR'd PDFs
From 2 comments
This is the library that I use, mainly around very noisy IRC chat transcripts and it works pretty well. OP I'd love to see a paragraph matching comparison benchmark against…
vunderba on HNThis would be useful when chunking PDFs scanned with OCR. I've done that before and paragraph breaks were detected pretty inconsistently.
cckolon on HN
Objections
No benchmarks showing performance against alternatives
From 2 comments
I applaud the FOSS initiative but as with anything ml: benchmarks please so we can see what test cases are covered and how well they align with a project's needs.
mentalgear on HNDid you evaluate it on a RAG benchmark?
jaggirs on HN
README lacked example output
From 3 comments
I feel you could improve your README.md considerably just by showing the actual output of the little snippet you show.
suddenlybananas on HNCame here to write exactly that. The author includes a large sentence in the sample, so it should show us the output.
HeavyStorm on HNLove that people are trying to improve chunkers, but just some examples of how it chunked some input text in the README would go a long way here!
petesergeant on HN
Requests
Benchmark against wtpsplit
From 1 comment
This is the library that I use, mainly around very noisy IRC chat transcripts and it works pretty well. OP I'd love to see a paragraph matching comparison benchmark against…
vunderba on HN
Support for other languages
From 2 comments
Does it work on other languages?
olavfosse on HNThe non english space in these fields is so far behind in terms of accuracy and reliability, it's crazy
fareesh on HN
Evaluation on a RAG benchmark
From 1 comment
Did you evaluate it on a RAG benchmark?
jaggirs on HN
Questions
Is chunking deterministic?
From 1 comment
Interesting idea - is the chunking deterministic? It would have to be to be useful, but I’m wondering how that interacts with the neural net.
rybosome on HN
Does it work on other languages?
From 1 comment
Does it work on other languages?
olavfosse on HN
Can it index fiction books in a vector DB for RAG?
From 1 comment
So I could use this to index i.e. a fiction book in a vector db, right? And the semantic chunking will possibly provide better results at query time for rag, did I understand…
sushidev on HN
Alternatives named
wtpsplit
From 1 comment
You might want to take a look at https://github.com/segment-any-text/wtpsplit It uses a similar approach but the focus is on sentence/paragraph segmentation generally and not…
mathis-l on HN