OmniAI OCR benchmark
AI
ocr benchmark
SHOW HN2025
How HN took it
6 positive, 13 neutral or asking, 1 mixed, 6 critical, out of 26 comments about the project. Thread score +0.05.
Performance: 3 for, 1 againstPricing: 0 for, 2 againstIdea: 1 for, 0 againstExecution: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Qwen 2.5 VL performs very well on OCR and vision tasks
From 3 comments
I've been very impressed with Qwen in my testing, I think people are underestimating it
CSMastermind on HNI've been consistently surprised by Gemini's OCR capabilities. And yeah, Qwen is climbing the vision ladder _fast_. In my workflows I often have multiple models competing…
fpgaminer on HNI wrote a small, client-side-JS-only app that does OCR and TTS on board game cards, so my friends and I can listen to someone read the cards' flavor text. On a few pages of text…
stavros on HN
Benchmark is fully open source with code and dataset
From 1 comment
Objections
Benchmark is run by an OCR vendor that ranks its own product first
From 2 comments
News update: OCR company touts new benchmark that shows its own products are the most performant.
azinman2 on HNThe model with the best accuracy in the linked benchmark is "OmniAI" (OP's company) which looks like a paid model, not open source [1]. [1]: https://getomni.ai/pricing
rustc on HN
Open models produce errors in precision for real use
From 1 comment
Its errors are interesting (averaging around one per paragraph). Semantically-correct, but wrong on precision (simple example, the English word "ardour" is transcripted as…
pmarreck on HN
Full benchmark costs are high and provider pricing varies widely
From 2 comments
Reproducing the whole benchmark would be expensive, OmniAi starts at $250/month.
rustc on HNYes, I'll add that to the writeup! You're right, initially excluded it because it was really dependent on the providers, so lots of variance. Especially with the Qwen models.…
themanmaran on HN
Requests
Add bounding box support to the benchmark
From 2 comments
I suppose none of these models can output bounding box coordinates for extracted text? That seems to be a big advantage of traditional OCR over LLMs. For applications I'm…
ks2048 on HNWe're also looking to test qwen and other for the bounding box support. Simon Willison had a great demo page where he used Gemini 2.5 to draw bounding boxes, and the results were…
themanmaran on HN
Add cost and latency results
From 1 comment
You mention that you measured cost and latency in addition to accuracy - would you be willing to share those results as well? (I understand that for these open models they would…
daemonologist on HN
Include Surya and MiniCPM models
From 2 comments
Is there a reason Surya isn’t included?
ianhawes on HNWhat about mini cpm v2.6?
sandreas on HN
Questions
How does Tesseract compare to LLM-based OCR
From 1 comment
Tesseract can manage 99% accuracy on anything other than handwriting. Without being an LLM. Is there an advantage of using an LLM here?
AndrewDucker on HN
Why is Llama 90b priced so high
From 1 comment
One of these things is not like the others. $8.50/1000?? Any chance that's a typo? Otherwise, for someone that has no experience with LLM pricing models, why is Llama 90b so…
dylan604 on HN
Alternatives named
Mistral OCR
From 1 comment
I'm very curious about the results - I've been using mistral-ocr for the last 2 weeks and it worked really well.
hdjjhhvvhga on HN
Tesseract
From 1 comment
Tesseract can manage 99% accuracy on anything other than handwriting. Without being an LLM. Is there an advantage of using an LLM here?
AndrewDucker on HN
Gemini 2.5 Pro
From 1 comment
In addition, gemini Pro 2.5 does really well with bounding boxes, but yeah not open source :(
kapitalx on HN
Doctly
From 1 comment
If you're limited to open source models, that's very true. But for larger models and depending on your document needs, we're definitely seeing very high accuracy (95%-99%) for…
kapitalx on HN