Faster UTF-8 validator
Languages and libraries
utf-8 validation library
SHOW HN2019
How HN took it
3 positive, 8 neutral or asking, 0 mixed, 1 critical, out of 12 comments about the project. Thread score +0.17.
Performance: 2 for, 0 againstExecution: 0 for, 1 againstDesign: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Clean, well-documented C code
From 2 comments
I'm not sure if the compiler is smart enough to do it for you, but you might be able to squeak a bit more performance by precomputing a partial sum (data + V_LEN - 1) outside the…
eyegor on HNI like the way you write and comment your C code. Very nicely written. Except all those spaces between the # and “define”, no sir I don’t like that :)
SlowRobotAhead on HN
Performance gains over prior validators
From 2 comments
Pretty nice. Always nice to see better optimized commonly required algorithms. If this is really faster than previous implementations (no reason to doubt, but don't have time to…
vardump on HNDo the instructions you use cause downclocking on Intel CPUs?
CodesInChaos on HN
Objections
Benchmark input distribution was not fully specified
From 1 comment
Good stuff. The key to a solid claim, especially for something as bold as "fastest in the world", is a complete specification of the inputs to the benchmark. You mention random…
BeeOnRope on HN
Validator fails some invalid UTF-8 test cases from Markus Kuhn's suite
From 1 comment
Hi! I took a shot at vectorized UTF-8 validation in 2012. I just put this faster validator, and Daniel Lemire's code, into my test harness at…
andrewf on HN
Requests
Add a Makefile
From 1 comment
Oh wow, that is most definitely a bug. Thank you very much for reporting that, before this spreads too much. How embarrassing... I was in a bit of a rush to stick this up on HN…
zwegner on HN
Benchmark with realistic language corpora
From 2 comments
Good stuff. The key to a solid claim, especially for something as bold as "fastest in the world", is a complete specification of the inputs to the benchmark. You mention random…
BeeOnRope on HNDefinitely a good idea. I had already downloaded a Mandarin Wikipedia page to look for test sequences, but hadn't benchmarked them. Getting some more pages there is a great place…
zwegner on HN
Questions
Does the approach differ from the simdjson validator PR?
From 1 comment
how does this differ from here: https://github.com/lemire/simdjson/pull/365
aqrit on HN
Do these AVX-512 or AVX2 instructions cause downclocking on Intel CPUs?
From 1 comment
Do the instructions you use cause downclocking on Intel CPUs?
CodesInChaos on HN
Alternatives named
Lemire's fastvalidate-utf-8
From 2 comments
Well, to be fair, my claim was that it was the fastest in the world that I'm aware of, which is a much weaker statement. :) The random UTF-8 in the benchmark was generated from…
zwegner on HNHi! I took a shot at vectorized UTF-8 validation in 2012. I just put this faster validator, and Daniel Lemire's code, into my test harness at…
andrewf on HN
simdjson UTF-8 validation PR
From 1 comment
how does this differ from here: https://github.com/lemire/simdjson/pull/365
aqrit on HN