Imagededup
Dev tools
image deduplication library
SHOW HN2019
How HN took it
8 positive, 27 neutral or asking, 1 mixed, 1 critical, out of 37 comments about the project. Thread score +0.24.
Execution: 1 for, 1 againstIdea: 1 for, 0 againstPerformance: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Useful tool for removing duplicate photos and memes.
From 2 comments
This is really nice. I have a semi-abandoned project that uses PHash to canvas my photo library to flag duplicates, and am quite likely to use this instead. Now if only Apple…
rcarmo on HNSweet. Just what I needed to remove duplicate memes from my phone.
dandigangi on HN
Nice, well-scoped project.
From 2 comments
This is really nice. I have a semi-abandoned project that uses PHash to canvas my photo library to flag duplicates, and am quite likely to use this instead. Now if only Apple…
rcarmo on HNNice project! I wonder how much of it could be adapted to finding duplicate documents, e.g. homeworks, CVs, etc. Presumably, the hashing would have to be adapted slightly. But…
cosmic_quanta on HN
Objections
Pinned dependency versions are too strict for a library.
From 1 comment
If this is a library then locking those dependencies down is not great. Does it really need Pillow 6.0.0, and not Pillow 6.0.1? As this is being consumed by larger applications…
orf on HN
Heavy dependencies such as TensorFlow.
From 1 comment
This looks nice, and seems to support the most common methods for fingerprinting/hashing. This comes with some heavy dependencies, though (which is reasonable): ```…
albertzeyer on HN
Does not scale beyond a couple thousand images locally.
From 1 comment
Scale is unfortunately not the focus of the current implementation. We would address this aspect in the future releases however. Considering the speed and memory requirements,…
tanujjain on HN
Requests
Support for animated GIFs.
From 1 comment
Any idea if there is something that supports animated gifs?
innagadadavida on HN
Use approximate nearest neighbor search such as FAISS.
From 2 comments
Use approximate nearest neighbor. The FAISS library is good.
dgacmu on HNWhat way is the hash represented? Have you looked at NN libraries like faiss [0] and NGT [1]? Those can quite easily handle a nearest neighbor search of 10 million vectors and,…
gravypod on HN
Ordering images by similarity.
From 1 comment
Could this be used to order a large set of images by similarity?
JonathanFly on HN
Questions
How does performance scale to millions of images?
From 2 comments
This looks really interesting. Can you give us an idea of the performance? E.g. roughly how long would it take to process 1 million 1920×1080 JPEGS, without GPU and with GPU?…
mkl on HN"The tricky part is the retrieval of duplicates, which on the same 10K dataset takes a few minutes" "Doesn't sound too bad." I doesn't? As a word of caution: If a 10k dataset…
jayalpha on HN
Whether it works for text documents.
From 2 comments
Thanks! This is a very important problemspace for us, as we do document processing and double/triple uploads are hard to detect and create unnecessary effort. Can you elaborate a…
Roritharr on HNFor documents I wouldnt do anything listed here. I would just compare the contents on a line by line or word by word basis
herohamp on HN
Alternatives named
imagehash library
From 1 comment
did you evaluate the 'imagehash' [1] library prior to working on this-- any limitations/concerns? the additional CNN seems to be the difference between the two libraries [1]…
Dowwie on HN
PhotoDNA hash
From 1 comment
Did you consider using the PhotoDNA hash algo for finding duplicates? If you’ve heard of it and ruled it out, love to know why. While designed for a very different (and dark)…
jsjohnst on HN
XnViewMP
From 1 comment
If anyone is interested in gui, XnViewMP does that too. https://www.journeybytes.com/2018/03/how-to-find-duplicate-photos-for-free-using-xnview.html
wruza on HN