Diffbot Article API
AI
content extraction api
SHOW HN2011
151points
49comments
#57of 1,097 in 2011
Top 6%of 2011 by points
How HN took it
MixedNo clear lean
12 positive, 14 neutral or asking, 1 mixed, 4 critical, out of 31 comments about the project. Thread score +0.23.
Idea: 4 for, 0 againstExecution: 0 for, 3 againstPerformance: 2 for, 1 againstDocs: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Works better than comparable services and gives good results
From 3 comments
Cool, works better than other services like this I've found. I tried it on Ars Technica's review of the Xoom tablet and it found all 10 pages. It didn't find the embedded video…
modeless on HNIn an earlier comment, you said that much of the time was spent in constructing the features (e.g. you had to implement CSS). Did you mean implementation time, or…
npp on HNlol
rickyyean on HN
Fast processing, about 80ms per article
From 2 comments
I just timed that Huffington Post example on my computer and it took around 80ms to extract at 100% CPU. The rub though is that, if you type in a totally new URL on our demo…
miket on HNThat's a really nifty API. Performance is nice too. Would be interested in knowing more about it.
mcfchan on HN
Interesting machine learning approach
From 2 comments
I like the "machine learning" part of the api, but there seems to be no way of improving the learning by giving feedback. Just testing another article on HN[1] that the tags are…
itsnotvalid on HNI'm an interested party as well. As a student of machine learning I would love to learn the techniques you've applied. I'm doing a data mining startup and this would be very…
jamongkad on HN
Objections
Returns only raw text, losing structure like headers and images
From 1 comment
Cool, works better than other services like this I've found. I tried it on Ars Technica's review of the Xoom tablet and it found all 10 pages. It didn't find the embedded video…
modeless on HN
Tag quality is poor for some articles
From 1 comment
I like the "machine learning" part of the api, but there seems to be no way of improving the learning by giving feedback. Just testing another article on HN[1] that the tags are…
itsnotvalid on HN
Fails on some pages such as forums and paywalled sites
From 2 comments
Duh... for the pages I tried, it always gives me the "No article at this URL". I'd like to have some kind of boilerplate removal that works well for forum content (e.g. phpBB and…
sqrt17 on HNI noticed that this will happen for NYT articles behind their registration-wall.
buss on HN
Requests
Return simplified HTML structure
From 2 comments
It would be great if you could return a normalized and simplified version of the HTML structure. I know a lot of people who would be interested in this.
bravura on HNRight now, the API just returns back the raw text for simplicity's sake, but it would be possible to make an option for returning a bit of HTML structure, which would address the…
miket on HN
Feedback mechanism to improve learning
From 1 comment
I like the "machine learning" part of the api, but there seems to be no way of improving the learning by giving feedback. Just testing another article on HN[1] that the tags are…
itsnotvalid on HN
Blog post explaining the technique
From 2 comments
I'd be happy to talk about the visual-based statistical classification technology in a follow-up blog post if there's interest.
miket on HN+1 for good computer science writeups
itsnotvalid on HN
Support for Wikipedia pages
From 1 comment
Looks good but doesn't work for wikipedia. Get it working there and you will have a lot more consumers.
Mamady on HN
Questions
How resource-intensive is it to run?
From 2 comments
This is fantastic! How resource-intense is it to run? Machine Learning, depending on the implementation, can be pretty costly from what I understand.
shantanubala on HNCan you quantify that? Like, on a reasonable computer, how long does it take to process the Huffington Post example article? And (though this may be a little trickier to…
pjscott on HN
What rendering engine is used for CSS?
From 1 comment
What do you use to get the visual CSS rendering of the elements on the page? A library like libmoz, perhaps?
ajays on HN
Alternatives named
Boilerpipe
From 2 comments
Diffbot's stuff is in a different league (but it's a hosted service with a large dataset) but if anyone's vaguely interested in this area, I've been working on a Ruby library…
petercooper on HNThere is an online demo of boilerpipe: http://boilerpipe-web.appspot.com/ I use boilerpipe a lot and highly recommend it. We've discussed it before on MetaOptimize:…
bravura on HN
Pismo
From 1 comment
Diffbot's stuff is in a different league (but it's a hosted service with a large dataset) but if anyone's vaguely interested in this area, I've been working on a Ruby library…
petercooper on HN
Purifyr
From 1 comment
We are using http://purifyr.com/ for this. Pretty happy with the unicode support and 20-50 documents per second speed.
alexdong on HN
Zemanta
From 1 comment
Exactly. For instance, if a user is typing in a blog post, right now the flow would be: 1) User submits post 2) I create a page for their post 3) I call your API to analyze the…
tansey on HN