Jam API
Dev tools
web scraping api
SHOW HN2016
300points
82comments
#65of 2,264 in 2016
Top 3%of 2016 by points
How HN took it
DebatedReal support and real criticism
12 positive, 17 neutral or asking, 2 mixed, 12 critical, out of 43 comments about the project. Thread score +0.04.
Idea: 6 for, 3 againstExecution: 0 for, 6 againstDesign: 2 for, 0 againstPricing: 0 for, 1 againstPerformance: 0 for, 1 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Good idea and useful for scraping without custom code
From 3 comments
This is a fantastic idea and I'm really surprised nothing like this has existed before, it seems like such a no-brainer. Great work.
adriancooney on HNI've been thinking about writing some website-to-JSON scrapers myself and this basically solves that problem (since I would have been going after CSS selectors or xpath anyway…
paulmd on HNVery nice idea. Although scraping should always be a last resort, I could imagine using this for semi-serious purposes, i.e. when I care enough about the output, will be doing…
chriswarbo on HN
Developer-friendly interface and low barrier to start
From 2 comments
We launched WrapAPI (https://wrapapi.com/) a few weeks ago with the same functionality, but a bit more complex and powerful process to get set up. You can not only specify CSS…
phsource on HNYes, yes, yes! I'm using Apifier at the moment, which I really like, but my biggest gripe is the awkwardness of source (and VCS) integration. The best I've come up with is to…
OJFord on HN
Objections
Scrapers are fragile and break when page markup changes
From 2 comments
I built something almost identical in 2011. It really doesn't have as much utility in practice as you think initially. CSS selectors are an interesting idea for extracting data…
geuis on HN> I'm really surprised nothing like this has existed before But how would you monetize it? Unlike an RSS feed, you really don't know how the JSON response would be used, so you…
mappy on HN
Hard to monetize
From 2 comments
> I'm really surprised nothing like this has existed before But how would you monetize it? Unlike an RSS feed, you really don't know how the JSON response would be used, so you…
mappy on HNI think the wanting to know how it will be monetized /support itself is a reasonable question. If there isnt an answer you know not to build something using it, expecting it to…
bryanrasmussen on HN
Sites block scraping via robots.txt and rate limits
From 2 comments
https://github.com/fizx/parsley/wiki looks pretty similar. Running this sort of thing as a service/api never panned out for us because you are almost universally robots.txt…
fizx on HNdone it as well. at time specialized to organize web comics (it was way before google reader times). real issue is that popularity will get you blocked fairly quickly. see also:…
LoSboccacc on HN
Does not handle JavaScript-rendered pages
From 2 comments
Wonderful idea. What about DOM nodes generated by JavaScript? Will Jam render the page before scraping?
daw___ on HNI built something almost identical in 2011. It really doesn't have as much utility in practice as you think initially. CSS selectors are an interesting idea for extracting data…
geuis on HN
Requests
Support for HTTPS requests
From 2 comments
If it's going to be used for serious purposes it really needs HTTPS support, as most APIs do these days.
nsgi on HNHTTPS results in 500 Internal Server Error. Edit: Well no, it's only some sites. E. g. https://medium.com
WA on HN
Support for JavaScript-rendered DOM
From 1 comment
Wonderful idea. What about DOM nodes generated by JavaScript? Will Jam render the page before scraping?
daw___ on HN
Form authentication with cookies
From 1 comment
I'm actually wondering if it would be possible to add forms authentication to this? Eg. Post with some sort of css selecters and then a "cookie memory".
NicoJuicy on HN
Publish to npm
From 1 comment
Nice work, thanks for adding the Github link. I can think of lots of immediate use for this. Consider publishing on NPM?
splatcollision on HN
Questions
How will it make money?
From 2 comments
> I'm really surprised nothing like this has existed before But how would you monetize it? Unlike an RSS feed, you really don't know how the JSON response would be used, so you…
mappy on HNA little word of warning/encouragement. I did something similar a long time ago (JSONDuit), which got posted to HN by someone else. You will probably run into a healthy mix of…
lost_my_pwd on HN
Will it render JavaScript before scraping?
From 1 comment
Wonderful idea. What about DOM nodes generated by JavaScript? Will Jam render the page before scraping?
daw___ on HN
Alternatives named
Kimono Labs
From 2 comments
There was a YC company a few years back that got acquired by Palantir in February that did something very similar. https://www.kimonolabs.com/
moeamaya on HNIf anyone remembers, they was a YC company that did exactly this. It was called Kimono Labs. I think it failed and just got acquired a year ago. "Jam API" will probably do way…
karlcoelho1 on HN
Apifier
From 2 comments
Well, https://www.apifier.com does essentially the same thing, plus it supports JavaScript, can crawl through the whole website etc. Disclaimer: I'm a cofounder there
jancurn on HNYes, yes, yes! I'm using Apifier at the moment, which I really like, but my biggest gripe is the awkwardness of source (and VCS) integration. The best I've come up with is to…
OJFord on HN
WrapAPI
From 1 comment
We launched WrapAPI (https://wrapapi.com/) a few weeks ago with the same functionality, but a bit more complex and powerful process to get set up. You can not only specify CSS…
phsource on HN
Parsley
From 1 comment
https://github.com/fizx/parsley/wiki looks pretty similar. Running this sort of thing as a service/api never panned out for us because you are almost universally robots.txt…
fizx on HN
xidel
From 1 comment
Using a 3rd party site to query HTML (which you should be able to do yourself, plenty of tools for that) isn't a fantastic idea. This one, for example…
xaduha on HN