Espion
Web and SaaS
web scraping service
SHOW HN2014
109points
54comments
#255of 2,272 in 2014
Top 12%of 2014 by points
How HN took it
CuriousMostly questions and neutral comments
4 positive, 16 neutral or asking, 1 mixed, 11 critical, out of 32 comments about the project. Thread score -0.28.
Idea: 2 for, 6 againstPricing: 0 for, 1 againstExecution: 0 for, 1 againstDesign: 1 for, 0 againstPerformance: 0 for, 1 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Good site design
From 1 comment
Does the location of Espion in Mauritius have anything to do with this? Are the servers located there?. Good design of the site and good work.
Jayd2014 on HN
Useful for businesses lacking time to build scraping setups
From 2 comments
Or for scraping anything non-trivial, do it yourself with Casperjs, AWS/Linode/DO, RabbitMQ and a bit of monitoring/alerting from someone like Datadog. It will be cheaper and a…
corford on HNThe general sentiment on this thread pretty much sums up the idea of "scraping as a service". To me, there is definitely a legitimate business need to be able scrape. Whether…
softdev12 on HN
Objections
Likely PhantomJS wrapped in a cloud service
From 1 comment
it's just PhantomJS wrapped up behind a cloud
asdf123456 on HN
IP pools get blacklisted at scale
From 1 comment
We run in-house data extraction structure very similar to this (spiders, phantomJS, ocr, anonymous proxies, etc), and it indeed takes some time to set it up properly. Main…
ivanhoe on HN
Legal and abuse risk from scraping and spam
From 3 comments
> CAPTCHAs and IP blacklistings are things you encounter routinely in perfectly legitimate and legal web scraping projects It's only legal if the site TOS says it is.You dont get…
aikah on HNWhat if instead of scraping, customers use the JS injection features for spamming? How are you approaching the liability issues involved, given that potentially anyone can sign…
logn on HNIt sounds a bit in a grey area, in-between legit use and something slightly dodgy (mentioning IP blacklisting and overriding CAPTCHAS...) I'm curious to know what kind of…
gingerlime on HN
Slower than non-JS scraping
From 1 comment
Can you talk about performance? Having written both vanilla non-js scrapers (which get you far enough) and js scrapers for those pesky web pages, I can confirm that our js…
ankimal on HN
Requests
GUI for selecting elements and generating scraper code
From 1 comment
This is an interested idea, but what I would pay for is a GUI for scrapping, for example: clicking an element, seeing what classes it got, and be able to click each class to see…
ivanca on HN
Video recording of scraped pages
From 1 comment
Just wondering, can the scraper record video of the site it's on? I feel like that's something I have a lot of use cases for. Even if it's not possible is there some hack you…
AlwaysBCoding on HN
Clearer non-technical front page copy
From 1 comment
Since this looks to be a product targeting businesses, I would say you might want to dumb down your techie speak on the front page. Talk to more of the benefits and value that…
dalacv on HN
Questions
How it differs from PhantomJS or Zombie.js
From 2 comments
How is it different than Phantom.js or Zombie.js ?
LunaSea on HNSo, is the Espion headless browser based on an existing one? If this is the case, which one?
gildas on HN
Who the target customer is
From 1 comment
Who is your target customer?
xyby on HN
Legality of scraping and terms of service
From 2 comments
> CAPTCHAs and IP blacklistings are things you encounter routinely in perfectly legitimate and legal web scraping projects It's only legal if the site TOS says it is.You dont get…
aikah on HNIt's more complex than that. Whether the site's TOS are enforceable is a matter of jurisdiction and may depend on the intent of the web scraper. Ryanair for example has lost…
TheEspion on HN
Alternatives named
PhantomJS and CasperJS
From 2 comments
Or for scraping anything non-trivial, do it yourself with Casperjs, AWS/Linode/DO, RabbitMQ and a bit of monitoring/alerting from someone like Datadog. It will be cheaper and a…
corford on HNSounds more like phantomjs/casperjs.
dale-cooper on HN
Selenium
From 2 comments
you can do this with selenium also btws
dmritard96 on HNSelenium could do that for you, and there's an IDE available as a Firefox plugin; branching might be tricky and you'd need to build some tooling around it in general, but it…
aaronem on HN
Kimono Labs
From 1 comment
If you are looking for a GUI for scraping, check out kimono labs. It's a GUI tool for scraping, and requires no code to set up and can find all similar elements from the one you…
juddernaught on HN
Cheerio
From 1 comment
This is cheerio with node.js and extra bells. Example: http://projs.hackhat.com/the-making-of-the-hackernews-crawler-interesting-tips-to-know/
bitplanets on HN