hncrawl
Dev tools
web scraper
SHOW HN2012
How HN took it
2 positive, 4 neutral or asking, 0 mixed, 6 critical, out of 12 comments about the project. Thread score -0.20.
Idea: 2 for, 2 againstExecution: 0 for, 3 againstPerformance: 0 for, 1 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Useful for people who want to scrape HN
From 1 comment
I think many people will find this useful. I certainly do!
kuromitsu on HN
Scrapy is a good framework
From 1 comment
Scrapy, what a fantastic framework it is! One thing bothers me with your code tough, the following could be replace with built in Scrapy-tools. Edit: The code got wrongly…
awsedwsq on HN
Objections
Crawling HN directly adds load; should use the API or a cache
From 2 comments
Please use this: http://api.ihackernews.com/ HN is slow enough as it is ;)
jacquesm on HNI'm sure your intentions are good. Did you take up the errors issue with the makers of the api? HN is built using a homebrew stack, it's a miracle it performs as well as it does.…
jacquesm on HN
Ignores HN's Crawl-delay in robots.txt
From 1 comment
Don't forget the Crawl-delay on HN's robots.txt: http://news.ycombinator.com/robots.txt
DanielRibeiro on HN
Code could use built-in Scrapy tools and is unnecessarily complex
From 2 comments
Scrapy, what a fantastic framework it is! One thing bothers me with your code tough, the following could be replace with built in Scrapy-tools. Edit: The code got wrongly…
awsedwsq on HNWhy noy just use requests and lxml, you could do the whole thing in about 10 lines in one file
the_cat_kittles on HN
Requests
Use the RSS feed instead of scraping
From 1 comment
I haven't run the crawler so I'm not sure what else it does, but if it only parses the home page and fetches the external links, why not read http://news.ycombinator.com/rss (you…
jimmy2times on HN
Remove hardcoded local path
From 1 comment
yeah, I modified a bit his script. I chose his script because it logs you in so you can grab your content. btw you have a hardcoded path in there /Users/mvanveen/root/dev/news/out/
zalew on HN
Questions
Legal considerations before crawling a website
From 2 comments
On the legal front, what all should I consider before crawling a website, apart from honoring robots.txt?
zerop on HNThanks...On the legal front, what all should I consider before crawling a website, apart from honoring robots.txt?
zerop on HN
Why BeautifulSoup was used instead of Scrapy's selector
From 1 comment
I'm curious why you used BeautifulSoup when scrapy has its own built-in HTML parser (HtmlXPathSelector). Is there an advantage to BeautifulSoup or is it just the tool that you're…
wrath on HN
Alternatives named
ihackernews API
From 1 comment
Please use this: http://api.ihackernews.com/ HN is slow enough as it is ;)
jacquesm on HN
hacker-news-confidence project
From 1 comment
Here's mine, written on a lazy Sunday evening in January: https://github.com/shadowsun7/hacker-news-confidence It runs on cron, scrapes HN once every 30 minutes, and sorts it…
shadowsun7 on HN
Zalew's Python HN grabber script
From 1 comment
relevant http://zalew.net/2011/12/08/grab-your-hackernews-stories-and-comments-python/ ;)
zalew on HN