Sukhoi
Dev tools
web crawler
SHOW HN2017
How HN took it
3 positive, 6 neutral or asking, 0 mixed, 1 critical, out of 10 comments about the project. Thread score +0.16.
Idea: 1 for, 1 againstDesign: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Looks more enjoyable to use than BeautifulSoup
From 1 comment
Pretty cool project. It looks more enjoyable to use than BeautifulSoup. How does it approach throttling or rate limiting? I didn't see this mentioned in the readme examples.…
tedmiston on HN
Interesting project that focuses on combining information from multiple sources
From 1 comment
Interesting timing! I just started using Scrapy today for a project, and I'm trying to figure out how to elegantly piece together information from different sources. I'm glad to…
dguo on HN
Objections
Python 2 only, no Python 3 support yet
From 2 comments
Is this Python 3 compatible? Searched but the wiki is empty and the readme has examples in Python 2.
pryelluw on HNIt is py2 now, however, i'm gonna port it to py3 soon. I'm planning to write some better docs for it tomorrow.
iogf on HN
Not clear how it compares to Scrapy
From 1 comment
How does this differ from Scrapy?
monksy on HN
Requests
Add throttling or rate limiting
From 1 comment
Pretty cool project. It looks more enjoyable to use than BeautifulSoup. How does it approach throttling or rate limiting? I didn't see this mentioned in the readme examples.…
tedmiston on HN
Add caching and ETag support
From 1 comment
Pretty cool project. It looks more enjoyable to use than BeautifulSoup. How does it approach throttling or rate limiting? I didn't see this mentioned in the readme examples.…
tedmiston on HN
Port to Python 3
From 1 comment
Is this Python 3 compatible? Searched but the wiki is empty and the readme has examples in Python 2.
pryelluw on HN
Consider libev or uvloop instead of custom event loop
From 1 comment
This is very interesting. did you consider using libev/uvloop - which are generally consider battle tested async frameworks ? Is there anything missing that prompted you to…
sandGorgon on HN
Questions
How does it differ from Scrapy?
From 1 comment
How does this differ from Scrapy?
monksy on HN
How does it handle throttling?
From 1 comment
Pretty cool project. It looks more enjoyable to use than BeautifulSoup. How does it approach throttling or rate limiting? I didn't see this mentioned in the readme examples.…
tedmiston on HN
Is it pure Python and what are its dependencies?
From 1 comment
I'm not able to figure out dependencies.. is this pure python ? Or are you using one of gevent, libev, uvloop, etc. Since it is py2, i suppose asyncio is out of the picture
sandGorgon on HN
How does it relate to Twisted?
From 1 comment
Awesome. It looks like you're reusing your own dependencies which is cool. Can you explain how untwisted relates to twisted a little more? I read the repo readme, but not sure…
tedmiston on HN
Alternatives named
Scrapy
From 2 comments
How does this differ from Scrapy?
monksy on HNFWIW I don't believe this would be overly convoluted in scrapy. I'd probably scrape the tags and quotes in one pass... Also, generator expressions would make the examples more…
bbernoulli on HN
BeautifulSoup
From 2 comments
Pretty cool project. It looks more enjoyable to use than BeautifulSoup. How does it approach throttling or rate limiting? I didn't see this mentioned in the readme examples.…
tedmiston on HNYes, BeautifulSoup is focused on parsing not crawling (BS supports the lxml parser out of the box). Scrapy is more of an opinionated scraping framework whereas BS is a parsing…
tedmiston on HN
Selenium with headless Chrome
From 2 comments
How useful are scrapers that don't execute Javascript these days? I find Selenium + PhantomJS (now Chrome Headless I guess) is pretty easy to drive from Python, and it works…
vosper on HNPhantom still gets blocked, as it reveals itself in the header. I've had success with a headless Chrome instance in a virtual display (xvfb) driven with Selenium, backed by…
dhruvkar on HN
Common Search
From 1 comment
Depending on your needs, sometimes it might be more interesting starting for there : https://about.commonsearch.org/ and then scrap whatever is missing or not fresh enough. The…
ldng on HN