pyspider
Dev tools
web crawler
SHOW HN2014
How HN took it
3 positive, 14 neutral or asking, 0 mixed, 4 critical, out of 21 comments about the project. Thread score 0.00.
Design: 2 for, 4 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Nice UX and API, more user-friendly than scrapy
From 3 comments
This looks really nice. The API seems more user-friendly than scrapy's.
meowface on HNI really like the flow/UX. Congratulations! Nice job! What is the roadmap? I am really inside scraping, it is one of my daily job. I could consider to integrate it in one of my…
_bitliner on HNThat is a nice tool....nice work!
bowlofstew on HN
Objections
Package layout uses a generic top-level 'libs' name that may collide when installed
From 3 comments
Take a look at source code. The package hirarchy is not pythonic (use "libs" as top package is not a good idea).
zbb on HNThe issue is that if you have two packages installed, and both use "libs" as their top-level package, they'll collide. Use "projectname.common" instead.
vertex-four on HNBecause the name "libs" is now installed into the global module namespace. It's better to use a less generic name.
ngoldbaum on HN
Setup instructions for the PhantomJS proxy are unclear
From 1 comment
You are running ``` phantomjs phantomjs_fetcher.js ``` and using it as proxy? The setup instructions are a bit unclear on this.
maratc on HN
Requests
Comparison with scrapy
From 2 comments
How does this compare to scrapy? Why would I use one over the other, or is either a fine choice?
adam-_- on HN+1 Scrapy comparison please Can you compare to scrapy as requested by other posters. Why could you not build on top of scrapy and leverage celery for scheduling etc…
skillachie on HN
PostgreSQL backend support
From 1 comment
Nice project! I do wish it supported a PostgreSQL backend rather than (or as well as I guess) MySQL.
mrmondo on HN
GUI-based scraper interface like Portia
From 1 comment
Any plans for a gui based web scraper interface similar to portia?
Immortalin on HN
Questions
How does it compare to scrapy in performance and use
From 2 comments
How does this compare to scrapy? Why would I use one over the other, or is either a fine choice?
adam-_- on HNHow's the performance compared to scrapy?
bjblazkowicz on HN
How the queue and architecture work
From 1 comment
Because I already have a powerful distributed architecture. I was curious about the architecture of pyspider. For example, how the queue is handled? Is it centralized? Is there a…
_bitliner on HN
What JavaScript page support means
From 1 comment
Furthermore, what you mean with `Javascript pages supported`? Could I just specify where it has to click or do I need to make a reverse engineering of the ajax calls?
_bitliner on HN
Alternatives named
Scrapy
From 2 comments
This looks really nice. The API seems more user-friendly than scrapy's.
meowface on HNHow does this compare to scrapy? Why would I use one over the other, or is either a fine choice?
adam-_- on HN
Portia
From 1 comment
Any plans for a gui based web scraper interface similar to portia?
Immortalin on HN
Celery
From 1 comment
+1 Scrapy comparison please Can you compare to scrapy as requested by other posters. Why could you not build on top of scrapy and leverage celery for scheduling etc…
skillachie on HN