Crawlee
Dev tools
web scraping library
SHOW HN2022
282points
80comments
#153of 2,570 in 2022
Top 6%of 2022 by points
How HN took it
LovedMost comments that take a side are positive
19 positive, 37 neutral or asking, 1 mixed, 3 critical, out of 60 comments about the project. Thread score +0.27.
Execution: 3 for, 1 againstDocs: 2 for, 1 againstDesign: 2 for, 0 againstIdea: 1 for, 1 againstPerformance: 1 for, 0 against
The thread, comment by comment
Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.
positive neutral or mixed critical about another commenter or something else the maker
What HN said
What people liked
Unified API for headless and HTTP scraping
From 2 comments
Looks like you took the good ideas from Scrapy's crawling engine and combined it with a great scraping API, which is all I ever wanted in a bot framework! I'm especially excited…
franga2000 on HNThis looks really neat, I love the idea of a single api for both traditional and headless scraping. From my experience headless scraping is in the order of 10-100x slower and…
samwillis on HN
Solid production experience and responsive maintainers
From 1 comment
We do a lot of web scraping (hundreds of millions of requests, multiple terabytes of data per month) and have been using Crawlee - previously known as Apify SDK - since its v0.20…
corford on HN
Good design, cleanly built on Scrapy-like ideas
From 1 comment
Looks like you took the good ideas from Scrapy's crawling engine and combined it with a great scraping API, which is all I ever wanted in a bot framework! I'm especially excited…
franga2000 on HN
Objections
Node.js and JavaScript are disliked as scraping languages
From 3 comments
> I'm not particularly happy to see JavaScript begin taking over another field as it truly is an awful language Sorry but this irked me. What exactly are you hangups with JS ?…
bilekas on HNnode.js dependency hell that often makes java's transitive dependencies on bigger projects look like hello world? Exception handling? Performance? No real multithreading afaik? I…
saiya-jin on HNNothing against dynamically typed languages, Python is one as well. Maybe "awful" was a bit of a strong word, but to quickly summarise my issues with JS: - all the syntax and…
franga2000 on HN
Rate limiting and proxy use may disrespect sites
From 1 comment
I see you basically recommend bypassing rate limits by using proxies etc? Why not just respect rate limits if set properly? A little bit of consideration for what/whomever is on…
boesboes on HN
Name Crawlee is awkward without an article
From 2 comments
This looks very cool, but am I the only one who has an aversion to any product/library calling itself the X instead of an X?
michaelmior on HNYes but on Github, the place that people sensitive to the definite article will see the most, it says "The web scraping and browser automation library." I opened a PR to change…
httbs on HN
Requests
Captcha solver plugin
From 1 comment
You can use any captcha solving service with Crawlee, but we plan to add a plugin to make its use much easier. It's on our roadmap.
mnmkng on HN
Better documentation, including login and cookie examples
From 3 comments
This looks really great. However, I can't find examples of how to handle scraping behind a login or a paywall, without having to 'type' credentials every time.
pqdbr on HNJust found this on the Apify documentation: https://docs.apify.com/tutorials/log-into-a-website-using-puppeteer#save-and-reuse-cookies Is there a similar guide for Crawlee?
pqdbr on HNIn a way, I hate you, but at the same time I love you. It's because I'm working on something similar to get data for my product. Seems like I'm going to use Ampify instead to…
alvis on HN
Python version
From 1 comment
Oh this looks lovely, congratulations! I would really like this but running in Python.
dennisy on HN
Questions
How does stealth and bot detection compare to other tools
From 2 comments
Hi! It looks really REALLY cool! Is there any kind of detection/stealthiness benchmark compared to libraries such as puppeteer-stealth or fakebrowser? Honestly no matter how…
OtmaneBenazzou on HNHi! Very cool project. Just out of curiosity, what trips up Crawlee on CreepJS? I haven't heard of anyone actually using it in production (actually don't think it's meant for…
thekeyper on HN
Can it use Apify proxies without paid hosting
From 1 comment
Is there a way to use Apify's paid proxies without paying for the hosting? That doesn't look like option on the website.
nerdix on HN
Does it run in AWS Lambda
From 1 comment
Can Crawlee run in AWS Lambdas?
cebert on HN
Alternatives named
Scrapy
From 2 comments
Looks like you took the good ideas from Scrapy's crawling engine and combined it with a great scraping API, which is all I ever wanted in a bot framework! I'm especially excited…
franga2000 on HNIn that they're trying to be crawling frameworks, and for sure Scrapy allows headed crawling via Splash, it's just not something I've needed or advocate for Scrapy also has a…
mdaniel on HN
Puppeteer-extra stealth plugin
From 2 comments
Hi! It looks really REALLY cool! Is there any kind of detection/stealthiness benchmark compared to libraries such as puppeteer-stealth or fakebrowser? Honestly no matter how…
OtmaneBenazzou on HNI'm not aware of a benchmark, but puppeteer-extra-plugin-stealth can be detected:…
jonatron on HN
Playwright
From 1 comment
In a way, I hate you, but at the same time I love you. It's because I'm working on something similar to get data for my product. Seems like I'm going to use Ampify instead to…
alvis on HN