KQ on HackerShow

Dev tools

job queue library

SHOW HN2016

Simple Job Queue for Python Using Kafka

Shown by joohwan on Nov 1, 2016. Click the card to turn it over.

116points

30comments

#307of 2,264 in 2016

Top 14%of 2016 by points

How HN took it

MixedNo clear lean

2 positive, 5 neutral or asking, 1 mixed, 4 critical, out of 12 comments about the project. Thread score -0.16.

Idea: 1 for, 3 againstDocs: 0 for, 1 againstExecution: 1 for, 0 against

Each row is a top-level comment and its replies, in HN's order. Select a cell to open that comment on HN.

positive neutral or mixed critical about another commenter or something else the maker

Kafka is a poor fit for job queues due to head-of-line blocking and per-message semantics
From 2 comments
- One of the biggest problems with treating Kafka as a job queue is that you suffer from head-of-line blocking. Kafka doesn't expose per-message visibility/acknowledgement…
  Xorlev on HN
- To me, Kafka as a job queue is a painful impedance mismatch. To achieve that, you need to: 1. Figure out which Kafka broker you're using. The concept of a consumer and consumer…
  jondot on HN
Requires Zookeeper, not mentioned in dependencies
From 1 comment
- Don't you need zookeeper to use Kafka? Perhaps you should mention that in your dependencies.
  stratospherein on HN
Memory-bound Redis queues like RQ cause production issues, which KQ may avoid
From 1 comment
- RQ is certainly useful (with finer control over messages), but we've been having a lot of problems with it lately in production due to it being memory bound (Redis). Improper…
  joowani on HN

Document when to use KQ versus RQ or Celery and Kafka's limits
From 3 comments
- RQ is certainly useful (with finer control over messages), but we've been having a lot of problems with it lately in production due to it being memory bound (Redis). Improper…
  joowani on HN
- Ok, that makes sense. I've never dealt with the scale you are talking about so have yet to run into these issues with redis. It may be best to put this info in the readme so…
  reubano on HN
- Thank you for the excellent feedback and insight. I will definitely give the pdf a read. I agree with all of your points, which admittedly I was not fully aware of when I first…
  joowani on HN
Fix the docs link that returned permission denied
From 1 comment
- Getting a Permission Denied page for the docs at http://kq.readthedocs.io/en/master/
  azundo on HN

Why use this over RQ?
From 1 comment
- Why should I use this over the simplier RQ?
  reubano on HN
What kinds of jobs are put into job queues?
From 1 comment
- What kind of jobs do people put into such queues? Because I've only seen job queues used as a poor man's RPC brokers and (poor?) replacement for crontab. I think I may miss some…
  dozzie on HN