Message Queue Patterns

Table of Contents

🔴 P0 — queues are the backbone of async architectures

Problem #

Asynchronous communication needs an intermediary that decouples producers from consumers, buffers bursts, and guarantees delivery semantics.

Core Queue Patterns #

Point-to-Point (Work Queue) #

Producer → [Queue] → Consumer₁
                   → Consumer₂  (competing consumers)
                   → Consumer₃

Each message processed by exactly one consumer. Good for task distribution.

Pub/Sub (Fan-out) #

Producer → [Topic] → Subscriber₁ (gets ALL messages)
                   → Subscriber₂ (gets ALL messages)
                   → Subscriber₃ (gets ALL messages)

Each message delivered to every subscriber. Good for event notification.

Delivery Guarantees #

Guarantee	Mechanism	Trade-off
At-most-once	Fire and forget	Fast, but messages can be lost
At-least-once	Ack-based with retry	No loss, but duplicates possible
Exactly-once	Idempotent consumers + dedup	Strongest, most complex/expensive

In practice, most systems use at-least-once with idempotent consumers. True exactly-once is expensive and often unnecessary if consumers are idempotent. See also: Idempotency.

Additional Patterns #

Dead Letter Queue (DLQ): Messages that fail processing N times are moved to a DLQ for manual inspection. Prevents poison messages from blocking the queue.
Priority Queue: Messages processed by priority rather than FIFO. Useful for tiered SLAs.
Delay Queue: Messages become visible only after a configurable delay. Useful for scheduled tasks.
Ordering: FIFO within a partition key (e.g. Kafka guarantees ordering within a partition). Global ordering is expensive and usually unnecessary.

Instinct #

Kafka for event streaming (high-throughput, ordered, replayable). SQS/RabbitMQ for task queues (simpler, fire-and-forget work distribution). Don’t use Kafka as a task queue; don’t use SQS as an event log. The tool choice follows the pattern.

PITFALL: Queues break strong latency requirements (e.g. <500ms response time). The queue overhead (enqueue + worker pickup + dequeue) adds latency that defeats the purpose of fast response times.
EXP: PerkPal’s external API processing required DLQs for handling poison messages from third-party webhooks. The DLQ pattern was essential for debugging failures without blocking the main processing queue.

References #

The Log: What every software engineer should know — Jay Kreps; foundational
Kafka vs RabbitMQ — Confluent

DDIA 2e Reference #

Chapter 11: Stream processing, message brokers, log-based messaging