Loading page...
distributed systems
12 minConsumer groups are the mechanism that turns Kafka partitions into scalable, failure-tolerant work units. Learn how rebalances, offsets, and lag interact under load before you ship your first serious pipeline.
Q&A
Q&AKeep consumer liveness within max.poll.interval.ms and commit only after durable work.
If processing exceeds max.poll.interval.ms, the group may consider the member failed and reassign its partitions; a later commit from the old generation is rejected. Reduce records per poll, process with bounded workers while preserving partition ordering, pause/resume appropriately, or adjust the interval based on measured worst-case work. Keep the fix from merely delaying failure detection indefinitely.
distributed systems
18 minDesign around Kafka group churn, payment idempotency, coordination limits, sagas, CQRS, and context propagation across async boundaries.
typescript
8 minUse for-await consumption, cancellation, and iterator return semantics without hiding queues.
Q&A
Q&APersist the business change and an event record together, then publish the event asynchronously.
A database transaction writes both the business state and an outbox row. A separate publisher retries delivery from that durable row. This avoids the gap where a database commit succeeds but broker publication fails. Delivery can still be repeated, so consumers should deduplicate by a stable event ID.
Q&A
Q&AOffsets are the durability checkpoint. Commit too early and you lose work; commit too late and you duplicate it.
Kafka commits represent when the consumer believes work has been finished. If the offset is committed before the business effect is truly durable, a crash can lose data. If it is committed only after a delay, the application may re-process messages when the consumer restarts. The correct strategy is to commit only after the state change is stored successfully and the work is idempotent enough to recover from retries.