Kafka Consumer Groups: Architecture & Common Pitfalls

Architecture

Kafka consumer groups are designed for horizontal scale across independent application instances. Each topic-partition is assigned to only one consumer in the group at a time, while different groups can read the same topic independently. That gives Kafka its natural fan-out model: one group can process data once for a pipeline, while another group can power analytics or a replay pipeline without interference. The hidden complexity is that the cluster’s coordinator and group leader maintain assignment state, and any membership change triggers a rebalance. A well-behaved application therefore treats rebalances as normal-but-expensive control-plane events, not as a curiosity.

@Bean
public ConsumerFactory<String, OrderEvent> consumerFactory() {
  Map<String, Object> props = new HashMap<>();
  props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker-1:9092,broker-2:9092");
  props.put(ConsumerConfig.GROUP_ID_CONFIG, "orders-processing-v2");
  props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
  props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
  props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "200");
  return new DefaultKafkaConsumerFactory<>(props, new StringDeserializer(), new JsonDeserializer<>(OrderEvent.class));
}

@Bean
public ConcurrentKafkaListenerContainerFactory<String, OrderEvent> kafkaListenerContainerFactory(
    ConsumerFactory<String, OrderEvent> consumerFactory) {
  ConcurrentKafkaListenerContainerFactory<String, OrderEvent> factory =
      new ConcurrentKafkaListenerContainerFactory<>();
  factory.setConsumerFactory(consumerFactory);
  factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE);
  factory.setConcurrency(3);
  return factory;
}

@KafkaListener(topics = "orders.created", groupId = "orders-processing-v2")
public void onOrderCreated(ConsumerRecord<String, OrderEvent> record, Acknowledgment ack) {
  OrderEvent event = record.value();
  try {
    orderService.process(event);
    orderRepository.save(event);
    ack.acknowledge();
  } catch (Exception ex) {
    logger.error("Order processing failed for key={}", record.key(), ex);
    throw ex;
  }
}

org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member.

at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:1223)
at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1111)
at com.example.orders.OrderWorker.process(OrderWorker.java:87)

ERROR WorkerThread: message processing exceeded max.poll.interval.ms

java.lang.IllegalStateException: Consumer is not registered to any group
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1217)

Why consumer groups matter

At a high level, Kafka consumer groups give a queue-like system enough coordination to behave like a distributed work pool. A topic can have many partitions, and multiple consumers can read from the same topic concurrently as long as they are in different groups. When consumers share a group, only one member sees a given partition at a time, which creates a natural boundary for parallel processing.

This design is why Kafka feels both simple and dangerous. The simple part is that you can scale workloads by adding more consumers. The dangerous part is that the hidden control plane—rebalancing, lag, offsets, and coordinator state—is easy to misunderstand. The wrong configuration creates duplicate processing, load imbalance, or a quiet stop-the-world pause during a rebalance.

Partition assignment is the real architecture

A consumer group is not a queue. It is an assignment protocol. The coordinator tracks group membership and the leader handles partition ownership. If you add consumers, Kafka may rebalance ownership quickly, and for a short period the system can pause or slow down. If consumption is fast, the rebalance cost is manageable. If the app performs expensive tasks within the poll loop or triggers frequent membership changes, the cost becomes visible as lag and jitter.

A common engineering mistake is treating concurrency as “the number of threads I should spin up.” In Kafka, the practical limit is usually the number of partitions in the topic or the number of consumers that can carry the workload. If your topic only has four partitions, running 20 consumers in one group yields 16 idle workers. That is not a performance win; it is just more coordination.

Offset semantics are where correctness lives

Kafka offsets are the durable record of where a consumer has processed data. The important rule is straightforward: commit after the business outcome is durable, not before. If you commit offsets immediately after reading from Kafka but before the database write succeeds, you can lose data on a crash. If you commit too late, you create duplicates on retriable failures.

The right pattern is often: poll messages, process them, persist the business state, then call commitSync() or rely on ack logic in the framework. This design makes the offset an audit point of successful processing rather than a guess about the future. If your consumer is idempotent, duplicates are manageable. If it is not, the offset model becomes a silent source of inconsistency.

Rebalances are not failures; they are state transitions

When a member joins or leaves the group, the coordinator triggers a rebalance. During a rebalance, partitions are reassigned and consumer members need to pause or pick up new work. This process is entirely normal, but it becomes a production issue when it happens too often.

The most common causes are:

• frequent consumer restarts;
• unstable network conditions;
• blocking work inside the consumer loop;
• too-short session timeout values;
• too-low heartbeat frequency compared to processing latency.

A single expensive blocking call can delay heartbeats and make the coordinator believe the member is dead. The result is a group churn that feels like a Kafka outage even when the cluster is healthy. The fix is to keep the poll loop responsive, keep processing outside the consumer thread, and tune the group semantics around the real arrival pattern of your events.

Backpressure is about business reality

The durable streaming system does not magically preserve fairness. If a consumer is slower than the producer, the lag grows. That is not a signaling bug; it is a throughput mismatch. You need explicit backpressure policies so the application can react instead of silently falling behind.

One good pattern is to use bounded max.poll.records and a fixed work queue. For example, if your consumer processes 200 records per poll, then your internal task queue should not accumulate unbounded memory. You should also set maximum concurrency based on CPU or IO limits. If a worker crashes, the consumer should use retry semantics and not allow a poison message to block the entire partition forever.

The most resilient applications treat lag as a product metric. If the lag is rising consistently, that is not just an operational issue; it is a product reliability risk. In many systems, a message backlog is the first sign that the system is no longer aligned with real-world demand.

Production-grade pattern: idempotent work plus explicit commit control

A robust Kafka consumer is usually built around a few rules:

1. Always use a unique business key for deduplication.
2. Persist business effects before acknowledging the message.
3. Offload slow side effects to async workers if necessary.
4. Tune max.poll.interval.ms, heartbeat, and session.timeout.ms to the actual work profile.
5. Alert on lag and rebalance churn instead of only on crash errors.

That pattern avoids the worst outcomes: duplicate invoices, partial state writes, and rebalance storms. In practice, the code is not just about Kafka; it is about making the app’s processing lifecycle match the durability semantics of the broker.

Final takeaway

Consumer groups are one of Kafka’s most important design levers, but they are also one of the easiest places to lose production time. The system is simple at a conceptual level: partitions are assigned to members of a group, and each partition is processed exactly once per group. The real challenge is operational correctness under real-world timing, failures, and uneven workloads.

If you approach consumer groups as a coordination and failure-handling problem rather than a raw throughput problem, your architecture becomes easier to reason about and safer to operate. That is the difference between a Kafka app that works in a demo and one that survives in production.