Distributed Systems Advanced: Rebalancing, Sagas, Consensus, and Trace Context

1. Kafka consumer progress and rebalances

Kafka consumer groups assign each partition to one group member at a time. Membership changes, partition changes, or missed polls can trigger a rebalance. CommitFailedException commonly means the consumer's generation/ownership changed before its commit was accepted. If processing one record takes longer than max.poll.interval.ms, the coordinator may consider the member unresponsive and reassign its partitions.

Keep polling while doing long work, lower max.poll.records, or move processing behind a bounded worker model while preserving per-partition ordering and commit discipline. Do not increase max.poll.interval.ms blindly; it delays detection of a truly dead consumer. Inspect group state and offsets with the CLI:

kafka-consumer-groups.sh --bootstrap-server broker:9092 \
  --describe --group billing-worker
kafka-consumer-groups.sh --bootstrap-server broker:9092 \
  --group billing-worker --topic payments --reset-offsets \
  --to-earliest --dry-run

Review reset output carefully before applying. Offset resets can replay large ranges or skip unprocessed data. Make handlers idempotent and commit only after durable processing.

2. Idempotency, deduplication, sagas, and 2PC

An idempotency key identifies one intended business command across retries. Persist the key and business result in the same database transaction; a uniqueness constraint arbitrates concurrent duplicate requests. An upsert can help, but its conflict target and update semantics must encode the actual business invariant. INSERT ... ON CONFLICT is PostgreSQL syntax; MySQL uses INSERT ... ON DUPLICATE KEY UPDATE; Oracle commonly uses MERGE. None automatically makes an external card charge atomic with the database.

Use a unique constraint on the idempotency key, then make the database-specific upsert return or preserve the first committed outcome:

-- PostgreSQL
INSERT INTO request_dedup (request_key, response_body)
VALUES (:key, :response)
ON CONFLICT (request_key) DO NOTHING;

-- MySQL
INSERT INTO request_dedup (request_key, response_body)
VALUES (?, ?)
ON DUPLICATE KEY UPDATE request_key = VALUES(request_key);

-- Oracle: MERGE against a one-row source; retain a unique constraint as the race guard
MERGE INTO request_dedup d
USING (SELECT :key request_key, :response_body response_body FROM dual) s
ON (d.request_key = s.request_key)
WHEN NOT MATCHED THEN
  INSERT (request_key, response_body) VALUES (s.request_key, s.response_body);

Read the stored response after a conflict. Do not overwrite the original result with a later retry payload, and handle a concurrent unique-key conflict according to the engine's transaction semantics.

For multi-service workflows, a saga records forward steps and compensating actions. Compensation is not a time machine: a refund is a new business action, not erasure of a prior charge. Two-phase commit coordinates participants through prepare/commit but introduces coordinator and availability coupling and is not supported uniformly across service APIs.

Redlock-style distributed locks have assumptions about clocks, process pauses, network delay, and lease validity. A lock lease does not prevent an expired holder from later writing. For critical state, use fencing tokens or a storage system that rejects stale writers; prefer a database constraint or idempotent state transition when that is enough.

3. Kafka acknowledgements under partition

With three brokers and min.insync.replicas=2, acks=all requires acknowledgement from the current in-sync replica set and refuses writes when that set falls below the configured minimum. This favors durability over availability during a partition. acks=1 can acknowledge after the leader writes locally, allowing more writes but risking loss if the leader fails before replication. Producer idempotence and retries reduce duplicates from producer retry behavior but do not make arbitrary downstream side effects exactly once.

4. Event sourcing, CQRS, and read models

Event sourcing stores a sequence of domain events as the source of truth; current state is rebuilt by folding events, often with snapshots to bound replay time. CQRS separates command validation/state changes from query-optimized read models. Read models may lag, so clients need an explicit consistency expectation and a way to correlate a command with its projection update.

Do not choose event sourcing simply to preserve an audit trail. Define event versioning, schema evolution, replay safety, retention, privacy erasure, and projection rebuild procedures first.

5. Trace context across threads and Kafka

W3C Trace Context carries trace and parent-span identifiers using headers such as traceparent. Instrumentation should inject context into Kafka record headers when producing and extract it when consuming. Thread pools can lose implicit thread-local context; pass context explicitly or use supported context-propagation instrumentation. OpenTelemetry baggage is for small, non-sensitive correlation values, not arbitrary user data. If logs use MDC, verify context is installed and cleared around each task so one request's identifiers do not leak into another.

6. Raft/ZooKeeper split-brain reasoning

A five-voter consensus group requires a majority of three. During a partition, only a side that can reach a quorum may elect a leader and commit new entries; a two-node minority cannot safely make progress as leader. With a 3/2 split, the three-node side can continue, while the two-node side must reject writes or remain unavailable. If no side has three, safety is preserved by stopping commits rather than allowing two independent leaders.

Interview and failure drills

Interview: Prevent CommitFailedException when processing exceeds max.poll.interval.ms: explain poll cadence, bounded record batches, cooperative handoff, processing time, and commit-after-durable-effect.

Troubleshooting: A payment is double-charged after a partition. Put an idempotency-key uniqueness check in the same transaction as the local payment intent, use a provider idempotency token for the external call, persist its result, and reconcile unknown outcomes instead of blindly retrying.

Challenge: With three brokers and min.insync.replicas=2, explain why acks=all rejects writes after the ISR drops to one, while acks=1 may accept a leader-local write that is lost on leader failure. State the durability/availability trade-off and the remaining role of producer idempotence.