Event-Time Windows: Watermarks, Late Data, and Correctable Results

The dashboard loses a sale

A checkout event occurs at 12:01:40Z but reaches the analytics worker at 12:04:10Z after a mobile client reconnects. A dashboard grouped by arrival minute counts it at 12:04. A dashboard grouped by occurrence minute wants it at 12:01. Both can be internally consistent; only one matches a report titled "sales by checkout minute."

Now add a requirement that the dashboard update within seconds. Waiting forever for every offline client is impossible. You need a policy for provisional results, a point at which normal updates stop, and a path for corrections beyond that point. This is a product contract, not merely a streaming-library configuration value.

Give a window precise boundaries

Use half-open windows: a minute window includes its start and excludes its end. An event exactly at 12:02:00Z belongs to the next minute. This avoids double counting at adjoining boundaries and makes database backfills match the live path.

The following JavaScript sketch isolates a per-partition bounded-disorder watermark. It uses integer epoch milliseconds, rejects implausibly future events, and does not implement a complete stream processor. State persistence, partition assignment, deduplication, and idle-partition handling remain explicit responsibilities of the surrounding system.

const allowedDisorderMs = 120_000;
const maximumFutureSkewMs = 30_000;
const partitionMaxima = new Map();

function observe(partition, eventMs, receivedMs) {
  if (!Number.isSafeInteger(eventMs) || !Number.isSafeInteger(receivedMs)) {
    throw new Error("integer millisecond timestamps required");
  }
  if (eventMs > receivedMs + maximumFutureSkewMs) {
    throw new Error("quarantine future event");
  }
  partitionMaxima.set(partition,
    Math.max(partitionMaxima.get(partition) ?? -Infinity, eventMs));
}

function watermark(activePartitions) {
  if (activePartitions.length === 0) return null;
  if (activePartitions.some(partition => !partitionMaxima.has(partition))) {
    return null;
  }
  return Math.min(...activePartitions.map(partition =>
    partitionMaxima.get(partition) - allowedDisorderMs));
}

function minuteStart(eventMs) {
  return Math.floor(eventMs / 60_000) * 60_000;
}

The minimum across active partitions protects a slow partition from being ignored by a fast one. A newly assigned partition with no observations holds progress back. Silently removing idle partitions can advance the watermark, so define when a partition becomes idle and how returning events are treated. A production engine may provide these facilities, but its defaults still encode assumptions you must understand.

Late is not the same as invalid

An event behind the watermark is late relative to your arrival policy. It may still be an authentic sale. Conversely, an event ahead of the watermark may be a duplicate or an untrusted future timestamp. Validate the producer contract independently from the lateness policy. One misconfigured clock should not push a partition maximum years into the future and close every current window.

Deduplicate by durable event identity, not timestamp. Two legitimate purchases can share a timestamp, and a retry can arrive with a different receipt time. Specify how long identities remain deduplicated; replaying an old batch after that retention period needs a separate reconciliation strategy.

Publish correction-friendly output

Represent a minute aggregate with a stable window key, a monotonically increasing revision, and a status such as provisional or settled-under-policy. A late sale may produce revision 4 for the 12:01 window. Consumers should upsert the newer revision rather than append another independent total. If the sink receives revisions out of order, an atomic version comparison prevents an older update from overwriting the corrected value.

Maintain a backfill path for events outside the live correction horizon. Financial finality may require an audited adjustment instead of rewriting an already issued statement. Those are different business rules even when the same event-time pipeline supplies the data.

Exercise the uncomfortable cases

Test shuffled arrivals, equal timestamps, a quiet partition, a partition that resumes, duplicate IDs, future-clock corruption, and worker restart with persisted state. Assert totals after corrections, not just the first provisional output. A timeline can help visualize the sample sequence, but a chronological display does not prove causal ordering. Record both occurrence and receipt timestamps so operational delays can be examined without changing the report's business grouping.