Incident Timelines: Separate Event Time, Receipt Time, and Clock Evidence

The rollback appears to precede the failure

During a release, a dashboard shows a rollback at 12:00:04Z and a database timeout at 12:00:07Z. The responder concludes that rollback caused the timeout. Later, the database host is found to have a clock five seconds ahead, and its log shipper had also buffered events. The chart offered a total order, but the available evidence never supported that certainty.

Build a timeline as a set of observations with provenance. Event time is what the emitting process says happened. Receipt time is when your collector observed the record. Deployment-controller time is another clock domain. None becomes globally authoritative just because a logging service sorts it neatly.

Keep enough fields to disagree

Here is a fictional incident extract, not output from a verified production system. Explicit UTC timestamps keep formatting comparable while the source field exposes whose clock produced each event.

{"event_at":"2026-10-06T12:00:02.100Z","received_at":"2026-10-06T12:00:02.150Z","source":"gateway-a","trace_id":"release-check-42","event":"request_started"}
{"event_at":"2026-10-06T12:00:07.300Z","received_at":"2026-10-06T12:00:03.000Z","source":"db-b","trace_id":"release-check-42","event":"statement_timeout","duration_ms":850}
{"event_at":"2026-10-06T12:00:04.000Z","received_at":"2026-10-06T12:00:04.040Z","source":"deploy-controller","event":"rollback_requested"}

The second record apparently arrives before its event timestamp. That is a diagnostic signal, not proof of time travel or collector corruption. Possible explanations include an advanced producer clock, an incorrectly parsed timezone, a unit conversion defect, or fabricated client input. Trace identity and the gateway's local duration help narrow the possibilities.

Do not blindly subtract these fields and label the result "network latency." The difference combines transport, buffering, clock error, and timestamp semantics. A histogram of that difference is still useful for noticing changes, provided its name and documentation acknowledge the limitations.

Gather clock evidence with platform boundaries

On a Linux host using systemd-timesyncd, these commands inspect the synchronization service and its journal. They are not universal macOS commands, and a machine running chrony needs that service's corresponding diagnostics.

timedatectl status
timedatectl timesync-status
journalctl -u systemd-timesyncd --since "2026-10-06 11:50:00 UTC"
journalctl -u payment-worker --utc --since "2026-10-06 11:55:00 UTC" -o short-iso-precise

Capture host identity, boot identity where available, timezone configuration, synchronization status, and evidence of a recent correction. A current "synchronized" flag does not prove the clock was accurate ten minutes ago. Containers normally share the host clock, so inspecting only a container's timezone environment variable is insufficient. Preserve the investigation time as well as the incident window.

Establish order through stronger evidence

A request span with a parent-child relationship supports a causal claim that timestamp order alone cannot. A monotonically measured local duration supports how long one process waited. A database transaction identifier or deployment revision ties an effect to a specific state transition. Use these signals to challenge the initial story rather than fitting them into a predetermined narrative.

Within one producer, an explicit sequence number can preserve emission order through a clock step. Across producers, sequence numbers are not automatically comparable. Collector receipt order is useful for investigating shipping delays but may reverse application execution order. Document uncertainty when the evidence cannot distinguish two events.

Make the next incident easier

Standardize structured UTC event fields with explicit units for durations, avoid localized month names, and keep original offsets when they matter for diagnosing ingestion. Distinguish client-reported occurrence from trusted server observation. Retain deployment digest, environment, trace ID, and service instance without logging credentials or full sensitive payloads.

Use the timeline utility for a working reconstruction and the timestamp utility to examine suspicious units. Before finalizing a postmortem, write down which clock domains were involved, which ordering claims are proven, and which remain estimates. A useful action item is a reproducible delayed-shipping or clock-step test in a safe environment, not simply "add more logs."