Kubernetes CronJobs: Timezones, Missed Starts, and Duplicate Work

A report runs twice after recovery

A daily settlement report usually starts at 06:00 UTC. During a controller outage it misses that start, then begins late after recovery. Meanwhile, an operator manually creates a Job to recover the report. Two executions now contend for the same business period. Changing the CronJob's concurrency policy cannot retroactively establish which report owns that period.

A CronJob is a controller-maintained schedule, not a transaction coordinator. Job creation can occasionally be duplicated or missed, and the application should tolerate both. Decide whether late work is still useful and whether rerunning a period replaces, appends, or rejects its output before choosing manifest values.

Make calendar intent visible

spec.timeZone is stable from Kubernetes 1.27. Check the actual API capabilities of older clusters. Kubernetes schedules use five cron fields; they are not Quartz schedules with a leading seconds field. Do not embed TZ or CRON_TZ into .spec.schedule; use the dedicated field on supported clusters.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: settlement-report
spec:
  schedule: "0 6 * * *"
  timeZone: "Etc/UTC"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 900
  successfulJobsHistoryLimit: 2
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 600
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: report
              image: registry.example.com/settlement:2026-10-06
              args: ["report", "--previous-utc-day"]

The image is an illustrative placeholder, not a deployable dependency. The manifest expresses a fifteen-minute start window and a ten-minute Job lifetime. They answer different questions: how late creation may occur versus how long an execution may continue. Include image pull and startup time in your lifetime budget.

Forbid skips a new scheduled run when a previous Job from this CronJob is still active. It does not serialize manually created Jobs or another CronJob targeting the same database. Replace terminates the previous run in favor of a new one; unfinished side effects do not automatically roll back. Allow requires capacity planning for overlap.

Reconstruct what the controller did

kubectl get cronjob settlement-report -o yaml
kubectl get jobs --sort-by=.metadata.creationTimestamp
kubectl describe cronjob settlement-report
kubectl describe job JOB_NAME
kubectl logs job/JOB_NAME --timestamps

Inspect schedule, suspension, deadline, last schedule time, Job creation time, Pod startup, and application completion separately. lastScheduleTime is not proof that a report was successfully published. Events are temporary operational evidence, not a durable audit trail. Starting with Kubernetes 1.32, created Jobs have a scheduled-timestamp annotation that helps distinguish scheduled time from creation time; do not assume it exists on earlier versions.

The controller typically checks schedules approximately every ten seconds. Extremely short start deadlines can prevent execution. Many missed schedules also have controller safety limits, so an extended outage is not equivalent to an unlimited backlog queue. Observe behavior on the deployed version instead of using a cron expression as a recovery plan.

Deduplicate the business period

Use a database uniqueness constraint on a key such as (report_kind, business_date). Claim the period transactionally, publish output with a recoverable state transition, and record the Job identity for diagnosis. If a worker dies after publishing but before marking completion, the retry must recognize that published output rather than sending a second settlement email.

Avoid deriving the period solely from the worker's startup date. A delayed execution near midnight could choose the wrong day. Pass or resolve a stable scheduled period, and make manual recovery accept the same explicit period. If the product requires a regional morning time, use its IANA zone and test spring gaps and autumn overlaps; a UTC schedule has different local-clock behavior.

Inspect candidate expressions with the cron utility, then test the cluster's own scheduler and application deduplication. Suspend and resume a non-production CronJob, simulate an overlapping manual run, and verify one authoritative output for the period.