DevOps Foundations: Safe Changes, Health Signals, and Rollback Boundaries

A release changes more than code

Deploying a binary is one part of a release. Configuration, database schema, permissions, message formats, and infrastructure can change at the same time. A rollback to yesterday's binary cannot necessarily reverse today's data migration or external side effect. List the changed state and identify which parts are reversible before starting the rollout.

Make a release small enough to attribute failures. If a latency regression follows a code update, a database migration, and a capacity change together, several explanations compete. Smaller independently observable changes shorten diagnosis and reduce the amount of state a recovery operation must handle.

Define health from the user's outcome

A process that is alive can still be unable to serve useful requests. A readiness check answers whether an instance should receive traffic; liveness answers whether restarting it is an appropriate recovery action. A liveness check that fails whenever a shared database is unavailable can restart every instance during a dependency outage and make recovery harder.

Use a small set of release signals: error rate, latency, saturation, and a representative business operation where appropriate. Compare them with a baseline and sufficient traffic. An empty canary with no errors is not evidence that the release is healthy. Define the observation window and rollback thresholds ahead of time so the decision is not improvised under pressure.

Roll out compatibility first

Old and new application versions often run simultaneously during a rolling deployment. Schema and message changes must tolerate that overlap. An expand-and-contract migration first adds a compatible shape, then moves readers/writers, and only later removes the old shape after old consumers are gone.

For example, renaming a database column by deleting the old name and immediately deploying new code can break old replicas. Adding a new column, migrating data, switching readers, and removing the old column in a later release makes the transition explicit. The exact sequence depends on write ownership and data volume; it needs a real backfill and consistency plan, not just a renamed field in an ORM model.

Separate deployment and exposure

A feature flag can limit exposure independently of artifact deployment, but it adds another state that must be owned and cleaned up. Define a default, an owner, and an expiry plan. Do not use a client-side flag as an authorization control. A user who can alter browser state can bypass that UI condition.

Rehearse recovery

In a disposable environment, deploy the previous artifact against the newly expanded schema and verify it still works. Exercise a paused rollout and a rollback, then check for background jobs or messages produced by the newer version that the older version cannot read.

Record who can trigger recovery and which dashboards or checks establish success. If a data change is irreversible, the plan may be a forward fix or restoration from a tested backup rather than a binary rollback. A recovery plan is credible only when its assumptions have been exercised.