Release checklists that cut down on rollbacks

Pilots use checklists not because they forget how to fly, but because the cost of forgetting one step is very high and the steps are boring. Software releases are similar. Teams that introduce a short, enforced checklist commonly see fewer releases end in a rollback.

The list is not clever. What makes it work is that every item exists because something went wrong, and it is checked by the pipeline wherever possible, not by memory.

The twelve checks

1. Migrations are backwards compatible. The previous app version must run against the new schema, because during rollout both versions are live. A dropped column once took a checkout page down for eleven minutes.
2. Migrations have been run on a production-sized copy. A harmless-looking index addition locked a table for twenty minutes in production after finishing in two seconds on staging.
3. Feature flags exist for anything user-facing and risky, with the default set to off.
4. New environment variables are present in production before the deploy, not after it.
5. Third-party credentials and certificates are not expiring within 30 days.
6. The build is the exact artefact tested in staging. No rebuilds between environments.
7. Automated tests are green, and the three flows that earn money (sign-up, checkout, billing) have passed a manual smoke test on staging.
8. Dashboards and alerts exist for any new endpoint or job. If we cannot see it, we cannot roll it back with confidence.
9. Error budget check: if the service has already used its monthly budget, the release needs a named approver.
10. Support and account teams know what is changing, in plain language, before customers do.
11. The rollback plan is written down in one line, and someone has confirmed it works. Data migrations that cannot be reversed are flagged and need explicit sign-off.
12. A named person is watching for the first 30 minutes after release, with a clear stop condition.

Automate what you can, own what you cannot

Roughly half the items are enforced in the pipeline: compatibility checks, artefact promotion, certificate expiry and alert coverage. The pipeline refuses to continue until they pass. The rest need a human, so the release ticket contains them as required fields. A release cannot be scheduled with a blank rollback line.

release-gate:
  stage: verify
  script:
    - ./scripts/check-migrations-backward-compatible.sh
    - ./scripts/check-secret-expiry.sh --days 30
    - ./scripts/check-alerts-defined.sh --service $SERVICE
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

Keep the list short and alive

The list stays at twelve on purpose. When a new incident suggests a thirteenth check, we look for something to remove, usually an item that automation now makes redundant. A long checklist gets ticked without reading, and that is worse than no list at all.

Every item links to the incident review that created it. New engineers read these in their first week, and it teaches them more about the system than any architecture diagram.

What changed besides the number

The bigger benefit was cultural. Releases became uneventful, so people stopped avoiding Fridays and batching changes into large, risky drops. Smaller releases are easier to check and to roll back, which reinforces the loop. If your releases feel tense, start by writing down the last three incidents, turn each into one question, and ask it before the next deploy. You will have a useful checklist in an afternoon.