How to write a rollback plan that covers the database

Each failure has its own route. Rolling back redeploys the previous build, rolling forward ships a small fix on top, and restoring replaces the database with a backup taken before the migration ran.

Code-only failures favour a rollback, so a broken checkout button goes back to the last good build within minutes and nobody touches the database at all. Data damage needs a restore, and every restore loses whatever customers wrote between the snapshot and the moment someone noticed the damage.

Pick the default route before release day. Write it on the plan. Do not improvise.

The whole trick is that old code keeps working against the new schema, so use the expand and contract pattern and add the new column first. Ship code that writes to both columns, backfill the history in batches so the table is never locked for long, then move the reads across.

Dropping the old column waits for a later release. Never drop and deploy together. One destructive migration turns every rollback into a restore, so keep the drop in a separate later release, once the new code has proven itself.

Write a down script for each migration and run it once on a copy of production, since an untested down script is a hope and not a plan.

Take a snapshot or a pg_dump just before the migration starts, then note the time, the file size and the storage bucket on the plan.

An untested backup is only a file. Restore it into a scratch database, run counts on orders, users and invoices, and compare them with live, because any gap means the backup is unfit.

Record how long the restore took. That is your real recovery time, and it is rarely the figure people guess, so write the measured number beside the promised one.

Name one person who can call a rollback, plus a deputy, because a committee wastes the only window you have while payments flow and error responses climb.

Write triggers as plain events. Checkout fails twice in a row, or the error rate stays above baseline for ten minutes, or the payment webhook queue grows past a set length.

Agree the cut-off too. After a set time, or once new customer orders have landed, rollback is no longer clean and the team moves to roll forward. Say so in advance.

Run the rollback once on staging with the same commands and the same person, because surprises are far cheaper there than in front of customers. Time it.

Store the plan on the release ticket, not in someone’s head or an old wiki page, and keep it to one page. Short plans get read.

After each release, note whether the plan was needed and which step you would tighten, because small edits after every deployment keep the document honest.