In this guide 6 sections
Restores usually fail for four reasons. The snapshot caught files mid write. Configuration and encryption keys were never saved. Database logs were not captured, so recent rows are gone. Or the backup sat on the disk that failed. A restore drill finds each early.
The snapshot caught the disk mid write
A block level snapshot copies the disk as it is at one instant. If the database was writing a page at that moment, the copy holds half of it. The restored server boots, tries to replay its journal and stops with a corruption error. Half a page is useless.
This is called a crash consistent backup. It suits many file servers. It is risky for databases. The fix is an application consistent backup: flush tables, freeze the filesystem or use the database’s own dump or backup tool.
On Windows hosts, check that VSS writers report success. On Linux, a pre snapshot hook running fsfreeze or pg_backup_start is the usual remedy for a PostgreSQL or MySQL volume.
- Snapshot taken with the database still writing
- VSS writer errors ignored in the backup log
- No pre and post hooks around the snapshot
- Restore stalls at fsck or journal replay
Configuration and keys were left out
A restored disk image may boot and still not work. The application needs its environment file, its TLS certificates, its cron entries and the license file. If these lived outside the backed up paths, the service starts. Then it fails on the first request.
Encrypted volumes and encrypted backups add a harder version. If the key sits in a console you can no longer reach, or in a password manager owned by one person who has left, the data is unreadable. Full stop.
List everything the app needs to run, beyond its data. Store that list and the keys in a place that survives the loss of the server and of any single employee.
Recent data is missing because logs were not captured
Nightly dumps only hold the state at the time of the dump. Orders entered after it are gone unless the database transaction logs are archived as well. This is the common reason a restore boots cleanly and still disappoints the owner. The gap hurts.
PostgreSQL needs write ahead log archiving. MySQL needs binary logs kept and copied. SQL Server needs log backups on a schedule, and its full recovery model turned on. Without these, point in time recovery is not possible.
Check the newest record in the restored database against the time of the incident. The gap is your real recovery point.
- Archive mode off on PostgreSQL
- Binary logs purged before they were copied
- SQL Server in simple recovery model
- Newest restored row hours older than expected
Backups kept next to the thing they protect
A copy on the same disk, the same server or the same cloud account goes down with the original. Ransomware, a deleted volume or a billing suspension can take both at once.
Keep one copy in a separate account or region, with permissions that let servers write new backups but not delete old ones. Object lock or versioning stops a stolen credential from erasing history.
Also check retention. A backup job that quietly failed three weeks ago leaves you with one stale copy. Check the job log.
Run a restore drill that finds these early
Once a quarter, restore the latest backup into an isolated environment. Boot it, log in as a user, open a recent record and compare the newest timestamp with production. Write down how long each step took.
Treat every surprise as a ticket. A missing certificate or a skipped log archive costs minutes to fix on a quiet Tuesday, yet it costs hours during an outage.
Pick one system this month, restore it to a scratch server and record the result in the drill log.
Restore drill log
A one page log with one row per drill. Columns record the system, the backup used and start and finish times. They also note whether it booted, the newest record found against expected, and whether config and keys were present. Problems and their owners follow. A footer lists the next drill date.
Open a blank worksheet to printQuestions owners ask
How often should we test a restore?
Quarterly for important systems, and after any change to the backup tool, database version or storage. Smaller systems can be tested twice a year.
Does a successful backup job mean the restore will work?
No. The job only proves that files were written. Only a restore into a clean environment proves they can be used.
Why does the restored server boot but the app fails?
Usually configuration, certificates, secrets or environment variables sit outside the backed up paths. Compare the application’s startup log with the list of files it needs.
Where should encryption keys be stored?
In a secrets manager or vault that survives loss of the server, with at least two named people able to retrieve them.