A backup you have not restored is a rumour
Every backup story starts the same way: the job is green, the log says “completed”, and nobody has tried to get the data back. The first restore is where the real configuration lives, and it is usually wrong.
The drill
Once a month, on a machine that is not the production host, restore the latest snapshot of one dataset and check three things: the file count, the total size, and one file that changed yesterday. The last check is the important one — it catches snapshots that are complete but a week old.
What the drill found
- Two directories were excluded by a pattern that matched more than intended — “cache” matched a directory called
cache-apithat held real data. - The restore ran as the wrong user, and half the files kept the wrong ownership. Services started, then failed on writing a lock file.
- The repository password was in the shell history of a host that had been rebuilt. It existed in one place only, and that place was gone.
None of this was visible from the backup logs. All three were visible within twenty minutes of a real restore.
The rule
If a restore has not been done, the backup is a hypothesis. Write the drill into the calendar, do it on a machine that is not production, and delete the copy afterwards — otherwise you will find it in six months and wonder what it is.