Ask a room of infrastructure people whether they have backups and every hand goes up. Ask when they last restored one into a working environment and watch the hands come down.
This is not negligence. It is a completely rational response to incentives. Backups run nightly and produce a green tick. Restores are manual, disruptive, take a day nobody has, and produce nothing except the knowledge that the thing you already assumed was true is in fact true. Right up until it is not.
What a green dashboard actually proves
It proves a job ran and wrote some bytes somewhere. That is genuinely useful, and it is a long way short of what people take it to mean.
It does not prove the data is consistent. It does not prove you have the encryption key. It does not prove the restore target still exists, that anyone has the permissions to perform the restore, that the retention policy kept the version you need, or that the whole thing finishes inside the recovery time you promised the business.
Every one of those is a separate failure mode, and none of them is visible from the backup side.
The four things a first restore test finds
We run a restore every quarter for every client under management, and send the evidence. The first one almost always finds something. In rough order of frequency:
The recovery time is wrong. Not by a little. A database that backs up in forty minutes may take six hours to restore, because restoring is a different operation with different bottlenecks. Teams who have signed an RTO of four hours discover it on the wrong day.
Something in the dependency chain was never in scope. The database was backed up. The configuration that tells the application where the database lives was in a machine image nobody versioned. The certificate was on a host that no longer exists. You can restore the data perfectly and still have nothing that runs.
The permissions do not exist under pressure. The restore requires a role that only two people hold, one of whom has left, and the break-glass procedure has never been rehearsed. This one is particularly common in estates that tightened IAM after the backup policy was written.
Retention did not mean what everyone assumed. Thirty-day retention on a corruption that started six weeks ago is not a recovery plan. Ransomware in particular is a retention problem before it is a backup problem, because the useful copy is the one from before the encryption started, and nobody knows when that was.
What a real test looks like
It is not a fire drill and it does not need to be dramatic. Pick one real system. Restore it into an isolated environment. Start the application. Log in. Run a query that returns data you can recognise. Write down how long each step took and what went wrong.
The isolation matters — a restore test that touches production is a production incident waiting to happen. The recognisable data matters too, because “the restore completed” and “the restore produced usable data” are different claims and only one of them is worth anything.
Then write it down and send it to whoever signed the RTO. That last step is what turns an engineering exercise into an assurance one.
Why quarterly, and why evidence
Quarterly is often enough that drift gets caught before it compounds and rare enough that it does not eat the roadmap. Estates change: a new service arrives, a policy is edited, a region is added. A restore that worked in January proves progressively less as the year goes on.
The evidence matters more than it sounds. An auditor asking whether you can recover wants something other than an assurance. A board asking the same question wants a date and a result. And the discipline of producing a document someone else will read is what stops the test from quietly becoming a tick.
None of this is sophisticated. It is simply the part of resilience that is tedious rather than interesting, which is exactly why it is the part that gets skipped — and why the first test finds something almost every time.
Quarterly restore tests with the evidence sent to you are part of every managed engagement we run. See Managed Cloud Services.