Restore drills: the monthly ritual that makes backups real

We've said it before about Postgres and again about CloudNativePG: a backup nobody has restored is wishful thinking, not disaster recovery. What we hadn't written down is that the database is the easy 20% of that promise. A real cluster also has Velero snapshots covering whole namespaces, filestore syncs living outside any database, and secrets and config that need to reappear in the right place, in the right order. The drill that proves all of that together is a monthly ritual, not a one-off: pick a backup at random, rebuild the stack from it, time the whole thing, and log what broke.

The three things that actually need restoring

Every cluster we operate has backups covering three different failure modes, and each one lies to you in a different way if you never exercise it:

• Velero snapshots namespaces — deployments, services, PVCs, secrets, the works — to object storage on a schedule. It answers "the whole namespace is gone," not "one file is corrupted."
• Postgres gets WAL-archived continuously via CNPG or dumped on a schedule where it isn't. It answers "point-in-time recovery for exactly this table," not "bring the namespace back."
• Filestore / object data — Odoo attachments, uploaded media, anything living on a volume rather than in a row — gets synced to a separate bucket prefix, because Velero's snapshot of a PVC and a verified, restorable copy of its contents are not the same guarantee.

None of the three substitutes for the other two. A Velero restore that comes back healthy tells you nothing about whether last Tuesday's Postgres dump is actually readable, and a clean pg_restore tells you nothing about whether the filestore sync silently stopped three weeks ago. The drill has to touch all three or it isn't testing the thing that will actually happen during an incident.

The drill itself

Once a month, someone on the team draws a target at random — a cluster, a namespace, and a backup timestamp that isn't always last night's — and restores it into a throwaway namespace:

velero restore create restore-drill-$(date +%s) \
  --from-backup nightly-erp-prod-2026-08-01 \
  --namespace-mappings erp-prod:erp-drill \
  --wait

kubectl cnpg restore erp-postgres-drill \
  --cluster erp-postgres \
  --backup-name nightly-2026-08-01 \
  -n erp-drill

Filestore comes back the same way Velero brought back the PVC — the question the drill is actually asking is whether the objects inside it open, not whether the volume exists. So the checklist after both legs finish is deliberately mundane: log in as an application user, open a record with an attached file, run a report that hits every table in the schema. If a step that should fail (posting an invoice in a closed period, say) actually fails, that's a pass too — it means constraints survived the round trip, not just rows.

What we log every time

The artifact that matters isn't a green checkmark — it's a number and a date, written down before anyone can talk themselves out of it:

Six months of these logs is worth more than a disaster-recovery policy document, because it's the only evidence that the number in the policy document — our RTO target is under two hours — is something the team has actually hit, repeatedly, under no real pressure at all.

What the drill has actually caught

This isn't a theoretical exercise. Drills on client clusters have surfaced: a storage-layer snapshot policy that silently stopped completing for a week because a bucket lifecycle rule expired the underlying credentials; a Velero backup schedule that quietly excluded a namespace after a rename, so "backups are running" was true and "this namespace is in them" was not; and a filestore rclone exclude pattern that was too broad and had been dropping a directory of scanned documents for two months without a single failed job to alert on. None of those show up in a dashboard that only checks whether the backup job exited zero. All three showed up the first time someone tried to actually use the backup for something.

That's the whole case for running this monthly instead of "when we remember" or "after the first incident makes it obvious": a schedule you don't test is a belief, and the only way to convert a belief about backups into a fact is to restore one. It's a standard part of every cloud hosting engagement we run, for exactly that reason — not because we distrust the tooling, but because untested disaster recovery is the expensive way to find out the tooling was never the weak link.