A client’s nightly backup had been running successfully, without a single failure, for as long as anyone could remember. Green checkmark, every night. It was also, for one specific piece of their data, not actually usable in a real recovery — and nobody would have known until the day they needed it.
The backup job wasn’t lying. It just wasn’t backing up what everyone assumed it was.
“The backup ran” and “you can restore from it” are different claims
A generic file-level backup takes a snapshot of whatever’s on disk at the moment it runs. For most files, that’s perfectly reliable. For an application with its own actively open database file — accounting software is a common example — a snapshot taken mid-use can capture the file in an inconsistent state: partway through a write, mid-transaction, technically “backed up” but not something the application can necessarily open cleanly on the other end.
The backup software has no way to know this happened. It successfully copied the bytes that were there. Success, from its point of view, and a real gap from yours.
Why this specific gap is so easy to miss
Most of the time, a live snapshot of an open database file is fine — many applications handle it gracefully, and the file reopens without issue. That’s exactly what makes the gap dangerous: it doesn’t fail every night, or even most nights. It fails unpredictably, tied to what the application happened to be doing at the precise moment the backup snapshot was taken. A clean backup log every night for months tells you nothing about whether last Tuesday’s snapshot, specifically, would actually open.
The only way to know for certain is to have the application itself export its data in a format designed for exactly this — a proper backup export, not a snapshot of a live file — before the backup software ever touches it.
What the actual fix looks like
Not a different backup product — a small step added in front of the existing one. A short script runs the application’s own built-in export first, producing a clean, application-verified file specifically meant to be portable and restorable. The regular nightly backup then picks up that export alongside everything else, so the thing actually being protected is a file the application itself guarantees is coherent, not a snapshot of whatever state it happened to be caught in.
This is a small, cheap fix. It’s just invisible right up until you need it — which is exactly the same shape of problem as the gap it closes.
Worth checking before you assume you’re covered
- Do any of your critical applications keep their data in a single actively-open file, rather than a proper database server?
- Has anyone actually test-restored a backup of that application recently, or just confirmed the backup job says “success”?
- If your backup software can’t tell the difference between a clean snapshot and a mid-write one, who can?
A backup that runs every night without complaint feels like proof it works. The only real proof is a restore that actually succeeds — and that’s worth testing before the day you’re forced to find out the hard way.