We rebuilt an old Dell PowerEdge R720xd into a proper TrueNAS SCALE storage server, and it wouldn’t reliably boot. Not “wouldn’t boot” in the dramatic, smoke-and-sparks sense — it would boot most of the time, then occasionally fall through to a network boot screen instead, like the whole install had vanished. Reboot it again and it would come right back up. That kind of intermittent failure is the worst kind, because it’s very good at making you distrust things that are actually fine.
What made it strange was that this exact hardware had booted this exact way for roughly two years before we touched it. Nothing about the boot configuration should have changed. But something clearly had — it just hadn’t been us who changed it, and it hadn’t happened recently.
Ruling things out, in order
The instinct with an intermittent boot problem is to suspect hardware first, because hardware failures are the most familiar shape of “sometimes it works, sometimes it doesn’t.” So that’s where we started. We reseated the SAS and power cabling to the drive backplane in case a marginal connection was the culprit. No change. We checked the boot device order in the server’s BIOS, since a network boot happening at all meant something in that priority list was in play. It looked correct.
Then we found something that looked like a smoking gun: an old SD card still seated in the server’s internal media slot, left over from a completely unrelated hypervisor install from years earlier. An orphaned boot device sitting in a server is exactly the kind of thing that causes this class of problem — it’s a plausible, satisfying answer. We pulled it. The intermittent failure kept happening anyway. It was a real leftover and a real thing worth removing, but it wasn’t the cause. That’s the trap with a diagnostic red herring — it’s not that it does nothing, it’s that fixing it feels enough like progress that you can talk yourself into calling the job done.
Cables: ruled out. BIOS order: ruled out. Leftover boot media: removed, but not the cause. That left the bootloader configuration itself — the part everyone checks last because it’s the part that’s supposed to just work.
A setting that was never really pointing where we thought
This server uses Clover, a UEFI bootloader that sits between the server’s firmware and the operating system, and decides which installed system to actually hand control to. Clover had a named boot entry configured for the TrueNAS install, and that entry had apparently been working for about two years. But “apparently working” and “correctly configured” turned out to be two different things.
Clover identifies which disk partition to boot from using a setting called DefaultVolume, and the value it wants isn’t a drive letter or a friendly name — it’s the partition’s GPT PARTUUID, a long unique identifier baked into the disk’s partition table when it was created. Ours was configured with the correct identifier. It was just in lowercase. Clover’s matching against that value is case-sensitive, and a lowercase PARTUUID doesn’t match an uppercase one even though every character is technically identical. When the match fails, Clover doesn’t error out loudly — it just quietly falls through to the next boot option in line, which in our case was PXE network boot. That’s the “vanished” screen we kept seeing.
The fix was a one-line change: replace the lowercase PARTUUID with the identical value in uppercase. Boots have been consistent ever since.
Why this sat quiet for two years
The honest answer is we don’t know exactly why it worked before, only that it did — some earlier boot path, an earlier firmware behavior, or plain coincidence in how the entry was originally written must have avoided triggering the mismatch. A rebuild changes enough small things under the hood — how the boot entry gets written, what order UEFI evaluates its options in — that a config value which was technically wrong the whole time finally got exercised in a way that exposed it. The bug didn’t get introduced during the rebuild. It got revealed by it.
That’s a useful thing to sit with, because it cuts against the natural assumption that a system which has run fine for two years must be soundly configured. Sometimes it’s just never been asked the question that would expose the gap.
What this is really about
- An intermittent problem is not evidence for whichever cause is easiest to imagine — it’s evidence that whatever’s actually wrong only gets triggered under specific conditions.
- Ruling out the obvious candidates in order, even when it feels slow, is what eventually leaves you with the real answer instead of a convenient one.
- A fix that resolves a coincidental issue (like removing that orphaned SD card) can feel like progress without being the actual fix — don’t stop checking just because something felt satisfying to correct.
- Configuration bugs can hide in plain sight for years if the exact conditions that expose them never happen to occur — “it’s worked fine forever” isn’t proof something is right, just proof it hasn’t been tested yet.
The server’s been rock solid since. But the real lesson wasn’t the fix — it was how many plausible wrong answers we had to walk past to get there, and how a single lowercase character had been quietly waiting two years for the right moment to matter.
It’s the same discipline behind the documented minimum standard we hold every environment we manage to — rule things out methodically, not by convenient guesswork.
If something on your own network has “just worked” for years without anyone checking why, that’s worth a second look before it decides to stop. Get in touch and we’ll take one.
