PVE install nuked by silently bad ECC RAM

[khh]

My primary PVE node in my homelab recently experienced silent memory corruption. This is a two node cluster with a tie breaker, and the other node is healthy and is keeping everything critical running while I'm getting the main server back up.

The problem first presented as ZFS filesystem errors, but it took me a little while to figure out what the cause of the problem was. However, once I ran MemTest86 it became clear the system was experiencing memory corruption, and despite this being a system designed for and equipped with ECC-ram, the error had somehow gone undetected. However, by the time I figured it out the install has been damaged to a point where it will no longer boot due to unrecoverable file system errors. I'm sure there are corruptions in the datasets for VM and file storage as well.

I have now replaced the RAM, motherboard and CPU (since I don't know which part failed to detect the bad RAM), and the system now passes extensive testing with no errors detected. Do you have any advice on the best way to proceed to get the system back up and running while having to restore as little as possible from slow backups?

Proxmox is installed on mirrored NVMe drives. These drives also hold most of the VMs, There's also an array of eight 3.5" HDDs in a RAIDZ2 configuration with bulk storage for the file server and the compute node. I have full local backups of all the VMs themselves, but the storage pool is backed up to remote storage which is very slow to access so if possible I'd prefer to only have to restore individual damaged files rather than the whole dataset.

What is the safest way to reinstall Proxmox VE without nuking my existing datasets?

 

[UriShay-CyberSnap]

I'd treat the NVMe mirror and the HDD pool as two separate jobs.

The installer only wipes the disks you pick as target, so if you select just the two NVMe drives the RAIDZ2 pool isn't touched. Before that, boot a live ISO with ZFS (or the installer debug shell), import rpool read only and copy off anything you don't already have in a backup. With full VM backups that's probably only bits of /etc. Your VM configs are still on the healthy node under /etc/pve/nodes/OLDNODE, so grab them there. Then run pvecm delnode for the old node from the healthy one before joining the fresh install, and pvecm updatecerts on it if you keep the same name and IP.

For the HDD pool, import it on the new install, scrub it and check zpool status -v. That gives you the exact list of files ZFS knows are damaged, and those are the only ones you need from the remote. The catch with bad RAM is that some data can get corrupted before ZFS checksums it, and those files look fine to a scrub. If you have snapshots from before the first errors showed up, zfs diff between that snapshot and the live dataset lists everything written during the bad window. I'd compare hashes of just those files against the remote copy and only pull back the ones that differ.

Disclosure: I'm founder and CEO of CyberSnap.

 

[J-Rod]

ECC memory can typically only correct single-bit errors, not multi-bit. So a decent motherboard should alert you as soon as there is a single bit error so that you can replace the module as soon as possible. Once you get past a single-bit error you typically only have detection of two-bit errors and not correction, and beyond that it can actually be far worse where errors aren't detected at all.

I have worked on some nicer systems where they tell me exactly which module has the error by slot number on the motherboard and serial number, and I've also worked on some where there is just the tiniest of notes in the motherboard BIOS that it happened, and I had to find the module myself. In that case, I disable ECC in the BIOS, install as few memory modules as possible and then run MemTest86+ to narrow down the specific faulty module.