I had a massive cluster failure, long story short after a series of failures one node went rogue and started up VMs that were already running so it ended up causing massive corruption across the board. All the VMs that are already running work but minute I reboot them they fail. So I've been double checking all my file backups and then rebooting the VM and he ones that fail I restore the VM backup within Proxmox, and then restore the file backups.
I restored a backup of one VM but for whatever reason that backup never worked properly I guess because how I'm getting the control-D screen indicating a corrupt file system. Going to have to reinstall the OS and reconfigure everything. It's a mail server which is always a pain in the ass to setup too. ugh.. I really don't need this right now.
Is there a reason why a backup would not work? Anything I can do at this screen to recover the OS or am I pretty much SOL and have to reinstall? I did do the backup live (months ago before the failure), is that not advised? Maybe I would need to shut down each VM when I back them up?
EDIT: I panicked too early. Very stressed right now dealing with this incident. Turns out when you restore a backup it doesn't reattach the non backed up disks and the error was simply because the disk was missing, not corruption. I have it setup so the data disks don't get backed up, as I have file level rsync backups for those. Attached the disk and now it boots up.
I restored a backup of one VM but for whatever reason that backup never worked properly I guess because how I'm getting the control-D screen indicating a corrupt file system. Going to have to reinstall the OS and reconfigure everything. It's a mail server which is always a pain in the ass to setup too. ugh.. I really don't need this right now.
Is there a reason why a backup would not work? Anything I can do at this screen to recover the OS or am I pretty much SOL and have to reinstall? I did do the backup live (months ago before the failure), is that not advised? Maybe I would need to shut down each VM when I back them up?
EDIT: I panicked too early. Very stressed right now dealing with this incident. Turns out when you restore a backup it doesn't reattach the non backed up disks and the error was simply because the disk was missing, not corruption. I have it setup so the data disks don't get backed up, as I have file level rsync backups for those. Attached the disk and now it boots up.
Attachments
Last edited:
