The RAID1 boot volume is missing,but the host is still running from RAM. How can I reinstall PVE without losing the VMs on a separate LVM-thin volume?

[rizkyraezr]

Hi all,

My Proxmox host (single node, hostname pve1) lost its boot RAID1 volume,
and I want to reinstall Proxmox on new disks while keeping the existing
VMs, which live on a separate virtual disk. I'd like a sanity check on
my plan before I touch anything.

HARDWARE
- Dell PowerEdge R630, PERC H730 Mini (hardware RAID), iDRAC 8
- VD 0: RAID1 (SATA SSD, bays 0-1), was the Proxmox OS. Now gone.
- VD "DATA-NEW": RAID1 (SATA SSD, bays 4-5), Online, redundancy 1.
- VD "DATA": old RAID1 (bays 6-7), Failed. Not used by any running VM
as far as I can tell.

WHAT HAPPENED
- iDRAC log shows disk 0 of VD 0 was dropped (error f0, reset, removed)
in January 2026. Nobody noticed, VD 0 ran degraded for ~8 months.
- In September 2026 disk 1 did the same, VD 0 failed and was deleted
by the controller. No foreign config is detected now.
- The host is still running from RAM. /etc/pve is still readable
(pmxcfs), but most binaries on disk (qm, pvesm, dmesg, tail, ...)
return "Input/output error". Bash builtins still work.
- 13 VMs are still running normally from DATA-NEW (4 more are stopped
or templates). I'd like to avoid any downtime beyond a planned window.

IMPORTANT: BACKUPS
Only 2 of the VMs have any backup. The remaining VMs have no backup at all,
and the data in them critical. A full-VM backup
(vzdump) does not exist for any of them, and I can't create one right now
because qm / vzdump are not usable on the host.

So my top priority is that NO VM disk is lost or modified during the
reinstall. I'd rather accept extra downtime or extra steps than take any
risk with the data on the "data" volume group.

CURRENT STORAGE (from /etc/pve/storage.cfg)
- dir: local
- lvmthin: vm (vgname vm, thinpool vm) <- old, only referenced by
"unused0" lines in some VM configs
- lvmthin: data (vgname data, thinpool data) <- all active VM disks
are here (scsi/ide/efidisk, plus cloud-init disks)

I already saved all VM configs (/etc/pve/qemu-server/*.conf) by reading
them from the running host.

PLAN
1. Shut down all VMs from inside each guest.
2. Replace bays 0-1 with new SAS drives, create a new RAID1 VD in iDRAC,
set it as boot device. Pull bays 4-5 during install so the installer
cannot touch DATA-NEW, then reinsert them afterwards.
3. Install the same (or newer) PVE version, hostname pve1, same network
(vmbr0).
4. Run vgscan / vgchange -ay data, check lvs, then add the storage with
pvesm add lvmthin data --vgname data --thinpool data
--content images,rootdir
5. Copy the saved .conf files into /etc/pve/nodes/pve1/qemu-server/
(after removing "unused0" lines pointing to the old "vm" storage and
ISO references to the old "local" storage).
6. Start the VMs one by one.

QUESTIONS
1. Is this the right way to re-attach existing VM disks after a fresh
install? Anything I'm missing for LVM-thin (thin pool metadata,
lvm.conf filters, auto-activation)?
2. Is it safe to drop the old .conf files into the new node's
qemu-server directory as long as the node name matches? Any issue
with vmgenid / smbios uuid / MAC reuse, or with cloud-init drives
(vm-XXX-cloudinit) that already exist on the thin pool?
3. Is there a way to shut down the guests cleanly while qm is
unavailable (some are Windows)? Otherwise they will be powered off
like a power loss.
4. Any way to make the installer safer around the existing
VG "data" (e.g. installing with the DATA-NEW disks physically
removed, versus keeping them)?
5. Anything else you would check before reinstalling?
6. Given there is no backup, what is the safest way to guarantee the
existing "data" VG / thin pool is never touched by the installer or
by the first boot of the new system? Would you also advise adding the
new RAID1 controller config in a specific order (create the new VD
first, then reinsert the DATA-NEW disks)?

I can post output of vgs / lvs / lsblk from the new install before
adding anything.

Thanks!

 

[Kingneutron]

First go through all the LXCs and VMs in the proxmox webui left pane, and write down / document the resources allocated in a spreadsheet.

Then attempt to copy everything in /etc to NAS or separate HD.

If the VMs are still running normally and can access the network, you can backup at least critical files from inside the VM to a shared drive with e.g. tar / rclone / rsync / winscp.

You could also put the Linux VMs in single-user mode, mount everything read-only and dd the disk(s) to a compressed image on the shared drive. To restore you would boot the vm with e.g. systemrescuecd and decompress on the fly + dd back to the in-vm disk(s).

https://github.com/nchevsky/systemrescue-zfs

For Win VMs you can do a full bare-metal backup/restore with Veeam Free Agent to a Samba share. Don't forget to create the bootable restore ISO per-vm, and copy it to the shared drive.

> Only 2 of the VMs have any backup. The remaining VMs have no backup at all, and the data in them critical.

If it's " critical " then it should have been backed up 3-2-1, with 1 copy offsite! I get that you inherited the mess, but there's NO EXCUSE for that kind of carelessness when you're running a business.

Save whatever you can, but you're gonna need to have That Conversation with the boss, setup Proxmox Backup server on separate hardware, and start documenting a DR Plan. With everything running from RAM in limp mode, you're probably going to lose some data. And I hope everything is on UPS.

You also need to implement monitoring for e.g. failing hardware, and start checking every.single.disk on at least a monthly basis. SMART as well as physical check for "failing" lights.

https://github.com/kneutron/ansitest/tree/master/proxmox

Look into the bkpcrit script, point it to NAS / separate HD, run it nightly in cron. Lots of other good scripts in there, including some for mirroring ZFS boot/root over network to iscsi - and recovering from same.

Wild Hair idea - if you can get proxmox installed to an external drive with ZFS boot/root, you might be able to mount-overlay " / " and friends by importing the rpool. I have seen this with ext4/XFS root and had to guard against it. But ONLY TRY THIS AS A LAST RESORT, after everything else has been backed up. You may also be able to soft-symlink essential commands to entries in /dev/shm and try running them from there.

You're lucky the d--nthing is still even running -- most systems would have fallen over - and it still might, if you look at it funny. Act quickly.

 

[alexskysilk]

1. Is this the right way to re-attach existing VM disks after a fresh
install? Anything I'm missing for LVM-thin (thin pool metadata,
lvm.conf filters, auto-activation)?

Yes. you're not missing anything.

2. Is it safe to drop the old .conf files into the new node's
qemu-server directory as long as the node name matches? Any issue
with vmgenid / smbios uuid / MAC reuse, or with cloud-init drives
(vm-XXX-cloudinit) that already exist on the thin pool?

Yes. its safe even if the node name DIDNT match. if you have any outside services that actively used ssh to connect to the node, it may be better for it NOT to have the same name as before.

3. Is there a way to shut down the guests cleanly while qm is
unavailable (some are Windows)? Otherwise they will be powered off
like a power loss.

Yes. shut them down from inside the guest.

4. Any way to make the installer safer around the existing
VG "data" (e.g. installing with the DATA-NEW disks physically
removed, versus keeping them)?

The installer doesnt usually care about anything other then the default vg named PVE. Its probably a good idea to either pull out the data disks or have ld down before installing just to avoid potential mistakes.

5. Anything else you would check before reinstalling?

as long as you have your bridge names and vm conf files you're pretty much all there. I'd make a copy of /etc/pve/storage.cfg just for reference in case you need to recreate mountpoints, but this is fairly easily handled even if you didnt by editing the store names in the conf files.

6. Given there is no backup, what is the safest way to guarantee the
existing "data" VG / thin pool is never touched by the installer or
by the first boot of the new system? Would you also advise adding the
new RAID1 controller config in a specific order (create the new VD
first, then reinsert the DATA-NEW disks)?

see above disable the ld before installing. or barring that just pull out the disks. I would add one more step in any case- before doing ANYTHING, boot to a linux livecd and make a ddimage backup of all your lvs. at least then you'll have a backup in any eventuality.