Question regarding ProxmoxVE 4.3

[dragonauta]

Hello.
I was tasked to do a disk replacement on a very, very, very old Proxmox installation.

# pveversion
pve-manager/4.3-1/e7cdc165 (running kernel: 4.4.19-1-pve)

The server was installed on ZFS raid 1 Array (mirror).

Before continuing let me say I was told: This server WONT be upgraded. I don't know the reasons, but they won't be touching the OS.
Not my problem anyway and I properly warned about this (and its implications).

The physical server is 1000 km away (620 miles) from me.
Someone else in this location will be taking out the bad drive and putting in the new one.
The server should boot up, they'd give me remote access and then I would run the commands to replace the drive on degraded mirror.
I hope everything goes according to plan, but...

Now my question.
What happens if it doesn't boot?
Would the "rescue" option (from a recent iso) work?
I mean, If the server won't boot, they could use a recent iso and select the rescue mode to properly boot proxmox, and then repair the mirror?
Anyone has tested it?

Thank you

 

[leesteken]

Don't use a later ISO and don't upgrade/repair the ZFS pools. GRUB won't be able to boot from it (known issue that cause Proxmox switch to systemd-boot on ZFS) and the older kernels won't be able to use the pools. If it won't boot, maybe try the 4.4 ISO from here.
Try to make sure it can boot from the good drive (grub-install) using the existing running installation before shutting down Proxmox.
If the Proxmox will only boot with the bad drive, then put it back in, fix the boot on the good drive, or insert the new drive (as a third one) and add it to the mirror before removing the bad drive.
Eventually the system will fail with no way to fix it. If it does not boot anymore, just call it deceased. The owner/user surely realizes that it could die at any time and has a contingency plan (or backups that are regularly tested, otherwise the system cannot be important to him or her).

 

[dragonauta]

Thanks @leesteken .
Some time ago I replaced /dev/sdb on same server. This time it should be /dev/sda drive, so that's why I'm worried.

Let's say aaa is good drive, bbb is bad drive and ccc is the new drive
Last time I did this:

#save a backup and loaded on new disk
sgdisk --backup=table /dev/disk/by-id/aaa
sgdisk --load-backup=table /dev/disk/by-id/ccc

# replicated the aaa's partition table to ccc
sgdisk /dev/disk/by-id/aaa -R /dev/disk/by-id/ccc
sgdisk -G /dev/disk/by-id/ccc

# copied part1 and part2 (boot and zfs partition)
dd if=/dev/disk/by-id/aaa-part1 of=/dev/disk/by-id/ccc-part1
dd if=/dev/disk/by-id/aaa-part2 of=/dev/disk/by-id/ccc-part2

# check status and remove bad drive bbb
zpool status
zpool offline rpool /dev/disk/by-id/bbb

# replace the missing partition with new
zpool replace -f rpool /dev/disk/by-id/bbb-part2 /dev/disk/by-id/ccc-part3
update-grub /dev/disk/by-id/ccc

Maybe I'm missing the grub-install? or should be good?

 

[leesteken]

If that worked before, then I don't know why it would not work for the other drive. If the system can only boot of one of the physical connectors, you could switch the drives (as it does not matter for Proxmox or /dev/disk/by-id/).
I have my doubt about just copying part1 and part2 of the drive. Are those the BIOS boot and ESP partitions? Or are you just doing unnecessary extra steps like with the sgdisk commands? You might indeed need an extra grub-install before the current, still working, drive is bootable!
Please don't assume that I know what I'm doing, as I'm only guessing from the information you provide and don't know what I might be missing.