Transfer root filesystem between drives

G

gabrioth

Active Member
Proxmox Subscriber
Oct 4, 2019
37
8
28
41
I have one of my proxmox nodes' root filesystem on a z-pool called rpool, consisting of three 512GB SSDs in raidz.
I would like to fully transfer that filesystem with name (rpool), subvolumes, settings and all, to two 1024GB M.2 nvme drives in a zfs raid-1 mirror.
Then take the three SSDs out, reboot the system and use the nvme drives instead.

Is this even possible through some simple process?
 
Since you are moving to larger disks, the recreating partition step should be a lot easier, just clone it to the new disks with sgdisk <old disk> -R <new disk> and then create new random UUIDs for the partitions on the new disk with sgdisk -G <new disk>.

Then you can expand the partition 3 in which ZFS lives.
 
  • Like
Reactions: RolandK
Thanks to both of you!

Yes, the drives are larger, but I'm migrating from three 512GB drives in R5 to two 1024GB drives in R1.

Usable pool size should be close to identical, but will it really be possible to clone the partitions if I don't have the same number of drives (and will not be running the same raid level)?
If it holds any importance, I'm only using about 300GB of the 1024GB total in the raidz1-pool, so less than half.
 
Last edited:
gabrioth said:
Usable pool size should be close to identical, but will it really be possible to clone the partitions if I don't have the same number of drives (and will not be running the same raid level)?
Click to expand...
Cloning of the partitions should work and is the easiest way to get them like the Proxmox VE installer would set them up. Just don't forget to randomize the GUIDs.

ZFS has the two levels, indicated by the two CLI tools. One to handle the pool side of things (zpool), that includes the vdevs which handle redundancy, and one to handle the dataset and file system side (zfs).

If your Proxmox VE installation is not ancient, the bootloader with a ZFS root should be systemd-boot. See this section in the Docs on how to determine which bootloader you are currently using. The sections before that explain the whole concept about how the boot partitions work on Proxmox VE and how they are kept in sync across multiple disks to be able to handle the failure of one.

TL;DR: yes it should work because the ZFS file systems don't care much about the underlying redundancy provided by the vdevs. Proxmox VE takes care of keeping the EFI partitions, that contain the boot image, in sync.
 
Allright, then that is probably what I'll do.

Is this the correct command to randomize all the GUIDs on a drive?
sgdisk -G /dev/sdX


Also, I'm guessing that cloning the partitions will not also clone the data to the new drives and that I will need to perform the below steps:
zfs snapshot -r rpool@migrate
zfs send -R rpool@migrate | zfs recv new-rpool -F
zpool set bootfs=new-rpool/ROOT/pve-1 new-rpool

And then rename the pool within the live-ISO and boot from it.
 
Last edited:
gabrioth said:
Is this the correct command to randomize all the GUIDs on a drive?
sgdisk -G /dev/sdX
Click to expand...
Yep.

One more thing that I can highly recommend in such a situation: Set up a smallish VM that is similar to the physical one, with 3 disks. Install Proxmox VE in it and take a snapshot ;)
The disks don't have to be very large.
Then add 2 more, larger disks and go through the procedure. This way you can test it without a lot of stress because in case something went wrong, your infrastructure is not affected, and rolling back to the snapshot gives you an easy way to start over.

This is how I test our such procedures the first time, where I think that it should work, but haven't done it yet.
 
  • Like
Reactions: takeokun
aaron said:
Yep.

One more thing that I can highly recommend in such a situation: Set up a smallish VM that is similar to the physical one, with 3 disks. Install Proxmox VE in it and take a snapshot ;)
The disks don't have to be very large.
Then add 2 more, larger disks and go through the procedure. This way you can test it without a lot of stress because in case something went wrong, your infrastructure is not affected, and rolling back to the snapshot gives you an easy way to start over.

This is how I test our such procedures the first time, where I think that it should work, but haven't done it yet.
Click to expand...
That is actually very good advice :)
I'll do that
 
aaron said:
Yep.

One more thing that I can highly recommend in such a situation: Set up a smallish VM that is similar to the physical one, with 3 disks. Install Proxmox VE in it and take a snapshot ;)
The disks don't have to be very large.
Then add 2 more, larger disks and go through the procedure. This way you can test it without a lot of stress because in case something went wrong, your infrastructure is not affected, and rolling back to the snapshot gives you an easy way to start over.

This is how I test our such procedures the first time, where I think that it should work, but haven't done it yet.
Click to expand...
This might be a bit off topic, but, actually, I am unable to boot into my new VM with either the 7.2 or 7.1/7.0 Proxmox VE ISOs.
It leaves me with the following error at boot:
""UEFI QEMU DVD-ROM QM00003 " from PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x1,0xFFFF,0x0): Access Denied"

If I load something like Ubuntu server 22.04 or Debian 11 ISO then the machine boots just fine.
 
@aaron , may i ask what is needed to be done to transfer grub/bios-legacy-boot to a new, smaller disk ? your documentation refers to uefi and the disk-replacement-documentation in the proxmox wiki is also unclear about grub boot and it's not clear, what proxmox-boot-tool does regarding grub on re-init/format

with grub-install i'm getting

# grub-install /dev/sdl
/usr/sbin/grub-install: 24: warn: not found
 
Last edited:
ah, ok , i see in "init" that there is grub being installed

but i don't really understand, how grub legacy boot works with "newer" proxmox installation.

where exaclty is grub being installed when legacy boot? what's on "BIOS boot" partition no 1 ?

sorry, don't want to hijack the thread. if not ok, i can open a new thread with this

Code: 
# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with legacy bios
CD88-E64B is configured with: grub (versions: 5.15.39-1-pve, 5.4.189-2-pve)
root@pve4-knju:/dev/disk/by-uuid# proxmox-boot-tool init /dev/sdl2
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
UUID="CD88-E64B" SIZE="536870912" FSTYPE="vfat" PARTTYPE="c12a7328-f81f-11d2-ba4b-00a0c93ec93b" PKNAME="sdl" MOUNTPOINT=""
Mounting '/dev/sdl2' on '/var/tmp/espmounts/CD88-E64B'.
Installing grub i386-pc target..
Installing for i386-pc platform.
Installation finished. No error reported.
Unmounting '/dev/sdl2'.
Adding '/dev/sdl2' to list of synced ESPs..
Refreshing kernels and initrds..
Running hook script 'proxmox-auto-removal'..
Running hook script 'zz-proxmox-boot'..
No /etc/kernel/cmdline found - falling back to /proc/cmdline
Copying and configuring kernels on /dev/disk/by-uuid/CD88-E64B
    Copying kernel 5.15.39-1-pve
    Copying kernel 5.4.189-2-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.15.39-1-pve
Found initrd image: /boot/initrd.img-5.15.39-1-pve
Found linux image: /boot/vmlinuz-5.4.189-2-pve
Found initrd image: /boot/initrd.img-5.4.189-2-pve
done
 
Last edited:
gabrioth said:
If I load something like Ubuntu server 22.04 or Debian 11 ISO then the machine boots just fine.
Click to expand...
Did you select UEFI and left the "Pre-Enroll Keys" enabled for the UEFI disk? If so, try to delete the EFI disk and recreate it with that checkbox disabled.
 
  • Like
Reactions: _gabriel
I managed to complete the whole operation in a lab environment :)
When running it on my production machine I get an error message from proxmox-boot-tool. Anybody know what its about?

root@proxmox:~# proxmox-boot-tool init /dev/nvme1n1p2
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
UUID="BCCE-7159" SIZE="536870912" FSTYPE="vfat" PARTTYPE="c12a7328-f81f-11d2-ba4b-00a0c93ec93b" PKNAME="nvme1n1" MOUNTPOINT=""
Mounting '/dev/nvme1n1p2' on '/var/tmp/espmounts/BCCE-7159'.
Installing systemd-boot..
Created "/var/tmp/espmounts/BCCE-7159/EFI/systemd".
Created "/var/tmp/espmounts/BCCE-7159/EFI/BOOT".
Created "/var/tmp/espmounts/BCCE-7159/loader".
Created "/var/tmp/espmounts/BCCE-7159/loader/entries".
Created "/var/tmp/espmounts/BCCE-7159/EFI/Linux".
Copied "/usr/lib/systemd/boot/efi/systemd-bootx64.efi" to "/var/tmp/espmounts/BCCE-7159/EFI/systemd/systemd-bootx64.efi".
Copied "/usr/lib/systemd/boot/efi/systemd-bootx64.efi" to "/var/tmp/espmounts/BCCE-7159/EFI/BOOT/BOOTX64.EFI".
Random seed file /var/tmp/espmounts/BCCE-7159/loader/random-seed successfully written (512 bytes).
Unable to write 'LoaderSystemToken' EFI variable (firmware problem?), ignoring: Invalid argument
Failed to create EFI Boot variable entry: Invalid argument
 
gabrioth said:
I managed to complete the whole operation in a lab environment :)
When running it on my production machine I get an error message from proxmox-boot-tool. Anybody know what its about?

root@proxmox:~# proxmox-boot-tool init /dev/nvme1n1p2
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
UUID="BCCE-7159" SIZE="536870912" FSTYPE="vfat" PARTTYPE="c12a7328-f81f-11d2-ba4b-00a0c93ec93b" PKNAME="nvme1n1" MOUNTPOINT=""
Mounting '/dev/nvme1n1p2' on '/var/tmp/espmounts/BCCE-7159'.
Installing systemd-boot..
Created "/var/tmp/espmounts/BCCE-7159/EFI/systemd".
Created "/var/tmp/espmounts/BCCE-7159/EFI/BOOT".
Created "/var/tmp/espmounts/BCCE-7159/loader".
Created "/var/tmp/espmounts/BCCE-7159/loader/entries".
Created "/var/tmp/espmounts/BCCE-7159/EFI/Linux".
Copied "/usr/lib/systemd/boot/efi/systemd-bootx64.efi" to "/var/tmp/espmounts/BCCE-7159/EFI/systemd/systemd-bootx64.efi".
Copied "/usr/lib/systemd/boot/efi/systemd-bootx64.efi" to "/var/tmp/espmounts/BCCE-7159/EFI/BOOT/BOOTX64.EFI".
Random seed file /var/tmp/espmounts/BCCE-7159/loader/random-seed successfully written (512 bytes).
Unable to write 'LoaderSystemToken' EFI variable (firmware problem?), ignoring: Invalid argument
Failed to create EFI Boot variable entry: Invalid argument
Click to expand...
My MSI X470 motherboard had an "enroll" setting under the secure boot menu (actually "windows boot settings"), in which I "enrolled" all PCI storage devices (there were only PCI storage devices listed). After that I booted and it worked.
 
  • Like
Reactions: aaron
You must log in or register to reply here.
Top Bottom