[SOLVED] Transfer root filesystem to ZFS mirror or mdraid mirror?

[Keeper of the Keys]

Hi,
I have a Proxmox 7.4 system running from a single SSD at the moment and want to create a mirrored device and transfer my root filesystem there.

If I just use mdadm to create this I'm pretty sure everything should work as long as I update the UUID in fstab and grub.cfg (1).

With ZFS based on the proxmox documentation it seems to me that Proxmox supports having the root filesystem on ZFS out of the box (though due to my installation method that was not an option), my understanding is that ZFS makes better data integrity promises than other systems which I find attractive

Are there any pros/cons I'm missing here? I really do not wish to reinstall this system.

1. legacy grub since it's not in EFI mode, possibly because I needed to run the install over serial console since I no longer own a VGA screen but not sure, either way I really don't want to reinstall over serial so switching uefi mode on at this stage is not really an option

 

[LnxBil]

To get your current install method, just look where your / is. Please post in CODE tags the output of

df -PHT /

In general, converting from anything but ZFS to mdadm is very easy. First install mdadm

 

[Keeper of the Keys]

Current install is just a single SSD with default Debian partitioning (ext4 on the full disk with a swap partition).

/Edit -
I have 2 additional SSDs and am replacing the original SSD with them, so I don't need anything to happen to the original SSD other than moving the data from one to the other.

 

[Keeper of the Keys]

I'm going to try the following sequence of actions:
1. Setup ZFS mirror on the two new SSDs
2. rsync the root filesystem to ZFS
3. Modify fstab on ZFS
4. use `proxmox-boot-tool` to install grub on the ZFS mirrored disks
5. Shutdown all running VMs
6. run rsync again with --exclude on fstab and grub.cfg
7. Shutdown system and unplug old SSD
8. Boot and pray

Did I miss anything?

 

[Keeper of the Keys]

Are there any options/arguments set in a proxmox ZFS install in fstab?

 

[ubu]

1. ZFS is great
2. Please do some Test with it before you switch, it is a bit different
3. I do not think your planned course of action will work (but good luck

4. Proxmox ISO -> ZFS Mirror -> select the 2 Disks -> install -> Restore your VMs from backup

The default looks like this:

The zpool is called zp_pve and consists of 2 mirrored disks

root@pvehe13:~# zpool status pool: zp_pve state: ONLINE scan: scrub repaired 0B in 00:03:24 with 0 errors on Sun May 14 00:27:25 2023 config: NAME STATE READ WRITE CKSUM zp_pve ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 nvme0n1p3 ONLINE 0 0 0 nvme1n1p3 ONLINE 0 0 0

There are the following subvolumes:

ZFS Volume (by default) are not mounted via fstzab but the mountpoint is set as an attribute (see below)

root@pvehe13:~# zfs list NAME USED AVAIL REFER MOUNTPOINT zp_pve 434G 2.95T 240K /zp_pve zp_pve/ROOT 4.24G 2.95T 96K /zp_pve/ROOT zp_pve/ROOT/pve-1 4.24G 2.95T 4.24G / zp_pve/data 96K 2.95T 96K /zp_pve/data zp_pve/subvol-1003043-disk-10 3.04G 6.96G 3.04G /zp_pve/subvol-1003043-disk-10 zp_pve/subvol-1003043-disk-5 5.54G 34.5G 5.54G /zp_pve/subvol-1003043-disk-5 zp_pve/subvol-1003043-disk-6 9.19G 291G 9.19G /zp_pve/subvol-1003043-disk-6

root@pvehe13:~# zfs get mountpoint zp_pve/ROOT/pve-1 NAME PROPERTY VALUE SOURCE zp_pve/ROOT/pve-1 mountpoint / local

 

[ubu]

You can install proxmox inside your proxmox as a kvm with 2 virt disks to play with zfs

 

[Keeper of the Keys]

Sadly I can't use the proxmox ISO for installation, the machine I am working with (Dell T130) only has a VGA output and I don't have a VGA compatible screen at home anymore.
Dell also put ipmi (iDrac) html console behind some license that even if I was willing to buy it is no longer being sold so I can only use Serial over LAN with ipmitool.

Since the Proxmox installer is GUI only my only recourse is Debian text install and convert Debian to Proxmox.

 

[Dunuin]

There are VGA to HDMI adapter cables for less than 10 bucks. Or even cheaper you get a used old LCD display with VGA input for free, people want to get rid of.

 

[LnxBil]

Dell also put ipmi (iDrac) html console behind some license that even if I was willing to buy it is no longer being sold so I can only use Serial over LAN with ipmitool.

You'll find them on ebay ... or somewhere else.

I also suggest to install inside of a PVE guest itself with two disks, boot a live linux over serial on the target machine and dd the two virtual disks over to the actual disks. I've done this multiple times and it works.

 

[Keeper of the Keys]

You'll find them on ebay ... or somewhere else.

LOL I never even thought to look for it there because I figured only Dell could issue these keys, though I must say I also find the whole model they (and HPE) have around their IPMI devices offensive, here I got the machine for free from a friend but at a previous employer we bought some expensive equipment that may even have included the license but they didn't add it in the factory and I found myself wasting so much time as a result while the competition sees it as basic functionality for such boards.

I also suggest to install inside of a PVE guest itself with two disks, boot a live linux over serial on the target machine and dd the two virtual disks over to the actual disks. I've done this multiple times and it works.

If I do that I could actually just pass through the new SSDs to the VM finish the install and then reboot the machine from them no?

 

[ubu]

You can do the install on another pc and then transfer the disks, probably you need to change the name of your network interface, but if you configure serial console on the install pc that should not be a problem.
Or throw a graphics card in your server for the install an remove it afterwards

 

[LnxBil]

If I do that I could actually just pass through the new SSDs to the VM finish the install and then reboot the machine from them no?

Yes, that'll work too, yet where is the fun in that?

 

[Keeper of the Keys]

Yes, that'll work too, yet where is the fun in that?

Hehe total inception this....

I'm finding all kinds of interesting differences between my old install (converted Debian) to the "native" install, for instance no /etc/pve/storage.cfg

At the moment I'm trying to get the new install to recognize the old one to import the VMs so far no luck :/ I added the directory /mnt/sdc1/var/lib/vz in storage.cfg but it doesn't show in the WUI.

 

[Keeper of the Keys]

Ok turns out I should have added it from the WUI and not by editing storage.cfg

 

[Keeper of the Keys]

I just realized I missed the memo that /etc/pve comes from config.db, I have attempted to copy config.db from the old install that did not give me the desired outcome, then I tried "unifying" the old and new config.db (ie. copy just those rows from the old that contain info I want and fixing their parent inode), this only did the trick partially.

How can I troubleshoot why/if a row is accepted/rejected?

 

[Keeper of the Keys]

Just wanted to check back in - I did what was suggested, linked the disks to a vm and installed proxmox.

After that I still had to modify the grub command line to enable serial output and of course fix the network config.

As for the issue I had with /etc/pve it turns out that in my zeal to copy the sqlite dbs in a non-destructive fashion I had copied the wrong version (the version of the clean machine), after resolving that I was able to boot from the new disk and migrate the vms from their old storage to the new storage.