ZFS mirror various disk size

[rafnizp]

Hi Guys,

Is is possible to install Proxmox on mirrored ZFS is disk size is different?
Any guid wold be much appreciated

Thanks,

 

[LnxBil]

Is is possible to install Proxmox on mirrored ZFS is disk size is different?

Not in the installer:



You could convert a stripped disk setup (also single) to a mirror and do everything manually.

 

[rafnizp]

I'm aware of above error, however it would be nice if someone can share manual setup guide as I believe this is not an easy step

 

[aaron]

Oneway would be to install it in ZFS RAID 0 mode on the smaller disk first. Then once the system is up and running, follow the "Changing a failed bootable device" procedure in the Proxmox VE Admin guide

The only difference! Instead of zpool replace, you use the zpool attach rpool <already installed disk> <new disk>.

You can get the list of the currently used disk with zpool status. When you attach the disk, choose it via the /dev/disk/by-id path. There it will also be exposed via the manufacturer, model and serial number. Having them in the ZFS status output can speed up the process of figuring out which disk has an issue a lot.

In the end you should have a mirrored rpool and the bootloader on both disks, so the system will boot if one fails.

 

[rafnizp]

What about if:
first disk is 100GB
second disk is 55GB
For proxmox installation I would like to use only 40GB from each disk and this to be set as mirrored zpool
Rest of disk space from fist drive I want to use for logs and from second for swap

 

[aaron]

For proxmox installation I would like to use only 40GB from each disk and this to be set as mirrored zpool

Then you can set the "hdsize" parameter during the installation. See https://pve.proxmox.com/pve-docs/pve-admin-guide.html#advanced_zfs_options

Rest of disk space from fist drive I want to use for logs and from second for swap

Why an extra partition for logs?
Using non-redundant swap does not sound like a good idea. What happens if that disk fails?

Also, may I ask what kind of disks (model, manufacturer, ...) you plan to use?

 

[gogito]

Then you can set the "hdsize" parameter during the installation. See https://pve.proxmox.com/pve-docs/pve-admin-guide.html#advanced_zfs_options


Why an extra partition for logs?
Using non-redundant swap does not sound like a good idea. What happens if that disk fails?

Also, may I ask what kind of disks (model, manufacturer, ...) you plan to use?

Hi Aaron,

I'm planning to do the same:
1x2TB NVME
1x4TB NVME

Mirroring 100GB from each and leave the rest unpartition so I can use them for other things.

For this, based on your instruction, I would have to:

1. Choose the 2TB for ZFS RAID0 during installation
2. Use hdsize = 100GB
3. Partition the 4TB to 100GB and the rest untouch
4. Follow the Changing a failed bootable device procedure in the Proxmox VE Admin guide and attach the 100GB partition from the 4TB NVME to the existing 100GB Raid0

Is this the correct procedure?

And just to confirm, I can use the rest of the storage on both NVME to build an LVM VG or any other usage right?

 

[Dunuin]

3. Partition the 4TB to 100GB and the rest untouch

Doesn't matter as you are wiping the partition table in step 4 anyway to make the second disk bootable. You would have to extend the partitions after doing step 4.

And just to confirm, I can use the rest of the storage on both NVME to build an LVM VG or any other usage right?

Yes.

 

[sumguy]

Oneway would be to install it in ZFS RAID 0 mode on the smaller disk first. Then once the system is up and running, follow the "Changing a failed bootable device" procedure in the Proxmox VE Admin guide

The only difference! Instead of zpool replace, you use the zpool attach rpool <already installed disk> <new disk>.

You can get the list of the currently used disk with zpool status. When you attach the disk, choose it via the /dev/disk/by-id path. There it will also be exposed via the manufacturer, model and serial number. Having them in the ZFS status output can speed up the process of figuring out which disk has an issue a lot.

In the end you should have a mirrored rpool and the bootloader on both disks, so the system will boot if one fails.

I've been reading and reading and can't get this to work.

When I do zpool status I get the following:

pool: rpool
 state: ONLINE
config:

        NAME                                               STATE     READ WRITE CKSUM
        rpool                                              ONLINE       0     0     0
          nvme-eui.000000000000000100a0752446a4a08c-part3  ONLINE       0     0     0

errors: No known data errors

I don't understand why that isn't a physical disk or why it says part3 at the end. I can't figure out how to properly choose and attach a disk using the zpool attach rpool <current disk> <new disk to add>

The manuals don't explain this stuff or have the correct syntax shown. It is very confusing for someone who is not familiar with the commands/usage already. It would be nice if the proxmox manual, for example, said how to list devices and how to choose them to add them using zpool.

I did cd /dev/disk/by-id and then ls

and got:

nvme-eui.000000000000000100a0752446a4a08c
nvme-eui.000000000000000100a0752446a4a08c-part1
nvme-eui.000000000000000100a0752446a4a08c-part2
nvme-eui.000000000000000100a0752446a4a08c-part3
nvme-INTEL_SSDPE2KX010T8_BTLJ908508XJ1P0FGN
nvme-INTEL_SSDPE2KX010T8_BTLJ908508XJ1P0FGN_1
nvme-INTEL_SSDPE2KX010T8_BTLJ908508XJ1P0FGN_1-part1
nvme-INTEL_SSDPE2KX010T8_BTLJ908508XJ1P0FGN_1-part9
nvme-INTEL_SSDPE2KX010T8_BTLJ908508XJ1P0FGN-part1
nvme-INTEL_SSDPE2KX010T8_BTLJ908508XJ1P0FGN-part9
nvme-Micron_7450_MTFDKCB800TFS_240446A4A08C
nvme-Micron_7450_MTFDKCB800TFS_240446A4A08C_1
nvme-Micron_7450_MTFDKCB800TFS_240446A4A08C_1-part1
nvme-Micron_7450_MTFDKCB800TFS_240446A4A08C_1-part2
nvme-Micron_7450_MTFDKCB800TFS_240446A4A08C_1-part3
nvme-Micron_7450_MTFDKCB800TFS_240446A4A08C-part1
nvme-Micron_7450_MTFDKCB800TFS_240446A4A08C-part2
nvme-Micron_7450_MTFDKCB800TFS_240446A4A08C-part3
nvme-nvme.8086-42544c4a393038353038584a31503046474e-494e54454c205353445045324b583031305438-00000001
nvme-nvme.8086-42544c4a393038353038584a31503046474e-494e54454c205353445045324b583031305438-00000001-part1
nvme-nvme.8086-42544c4a393038353038584a31503046474e-494e54454c205353445045324b583031305438-00000001-part9
usb-Generic_Flash_Disk_A38EC09C-0:0
usb-Generic_Flash_Disk_A38EC09C-0:0-part1

I only have 2 nvme disks installed. They are installed behind a pcie nvme switch. Then of course a usb flash disk is inserted at the moment.

Hopefully the answer to this will help others who have drives that are not the same size and trying to zfs mirror them. Thank you.

 

[Dunuin]

You can't add a disk without formating it first and cloning the bootloader. And thats partition 3 of your physical disk because partition 1+2 are needed for the bootloader. See chapter "Replacing a failed bootable device" but instead of the "zpool replace..." you use a "zpool attach" for a raid1: https://pve.proxmox.com/wiki/ZFS_on_Linux#_zfs_administration

If you really want a raid0 instead of a raid1 (not recommended!) you could skip the partitioning/bootloader part and directly add the whole disk with the "zpool add" command.

 

[sumguy]

Thank you for that information. There wasn't a mention of needing to format it and clone the bootloader in the previous post to which I was replying.

I am trying to get a mirror of the system bootdisk going. I have 2 disks but they are not equal size, so the installer was whiny about it. It would be really nice if the installer could replace the "you can't do this" message with "you can do this, but you will only get the size of the smaller disk." This would be a huge improvement and benefit a lot of people I am sure.

Could someone walk us through how to move forward getting a zfs mirror going starting from the point of 1 disk that proxmox was installed onto in zfs "raid 0" mode? I don't know how to clone the bootloader and otherwise prepare it to use the zpool attach command. I'm not 100% sure on what syntax to use for the attach command when I get to that point either. I think running through it would benefit a lot of people though, if someone is willing to do so.

 

[Dunuin]

As I already said, its the same as replacing a failed disk of a mirror, you just don't replace a disk but add a new one. See the article linked above. So:

Changing a failed bootable device​

Depending on how Proxmox VE was installed it is either using systemd-boot or GRUB through proxmox-boot-tool [2] or plain GRUB as bootloader (see Host Bootloader). You can check by running:
# proxmox-boot-tool status
The first steps of copying the partition table, reissuing GUIDs and replacing the ZFS partition are the same. To make the system bootable from the new disk, different steps are needed which depend on the bootloader in use.
# sgdisk <healthy bootable device> -R <new device>
# sgdisk -G <new device>
# zpool attach -f <pool> <part3 of existing disk > <part3 of new disk>

Use the zpool status -v command to monitor how far the resilvering process of the new disk has progressed.

With proxmox-boot-tool:
# proxmox-boot-tool format <new disk's ESP (part2)>
# proxmox-boot-tool init <new disk's ESP (part2)> [grub]

ESP stands for EFI System Partition, which is setup as partition #2 on bootable disks setup by the Proxmox VE installer since version 5.4. For details, see Setting up a new partition for use as synced ESP.

Make sure to pass grub as mode to proxmox-boot-tool init if proxmox-boot-tool status indicates your current disks are using GRUB, especially if Secure Boot is enabled!

With plain GRUB:
# grub-install <new disk>

Plain GRUB is only used on systems installed with Proxmox VE 6.3 or earlier, which have not been manually migrated to using proxmox-boot-tool yet.

 

[sumguy]

:~# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with uefi
77EB-8D67 is configured with: uefi (versions: 6.8.4-2-pve)

I'm assuming this is systemd-boot?

Do you know why 4 nvme devices are listed when I only have 2 physical disks installed? I was surprised that rpool says it's on one of these 2 devices and not the physical disk that it's actually on (the Micron).

Would I then do:

sgdisk nvme-eui.000000000000000100a0752446a4a08c -R nvme-INTEL_SSDPE2KX010T8_BTLJ908508XJ1P0FGN

or would it be

sgdisk nvme0n1 -R nvme1n1

So I would do one of the commands above? And then sgdisk -G <samenewdevice> ?
I don't see a part3 on the Intel disk yet. Will that be created with sgdisk?

I'm willing to try this, just want to make sure I'm doing it right and won't mess anything up on a working system right now. I really appreciate your help.

 

[Dunuin]

I'm assuming this is systemd-boot?

Correct

Do you know why 4 nvme devices are listed when I only have 2 physical disks installed?

Use "ls -la /dev/disk/by-id" instead of your "cd /dev/disk/by-id; ls". An SSD could have multiple IDs pointing to the same SSD.

I don't see a part3 on the Intel disk yet. Will that be created with sgdisk?

Yes. It will clone the partition table and both disks will have the same partitions. So you might want to install PVE to the smaller one so cloning won'T fail because of missing space.

 

[sumguy]

I tried sgdisk -R but am getting the following:

root@x:~# sgdisk nvme-Micron_7450_MTFDKCB800TFS_240446A4A08C -R nvme-INTEL_SSDPE2KX010T8_BTLJ908508XJ1P0FGN
Problem opening nvme-Micron_7450_MTFDKCB800TFS_240446A4A08C for reading! Error is 2.
The specified file does not exist!
root@x:~# sgdisk nvme0n0 -R nvme1n1
Problem opening nvme0n0 for reading! Error is 2.
The specified file does not exist!

 

[Dunuin]

Point it to the actual device: sgdisk /dev/disk/by-id/nvme-Micron_7450_MTFDKCB800TFS_240446A4A08C ...

 

[sumguy]

Point it to the actual device: sgdisk /dev/disk/by-id/nvme-Micron_7450_MTFDKCB800TFS_240446A4A08C ...

# sgdisk <healthy bootable device> -R <new device>
# sgdisk -G <new device>

Thank you so much! These reported "The operation has completed successfully"

On the zpool attach I'm getting "no such device in pool"

root@rx:~# sgdisk /dev/disk/by-id/nvme-Micron_7450_MTFDKCB800TFS_240446A4A08C -R /dev/disk/by-id/nvme-INTEL_SSDPE2KX010T8_BTLJ908508XJ1P0FGN
The operation has completed successfully.
root@x:~# sgdisk -G /dev/disk/by-id/nvme-INTEL_SSDPE2KX010T8_BTLJ908508XJ1P0FGN
The operation has completed successfully.
root@x:~# zpool attach -f rpool /dev/disk/by-id/nvme-Micron_7450_MTFDKCB800TFS_240446A4A08C-part3 /dev/disk/by-id/nvme-INTEL_SSDPE2KX010T8_BTLJ908508XJ1P0FGN-part3
cannot attach /dev/disk/by-id/nvme-INTEL_SSDPE2KX010T8_BTLJ908508XJ1P0FGN-part3 to /dev/disk/by-id/nvme-Micron_7450_MTFDKCB800TFS_240446A4A08C-part3: no such device in pool
root@x:~# zpool attach -f rpool nvme-Micron_7450_MTFDKCB800TFS_240446A4A08C-part3 nvme-INTEL_SSDPE2KX010T8_BTLJ908508XJ1P0FGN-part3
cannot attach nvme-INTEL_SSDPE2KX010T8_BTLJ908508XJ1P0FGN-part3 to nvme-Micron_7450_MTFDKCB800TFS_240446A4A08C-part3: no such device in pool
root@x:~# zpool attach -f rpool nvme0n1p3 nvme1n1p3
cannot attach nvme1n1p3 to nvme0n1p3: no such device in pool

 

[Dunuin]

Your pool only knows "nvme-eui.000000000000000100a0752446a4a08c-part3". So try something like: "zpool attach -f rpool nvme-eui.000000000000000100a0752446a4a08c-part3 /dev/disk/by-id/nvme-INTEL_SSDPE2KX010T8_BTLJ908508XJ1P0FGN-part3"

 

[sumguy]

That seems to have done it! Disks > ZFS > pool detail says it is resilvering

Sorry for being slow - the lack of documentation on syntax and referencing disks was difficult. I bet this will help a lot of people out. Thank you for taking the time to help explain and troubleshoot it with me.