[SOLVED] Proxmox zfs raid1 installlation/configuration

[V3nom]

I recently had one of my nvme disks fail on me in my Proxmox server. I installed Proxmox on a default zfs raid1 installation. I was able to replace the disk and add it back to the pool but while doing that I noticed that the mirror only consists of partition3 which looks like this.

root@pve:~# zpool status
  pool: rpool
 state: ONLINE
  scan: resilvered 15.7M in 00:00:00 with 0 errors on Mon Mar  3 16:23:07 2025
config:

        NAME                                 STATE     READ WRITE CKSUM
        rpool                                ONLINE       0     0     0
          mirror-0                           ONLINE       0     0     0
            nvme-eui.0025384141400676-part3  ONLINE       0     0     0
            nvme-eui.0025384151b3f8db-part3  ONLINE       0     0     0

I installed Proxmox in a vm to be sure that is correct and it installed it the same way because I would think you would want to mirror the entire disk for if one of the disks breaks, that you can still boot. Because right now if my disk breaks where the boot partitions are on I won't be able to boot Proxmox.

I did find "proxmox-boot-tool format/init" but when using that it requires that the efi partition is at least 256M while the defaul installation makes it 1007Kb. I don't see how a reinstall would fix my problem because when choosing ZFS raid1 it only setups the mirror for partition3.

This is my partition setup for my first disk, so where the boot partitions and the boot files on those partitions are correct.

Number  Start (sector)    End (sector)  Size       Code  Name
   1              34            2047   1007.0 KiB  EF02
   2            2048         2099199   1024.0 MiB  EF00
   3         2099200      7814037134   3.6 TiB     BF01

This is my secondary disk where I manually created the partitions after replacing it, however for partition 1 and 2 it doesn't contain the needed boot files to boot.

Number  Start (sector)    End (sector)  Size       Code  Name
   1              34            2047   1007.0 KiB  EF02
   2            2048         2099199   1024.0 MiB  EF00
   3         2099200      7814037134   3.6 TiB     BF01

nvme1n1     259:0    0  3.6T  0 disk
├─nvme1n1p1 259:1    0 1007K  0 part
├─nvme1n1p2 259:2    0    1G  0 part
└─nvme1n1p3 259:3    0  3.6T  0 part
nvme0n1     259:4    0  3.6T  0 disk
├─nvme0n1p1 259:5    0 1007K  0 part
├─nvme0n1p2 259:6    0    1G  0 part
└─nvme0n1p3 259:7    0  3.6T  0 part

I would like to have a setup for if one of my os disks breaks that I can still boot Proxmox. With my current zpool rpool setup. How can I configure it to do that?

 

[leesteken]

The second partition (with EF00) is the ESP (with is nowadays 1GB) that you should format and init with proxmox-boot-tool. The very small first partition (with EF02) is for GRUB and is initialized by grub-install (and unused when your Proxmox uses systemd-boot: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot ).

 

[V3nom]

The second partition (with EF00) is the ESP (with is nowadays 1GB) that you should format and init with proxmox-boot-tool. The very small first partition (with EF02) is for GRUB and is initialized by grub-install (and unused when your Proxmox uses systemd-boot: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot ).

Thanks for your reply. It looks with the default install Grub is the default bootloader because I didn't pick when during the time when I installed this Proxmox install. What's the best way to go here, manually redo the bootloader setup and switch from Grub to systemd-boot or is there a better option without switching bootloaders?

 

[leesteken]

It looks with the default install Grub is the default bootloader because I didn't pick when during the time when I installed this Proxmox install. What's the best way to go here, manually redo the bootloader setup and switch from Grub to systemd-boot or is there a better option without switching bootloaders?

You don't need to switch bootloaders. Proxmox picked the one for your system when installing.

I was just responding to your issues with a too small ESP, which I think is based on a misunderstanding. The second partition (with EF00) is the ESP (which is nowadays 1GB) that you should format and init with proxmox-boot-tool.

 

[V3nom]

You don't need to switch bootloaders. Proxmox picked the one for your system when installing.

That's what I thought.

I did check the status of the boot tool.

root@pve:~# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with uefi
WARN: /dev/disk/by-uuid/EFE9-DC66 does not exist - clean '/etc/kernel/proxmox-boot-uuids'! - skipping
EFEA-A4D6 is configured with: uefi (versions: 6.8.12-6-pve, 6.8.12-8-pve)

Does this mean I am directly booted with uefi boot or using Grub, it's quite confusing because I do have the directory /boot/grub on my boot partition?

 

[leesteken]

Does this mean I am directly booted with uefi boot or using Grub, it's quite confusing because I do have the directory /boot/grub on my boot partition?

UEFI and GRUB are not mutual exclusive. Your system appears to be booting in UEFI mode with GRUB. Everything is fine, don't worry too much.

I did check the status of the boot tool.

root@pve:~# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with uefi
WARN: /dev/disk/by-uuid/EFE9-DC66 does not exist - clean '/etc/kernel/proxmox-boot-uuids'! - skipping
EFEA-A4D6 is configured with: uefi (versions: 6.8.12-6-pve, 6.8.12-8-pve)

One of the original/old ESP is missing and can be removed by running proxmox-boot-tool clean.
Format and initialize the new ESP as you intended with proxmox-boot-tool format /dev/nvme1n1p2 (or nvme0n1p2 since I don't know which drive is the new one) followed by proxmox-boot-tool init for the same device. Or do both drives if this is too complicated and run proxmox-boot-tool clean afterwards.

 

[V3nom]

One of the original/old ESP is missing and can be removed by running proxmox-boot-tool clean.

Useful tool! I had first manually removed it from the /etc/kernel/proxmox-boot-uuids file, that seems the better way of doing it.

Format and initialize the new ESP as you intended with proxmox-boot-tool format /dev/nvme1n1p2 (or nvme0n1p2 since I don't know which drive is the new one) followed by proxmox-boot-tool init for the same device. Or do both drives if this is too complicated and run proxmox-boot-tool clean afterwards.

I ran it on both to be sure and then ran a clean afterwards. Then rebooted my Proxmox system afterwards and my system booted afterwards.

So just in short recap for me that I understand why I did this is. It's normal that that during the Proxmox installation only partition3 is setup as a zfs raid1 mirror. To be able to boot from both disks from the boot partitions which are not mirrored like partition3 you run the proxmox-boot-tool on the secondary(or both disks on partition2) because that's what's used to boot since under /boot the uefi boot files are located?

Is that my correct understanding of it?

@leesteken Thanks so much for your help, you saved me a lot of frustration and stress with this!

 

[V3nom]

One last question though. This is my zpool.

root@pve:~# zpool status
  pool: rpool
 state: ONLINE
  scan: resilvered 15.7M in 00:00:00 with 0 errors on Mon Mar  3 16:23:07 2025
config:

        NAME                                 STATE     READ WRITE CKSUM
        rpool                                ONLINE       0     0     0
          mirror-0                           ONLINE       0     0     0
            nvme-eui.0025384141400676-part3  ONLINE       0     0     0
            nvme-eui.0025384151b3f8db-part3  ONLINE       0     0     0

Why does it mention mirror-0, I remember having used zfs raid1 during the installation, or does mirror-0 mean zfs raid1 since it does mention that it's mirror?

 

[leesteken]

It's normal that that during the Proxmox installation only partition3 is setup as a zfs raid1 mirror.

Yes because ZFS supports RAID1/mirror. Boot partitions don't support such things.

To be able to boot from both disks from the boot partitions which are not mirrored like partition3 you run the proxmox-boot-tool on the secondary(or both disks on partition2) because that's what's used to boot since under /boot the uefi boot files are located?

That's what proxmox-boot-tool was designed to do. Keep all boot partitions in sync (when there are updates or new kernel versions) so that you can boot from any and all drives (that you configure for use by proxmox-boot-tool).
If one drive fails, you sometimes have to select another drive to boot from in the motherboard BIOS.
There are many threads on this forum about (replacing drvies in) bootable ZFS pools and GRUB and systemd-boot and other bootloader issues and misunderstandings.

Thanks so much for your help, you saved me a lot of frustration and stress with this!

You simply misunderstood the purposes of the first and second partition. I fear that my link to the manual (which explains many things but rather technically) caused unnecessary confusion, sorry for that.

root@pve:~# zpool status
  pool: rpool
 state: ONLINE
  scan: resilvered 15.7M in 00:00:00 with 0 errors on Mon Mar  3 16:23:07 2025
config:

        NAME                                 STATE     READ WRITE CKSUM
        rpool                                ONLINE       0     0     0
          mirror-0                           ONLINE       0     0     0
            nvme-eui.0025384141400676-part3  ONLINE       0     0     0
            nvme-eui.0025384151b3f8db-part3  ONLINE       0     0     0

Why does it mention mirror-0, I remember having used zfs raid1 during the installation, or does mirror-0 mean zfs raid1 since it does mention that it's mirror?

This is simply how ZFS works. Your pool consist of one vdev named mirror-0 (because it is mirrored and it starts counting from zero). ZFS documentation (on the internet) can explain all possible types of vdev configurations and combinations.

 

[V3nom]

Yes because ZFS supports RAID1/mirror. Boot partitions don't support such things.

This is simply how ZFS works. Your pool consist of one vdev named mirror-0 (because it is mirrored and it starts counting from zero). ZFS documentation (on the internet) can explain all possible types of vdev configurations and combinations.

I don't know too much about zfs but I've been using it because I've heard it's good for things such as storage pools. That explains why only partition3 was setup as a mirror, glad to have learned something useful about zfs as well with this.

That's what proxmox-boot-tool was designed to do. Keep all boot partitions in sync (when there are updates or new kernel versions) so that you can boot from any and all drives (that you configure for use by proxmox-boot-tool).
If one drive fails, you sometimes have to select another drive to boot from in the motherboard BIOS.
There are many threads on this forum about (replacing drvies in) bootable ZFS pools and GRUB and systemd-boot and other bootloader issues and misunderstandings.

Glad to hear that I'm not the only with questions about this.

You simply misunderstood the purposes of the first and second partition. I fear that my link to the manual (which explains many things but rather technically) caused unnecessary confusion, sorry for that.

It's because on most other Linux systems that I use the first partition is the efi partition and then /boot the second partition, I did come across that boot tool and tried it but then got the error that the partition is too small. It wasn't until you mentioned it about partition 2 being the uefi partition that I went "Ooh, I get it now!" and everything I ran into and what you shared made sense after that.


You helped me learn something new and useful about Proxmox that will help me for later disk failures as well! Thanks again so much for your help and time!