Proxmox ZFS UEFI Boot failure

[Daniel Luyckx]

Hello all,
Some weeks ago our Server didnt boot automatically anymore. We scheduled a reinstallation for next week, due the system was installed on UEFI what is not best practise.

Since that Problem occurs, we were always able to boot the System via Proxmox Rescue Boot.

Today the Server was restarted, but the Rescue Boot is no more working. I will attach some screenshots.

I am going now to try with rescueCD to restore GRUB and boot from BIOS then. Anyone has an idea of how that Problem should be fixed ?

Greetings

 

[Daniel Luyckx]

Alright i got a little bit forward..

 

[Stoiko Ivanov]

How is this system set up?:
* what kind of hardware? (especially the HBA/disk-controller)
* which version of PVE was first installed there (assuming you used the PVE-installer for ZFS on root)
* which version of PVE is currently running there (`pveversion -v`)
seems like a problem with a feature mismatch between grub's ZFS implementation and the one in the kernel - or like a problem with the driver for the disk-controller in GRUB.

Starting with PVE 6.0 there is support for booting from ZFS via systemd-boot - see https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#_installation_as_root_file_system and https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysboot
This gives a more stable experience (but might need a fresh setup)

I hope this helps!

 

[Daniel Luyckx]

Hello!
I was able to boot the System one time with this Topic:
https://forum.proxmox.com/threads/grub2-recovery-on-zfs-proxmox-ve-3-4.21306/

I will not repair the UEFI boot due UEFI is not recommended for ZFS and Proxmox. I transfered all VMs to another Server and will reinstall the Problem-Server tonight or tomorrow evening.

How is this system set up?:
* what kind of hardware? (especially the HBA/disk-controller)
* which version of PVE was first installed there (assuming you used the PVE-installer for ZFS on root)
* which version of PVE is currently running there (`pveversion -v`)
seems like a problem with a feature mismatch between grub's ZFS implementation and the one in the kernel - or like a problem with the driver for the disk-controller in GRUB.

Starting with PVE 6.0 there is support for booting from ZFS via systemd-boot - see https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#_installation_as_root_file_system and https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysboot
This gives a more stable experience (but might need a fresh setup)

I hope this helps!

There is a HBA controller, i dont know the Manufacturer right now, sorry, i dont have inventory data here. It should be a LSI controller...
Yes, i used the PVE Installer with the ZFS Mirror Option. I installed directly PVE 6 (Which is pretty Nice by the way, good Job there!)

Output of Pveversion:

pveversion -v
proxmox-ve: 6.0-2 (running kernel: 5.0.21-2-pve)
pve-manager: 6.0-7 (running version: 6.0-7/28984024)
pve-kernel-5.0: 6.0-8
pve-kernel-helper: 6.0-8
pve-kernel-5.0.21-2-pve: 5.0.21-6
pve-kernel-5.0.21-1-pve: 5.0.21-2
pve-kernel-5.0.18-1-pve: 5.0.18-3
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.12-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-5
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-9
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-65
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-7
pve-cluster: 6.0-7
pve-container: 3.0-7
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-7
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-3
pve-qemu-kvm: 4.0.0-5
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-7
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.1-pve2

So, im moving definetly to BIOS now
Thanks for the Help very much, very appreciated.

Greetings

 

[Stoiko Ivanov]

I will not repair the UEFI boot due UEFI is not recommended for ZFS and Proxmox.

That used to be the case - but as of PVE-6.0 it is supported (and seems to work for most users)
The initial screenshot you posted looks like it is grub which is actually booting - which should not happen if you setup ZFS on root with PVE 6.0's installer.

you can check if systemd-boot was used by looking at the ESP-partitions (the second partition of both disks in the first mirror)

In any case - glad you managed to get the system booting again!

 

[Daniel Luyckx]

That used to be the case - but as of PVE-6.0 it is supported (and seems to work for most users)
The initial screenshot you posted looks like it is grub which is actually booting - which should not happen if you setup ZFS on root with PVE 6.0's installer.

you can check if systemd-boot was used by looking at the ESP-partitions (the second partition of both disks in the first mirror)

In any case - glad you managed to get the system booting again!

Sorry, the System is already long time Running with BIOS again :=)

But now.. Exept really slow migration and not working NFS / SMB Shares for me.. everything is working fine :=)

im not gonna touch the Cluster again, im gonna setup a testing enviroment to troubleshot the Problems, haha

Thanks for your Help, very Appreciated and always Recommending Proxmox

Greetings

 

[Stoiko Ivanov]

im not gonna touch the Cluster again, im gonna setup a testing enviroment to troubleshot the Problems, haha

Thanks for your Help, very Appreciated and always Recommending Proxmox

Testsetup sounds like a good idea!

Thanks for the feedback - Always nice to hear such things!