Failed to find rpool after kernel upgrade

[techniclab]

Hello!
I'm trying to install Proxmox with root on zfs(raid0, one disk). Unfortunately system update after initial installation which includes kernel update to 4.15.18-7-pve from 4.15.17-1-pve causes next boot to fail with "Message: cannot import 'rpool': no such pool available Error: 1". In emergency shell I do not see any disks when using "ls /dev/disk/by-id" command. Already tried adding rootdelay=60 parameter and there is no effect. Manually switching to kernel 4.15.17-1-pve under grub solves this problem and allows system to boot normally.
Output of "ls /dev/disk/by-id/" after successful boot.

ata-HL-DT-ST_DVDRAM_GTC0N_KZJG68D2147 usb-Virtual_Floppy_serial-0:1

scsi-3600144f0c7a5bc61000058b59d7c001b wwn-0x5001480000000000

scsi-3600144f0c7a5bc61000058b59d7c001b-part1 wwn-0x600144f0c7a5bc61000058b59d7c001b

scsi-3600144f0c7a5bc61000058b59d7c001b-part9 wwn-0x600144f0c7a5bc61000058b59d7c001b-part1

scsi-3600144f0c7a5bc61000059e0c19a0001 wwn-0x600144f0c7a5bc61000058b59d7c001b-part9

scsi-3600144f0c7a5bc61000059e0c19a0001-part1 wwn-0x600144f0c7a5bc61000059e0c19a0001

scsi-3600144f0c7a5bc61000059e0c19a0001-part2 wwn-0x600144f0c7a5bc61000059e0c19a0001-part1

scsi-3600144f0c7a5bc61000059e0c19a0001-part9 wwn-0x600144f0c7a5bc61000059e0c19a0001-part2

usb-Virtual_CDROM_serial-0:0 wwn-0x600144f0c7a5bc61000059e0c19a0001-part9

 

[techniclab]

Additional info: this server is diskless, booting from fc qlogic ql2670.

 

[Stoiko Ivanov]

do you see anything relevant in the `dmesg` or `journalctl` output in the rescue shell?

 

[techniclab]

No errors in dmesg/journalctl. I was able to boot with latest kernel after adding ql2600_fw.bin to /lib/firmware and running 'update-initramfs -u'. Unfortunatly it only works for one boot sequence, after that I still need to boot with old kernel and run 'update-initramfs -u'.

 

[Stoiko Ivanov]

could you please post the output of `pveversion -v` ?

Does it work repeatedly if you run `update-initramfs -k all -u` ?

 

[techniclab]

Output:
proxmox-ve: 5.2-2 (running kernel: 4.15.18-7-pve)

pve-manager: 5.2-10 (running version: 5.2-10/6f892b40)

pve-kernel-4.15: 5.2-10

pve-kernel-4.15.18-7-pve: 4.15.18-27

pve-kernel-4.15.17-1-pve: 4.15.17-9

corosync: 2.4.2-pve5

criu: 2.11.1-1~bpo90

glusterfs-client: 3.8.8-1

ksm-control-daemon: 1.2-2

libjs-extjs: 6.0.1-2

libpve-access-control: 5.0-8

libpve-apiclient-perl: 2.0-5

libpve-common-perl: 5.0-41

libpve-guest-common-perl: 2.0-18

libpve-http-server-perl: 2.0-11

libpve-storage-perl: 5.0-30

libqb0: 1.0.1-1

lvm2: 2.02.168-pve6

lxc-pve: 3.0.2+pve1-3

lxcfs: 3.0.2-2

novnc-pve: 1.0.0-2

proxmox-widget-toolkit: 1.0-20

pve-cluster: 5.0-30

pve-container: 2.0-29

pve-docs: 5.2-9

pve-firewall: 3.0-14

pve-firmware: 2.0-5

pve-ha-manager: 2.0-5

pve-i18n: 1.0-6

pve-libspice-server1: 0.14.1-1

pve-qemu-kvm: 2.12.1-1

pve-xtermjs: 1.0-5

qemu-server: 5.0-38

smartmontools: 6.5+svn4324-1

spiceterm: 3.0-5

vncterm: 1.5-3

zfsutils-linux: 0.7.11-pve1~bpo1

 

[techniclab]

"Does it work repeatedly if you run `update-initramfs -k all -u` ?" - yes, it works.

 

[Stoiko Ivanov]

I did a quick search for the firmware file - and it seems it never got released as open-source (reading the following thread: https://www.spinics.net/lists/linux-scsi/msg85340.html).

However according to the thread the firmware on those HBAs is read from flash - so maybe there is a firmware update for the HBA available?

 

[techniclab]

Yes, I already tried reflashing ql2670. Unfortunately it seems that is is not helping.

 

[techniclab]

Image of failed boot - https://imgur.com/a/vmWtLjK.
For some reason there is no ql2600_fw.bin file but it is present when system is booted normally,

 

[Stoiko Ivanov]

I thought the issue is resolved when running `update-initramfs -k all -u`?

You can check whether the firmware is in the initramfs by `lsinitramfs -l <initrd-file>`
You can add a hook to unconditionally add the file to the initramfs by adding a hook - see https://unix.stackexchange.com/questions/92853/adding-a-file-to-etc-of-initramfs-in-debian-wheezy (the instructions are quite dated, but should still work)

Hope that helps

 

[techniclab]

Strangely, `update-initramfs -k all -u` is not fixing problem anymore, Added firmware file to initramfs using hooks, still no luck, not able to boot system with new kernel.