VMs unable to boot from ZFS after upgrade to PVE 7

[luphi]

Hello all,

I just updated from 6.4 to 7. Everything went smooth, but....

I have an issue booting VMs (CTs are doing fine) as long as any of the assigned disks is on ZFS.
It doesn't matter whether the VM tries to boot from CD or the disk. as soon as there is a disk configured based on ZFS, the VM hangs at this point:



When I move the disk to qcow2, the VM boots fine.

I really appreciate, if someone can put me in the right direction for further troubleshooting.

Cheers,
luphi

 

[luphi]

possible root cause: the system is still running kernel 5.4
next issue: 5.11 panics immediately

Cheers,
luphi

 

[dcsapak]

next issue: 5.11 panics immediately

what is the exact error ?

what does the log say (journal/syslog/dmesg)

 

[luphi]

I think, it's related to my ugly disk setup:

root@pve:/var/lib/vz# fdisk -l /dev/sdg
Disk /dev/sdg: 1.86 TiB, 2048408248320 bytes, 4000797360 sectors
Disk model: Samsung SSD 860
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: A3BF44C8-0F84-4E0F-AAA9-29C9CEB78879

Device          Start        End    Sectors  Size Type
/dev/sdg1          34       2047       2014 1007K BIOS boot
/dev/sdg2        2048    1050623    1048576  512M EFI System
/dev/sdg3     1050624   41943040   40892417 19.5G Solaris /usr & Apple ZFS
/dev/sdg4    41943048  943718400  901775353  430G Solaris /usr & Apple ZFS
/dev/sdg5   943720448 3879733248 2936012801  1.4T Solaris /usr & Apple ZFS
/dev/sdg6  3879735296 3946844160   67108865   32G Linux swap
root@pve:/var/lib/vz#
root@pve:/var/lib/vz# fdisk -l /dev/sdf
Disk /dev/sdf: 465.76 GiB, 500107862016 bytes, 976773168 sectors
Disk model: Samsung SSD 850
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: ED83AC7B-63B4-408B-B009-2EA157D537DB

Device        Start       End   Sectors  Size Type
/dev/sdf1        34      2047      2014 1007K BIOS boot
/dev/sdf2      2048   1050623   1048576  512M EFI System
/dev/sdf3   1050624  41943040  40892417 19.5G Solaris /usr & Apple ZFS
/dev/sdf4  41943048 943718400 901775353  430G Solaris /usr & Apple ZFS
root@pve:/var/lib/vz#
root@pve:/var/lib/vz# zpool status rpool mpool spool
  pool: mpool
 state: ONLINE
  scan: scrub repaired 0B in 00:11:19 with 0 errors on Sun Sep 12 00:35:30 2021
config:

        NAME                                                     STATE     READ WRITE CKSUM
        mpool                                                    ONLINE       0     0     0
          mirror-0                                               ONLINE       0     0     0
            ata-Samsung_SSD_850_EVO_500GB_S374NX0H503297R-part4  ONLINE       0     0     0
            ata-Samsung_SSD_860_PRO_2TB_S42KNF0K600987B-part4    ONLINE       0     0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:37 with 0 errors on Sun Sep 12 00:24:49 2021
config:

        NAME                                                     STATE     READ WRITE CKSUM
        rpool                                                    ONLINE       0     0     0
          mirror-0                                               ONLINE       0     0     0
            ata-Samsung_SSD_850_EVO_500GB_S374NX0H503297R-part3  ONLINE       0     0     0
            ata-Samsung_SSD_860_PRO_2TB_S42KNF0K600987B-part3    ONLINE       0     0     0

errors: No known data errors

  pool: spool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:07 with 0 errors on Sun Sep 12 00:24:21 2021
config:

        NAME                                                 STATE     READ WRITE CKSUM
        spool                                                ONLINE       0     0     0
          ata-Samsung_SSD_860_PRO_2TB_S42KNF0K600987B-part5  ONLINE       0     0     0

errors: No known data errors
root@pve:/var/lib/vz#

Interestingly it boots the 5.11 kernel, if I select the 2TB SSD as boot device instead of the 500GB SSD.

Currently the system is running and I'm not able to find anything in the logs.

When the kernel crashed, the console has not started scrolling. I don't want to reboot again to take a picture.

Cheers,
luphi