ZFS error in journal log

[resprox]

Build/test a server, running proxmox-ve 9.1.0 on kernel: 6.17.9-1-pve on top of Debian 13, machine have 5 disks (3x ssd, 1x nvme, 1x sata): disk1-ssd (OS), efi and boot are not encrypted, root filesystem ext4 is on luks encrypted partition, clevis/tang is configured so the root file system get auto decrypted when boot.

disk2-ssd zfs, disk3-sdd zfs, disk4-nvme zfs are full disk encrypted with luks, zfs on top of those luks encrypted disks, all striped, so single disk per pool.
During boot the 3 disks get auto decrypted (use key files stored on root file system) and configured in /etc/crypttab, disk 5-sata ext4 is for storing (pbs) encrypted backups only, so no need to encrypt in my case.

All went fine, except that every time the server is rebooted, the journal log shows that the 3 pools could not be opened, those events are written at the end of the journal log, when machine finished boot.
There are no other errors, pools are imported fine, an show healthy, vm's starting without issues, so no issue at all, I configured delays in /etc/default/grub and /etc/default/zfs, executed update-grub, update-initramfs -u -k all, pve-efiboot-tool refresh

I have no clue, if I could ignore this safely, as if just annoys me that it shows those errors.
I verified and those errors are only at boot, not duplicated, so only 1 error per pool after boot.

Could someone help me with this?

Grt,

René

Feb 14 10:29:30 jupiter pvestatd[5517]: zfs error: cannot open 'disk4_ssd3': no such pool
Feb 14 10:29:31 jupiter zed[5836]: eid=2 class=config_sync pool='disk4_ssd3'
Feb 14 10:29:31 jupiter zed[5837]: eid=3 class=pool_import pool='disk4_ssd3'
Feb 14 10:29:31 jupiter zed[5870]: eid=5 class=config_sync pool='disk4_ssd3'
Feb 14 10:29:31 jupiter pvestatd[5517]: zfs error: cannot open 'disk1_nvme': no such pool
Feb 14 10:29:31 jupiter zed[6080]: eid=7 class=config_sync pool='disk1_nvme'
Feb 14 10:29:31 jupiter zed[6083]: eid=8 class=pool_import pool='disk1_nvme'
Feb 14 10:29:31 jupiter zed[6114]: eid=10 class=config_sync pool='disk1_nvme'
Feb 14 10:29:31 jupiter pvestatd[5517]: zfs error: cannot open 'disk3_ssd2': no such pool
Feb 14 10:29:31 jupiter zed[6329]: eid=12 class=config_sync pool='disk3_ssd2'
Feb 14 10:29:31 jupiter zed[6330]: eid=13 class=pool_import pool='disk3_ssd2'
Feb 14 10:29:31 jupiter zed[6361]: eid=15 class=config_sync pool='disk3_ssd2'

 

[pulipulichen]

Just a thought – since you mentioned that the pools are on top of LUKS and that Clevis/Tang is used for auto-unlock, could this simply be a boot-time ordering issue?

From the log snippet it looks like pvestatd tries to access the pools and initially reports cannot open 'pool': no such pool, but shortly afterwards the pools are imported and everything becomes healthy.

If the LUKS devices are unlocked slightly later during boot, ZFS or pvestatd might briefly check the pools before the /dev/mapper/... devices actually exist.

That could explain why the errors only appear once during boot and never afterwards.

You might want to check the boot journal and see whether the mapper devices appear after the first ZFS access attempt. If that's the case, it could just be a harmless race condition between LUKS unlock and ZFS/pvestatd initialization.