Failed to insert module 'zfs': Key was rejected by service - problems since kernel 6.8.12-9-pve?

W

Wesumat

New Member
Apr 6, 2025
8
1
3
Germany
Hi,

i'm having a strange problem with my bare metal offsite Proxmox Backup system, obviously since the laste update
to the newest kernel 6.8.12-9-pve. After the bootup the zfs pools cannot be imported by the system accordingly to
problem with loading the zfs kernel modules, also the systemd-modules-load.service them to fail during boot process.

The system is an hp Microserver gen8 (Intel Core i5-3470T, 8GB RAM) booting in legacy mode. Looking for key reject
failure articles often points to signing problems in relationship to uefi boot - which for me doesn't fit in this situation.

At the moment I pinned the kernel to 6.8.12-8-pve, which is working without any zfs problems so far.


errors in the logfile with kernel 6.8.12-9-pve:
Code: 
...
Apr 05 08:35:14 pbs-offsite systemd-journald[311]: Journal started
Apr 05 08:35:14 pbs-offsite systemd-journald[311]: Runtime Journal (/run/log/journal/98b229f1f0e1463fb3a1a93a9a86966c) is 8.0M, max 78.8M, 70.8M free.
Apr 05 08:35:12 pbs-offsite systemd-modules-load[312]: Failed to insert module 'zfs': Key was rejected by service
Apr 05 08:35:14 pbs-offsite systemd[1]: Started systemd-journald.service - Journal Service.
Apr 05 08:35:12 pbs-offsite systemd-udevd[331]: Using default interface naming scheme 'v252'.
Apr 05 08:35:13 pbs-offsite lvm[379]: PV /dev/sda3 online, VG pbs is complete.
Apr 05 08:35:13 pbs-offsite lvm[379]: VG pbs finished
Apr 05 08:35:14 pbs-offsite kernel: usbcore: registered new interface driver usbmouse
Apr 05 08:35:14 pbs-offsite kernel: usbcore: registered new interface driver usbkbd
Apr 05 08:35:13 pbs-offsite systemd-modules-load[382]: Failed to insert module 'zfs': Key was rejected by service
Apr 05 08:35:13 pbs-offsite (udev-worker)[335]: sdc1: Process '/sbin/modprobe zfs' failed with exit code 1.
Apr 05 08:35:13 pbs-offsite (udev-worker)[345]: sdb1: Process '/sbin/modprobe zfs' failed with exit code 1.
Apr 05 08:35:13 pbs-offsite (udev-worker)[338]: sdd1: Process '/sbin/modprobe zfs' failed with exit code 1.
Apr 05 08:35:14 pbs-offsite lvm[303]:   2 logical volume(s) in volume group "pbs" monitored
Apr 05 08:35:13 pbs-offsite systemd-modules-load[412]: Failed to insert module 'zfs': Key was rejected by service
Apr 05 08:35:13 pbs-offsite systemd-modules-load[444]: Failed to insert module 'zfs': Key was rejected by service
Apr 05 08:35:14 pbs-offsite udevadm[334]: systemd-udev-settle.service is deprecated. Please fix zfs-import-cache.service, zfs-import-scan.service not to pull it in.
Apr 05 08:35:14 pbs-offsite kernel: input: BMC Virtual Keyboard  as /devices/pci0000:00/0000:00:1c.7/0000:01:00.4/usb1/1-1/1-1:1.0/0003:03F0:7029.0001/input/input5
Apr 05 08:35:14 pbs-offsite systemd[1]: Starting systemd-journal-flush.service - Flush Journal to Persistent Storage...
Apr 05 08:35:14 pbs-offsite systemd-journald[311]: Time spent on flushing to /var/log/journal/98b229f1f0e1463fb3a1a93a9a86966c is 148.751ms for 1054 entries.
Apr 05 08:35:14 pbs-offsite systemd-journald[311]: System Journal (/var/log/journal/98b229f1f0e1463fb3a1a93a9a86966c) is 74.5M, max 4.0G, 3.9G free.
Apr 05 08:35:14 pbs-offsite systemd-journald[311]: Received client request to flush runtime journal.
Apr 05 08:35:14 pbs-offsite kernel: hid-generic 0003:03F0:7029.0001: input,hidraw0: USB HID v1.01 Keyboard [BMC Virtual Keyboard ] on usb-0000:01:00.4-1/input0
Apr 05 08:35:14 pbs-offsite kernel: input: BMC Virtual Keyboard  as /devices/pci0000:00/0000:00:1c.7/0000:01:00.4/usb1/1-1/1-1:1.1/0003:03F0:7029.0002/input/input6
Apr 05 08:35:14 pbs-offsite kernel: hid-generic 0003:03F0:7029.0002: input,hidraw1: USB HID v1.01 Mouse [BMC Virtual Keyboard ] on usb-0000:01:00.4-1/input1
Apr 05 08:35:14 pbs-offsite systemd[1]: Finished systemd-journal-flush.service - Flush Journal to Persistent Storage.
Apr 05 08:35:14 pbs-offsite systemd[1]: Finished ifupdown2-pre.service - Helper to synchronize boot up for ifupdown.
Apr 05 08:35:14 pbs-offsite systemd[1]: Finished systemd-udev-settle.service - Wait for udev To Complete Device Initialization.
Apr 05 08:35:14 pbs-offsite systemd[1]: Starting zfs-import@offsite\x2dzfs.service - Import ZFS pool offsite\x2dzfs...
Apr 05 08:35:14 pbs-offsite zpool[467]: Failed to initialize the libzfs library.
Apr 05 08:35:14 pbs-offsite systemd[1]: zfs-import@offsite\x2dzfs.service: Main process exited, code=exited, status=1/FAILURE
Apr 05 08:35:14 pbs-offsite systemd[1]: zfs-import@offsite\x2dzfs.service: Failed with result 'exit-code'.
Apr 05 08:35:14 pbs-offsite systemd[1]: Failed to start zfs-import@offsite\x2dzfs.service - Import ZFS pool offsite\x2dzfs.
Apr 05 08:35:14 pbs-offsite systemd[1]: zfs-import-cache.service - Import ZFS pools by cache file was skipped because of an unmet condition check (ConditionPathIsDirectory=/sys/module/zfs).
Apr 05 08:35:14 pbs-offsite systemd[1]: zfs-import-scan.service - Import ZFS pools by device scanning was skipped because of an unmet condition check (ConditionPathIsDirectory=/sys/module/zfs).
Apr 05 08:35:14 pbs-offsite systemd[1]: Reached target zfs-import.target - ZFS pool import target.
Apr 05 08:35:14 pbs-offsite systemd[1]: zfs-mount.service - Mount ZFS filesystems was skipped because of an unmet condition check (ConditionPathIsDirectory=/sys/module/zfs).
Apr 05 08:35:14 pbs-offsite systemd[1]: Reached target local-fs.target - Local File Systems.
...


Trying to load the module manually is reproducing the error

root@pbs-offsite:~# modprobe zfs
modprobe: ERROR: could not insert 'zfs': Key was rejected by service

The systemd-modules-load.service is also in error state after bootup

root@pbs-offsite:~# systemctl status systemd-modules-load.service
× systemd-modules-load.service - Load Kernel Modules
Loaded: loaded (/lib/systemd/system/systemd-modules-load.service; static)
Active: failed (Result: exit-code) since Sat 2025-04-05 08:52:03 CEST; 4min 58s ago
Docs: man:systemd-modules-load.service(8)
man:modules-load.d(5)
Process: 431 ExecStart=/lib/systemd/systemd-modules-load (code=exited, status=1/FAILURE)
Main PID: 431 (code=exited, status=1/FAILURE)
CPU: 53ms


Comparison 6.8.12-8-pve (-b -1) vs. 6.8.12-9-pve (-b):

root@pbs-offsite:~# journalctl -b -1 -k -p err
Apr 05 08:45:56 pbs-offsite kernel: ACPI: SPCR: [Firmware Bug]: Unexpected SPCR Access Width. Defaulting to byte size
Apr 05 08:45:56 pbs-offsite kernel: DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x00000000000e8000-0x00000000000e8fff], contact BIOS vendor for fixes
Apr 05 08:45:56 pbs-offsite kernel: [Firmware Bug]: the BIOS has corrupted hw-PMU resources (MSR 38d is 330)
Apr 05 08:45:56 pbs-offsite kernel: DMAR: DRHD: handling fault status reg 2
Apr 05 08:45:56 pbs-offsite kernel: DMAR: [INTR-REMAP] Request device [01:00.0] fault index 0x17 [fault reason 0x26] Blocked an interrupt request due to source-id verification failure

root@pbs-offsite:~# journalctl -b -k -p err
Apr 05 08:52:02 pbs-offsite kernel: ACPI: SPCR: [Firmware Bug]: Unexpected SPCR Access Width. Defaulting to byte size
Apr 05 08:52:02 pbs-offsite kernel: DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x00000000000e8000-0x00000000000e8fff], contact BIOS vendor for fixes
Apr 05 08:52:02 pbs-offsite kernel: [Firmware Bug]: the BIOS has corrupted hw-PMU resources (MSR 38d is 330)
Apr 05 08:52:03 pbs-offsite kernel: DMAR: DRHD: handling fault status reg 2
Apr 05 08:52:03 pbs-offsite kernel: DMAR: [INTR-REMAP] Request device [01:00.0] fault index 0x14 [fault reason 0x26] Blocked an interrupt request due to source-id verification failure
Apr 05 08:52:04 pbs-offsite systemd[1]: Failed to start systemd-modules-load.service - Load Kernel Modules.
Apr 05 08:52:04 pbs-offsite systemd[1]: Failed to start systemd-modules-load.service - Load Kernel Modules.
Apr 05 08:52:04 pbs-offsite systemd[1]: Failed to start systemd-modules-load.service - Load Kernel Modules.


So far i tried the following

- disabling VT-d and VT-x in the BIOS options
- disabling iommu in the default grub options (Apr 05 22:27:02 pbs-offsite kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.12-9-pve root=/dev/mapper/pbs-root ro quiet intel_iommu=off)

The problem seems to persist, any suggestions on this behavior?
 
Please post the outpot of following commands from the shell:
Bash: 
# Which kernel options were passed at boot?
cat /proc/cmdline
# What sysctl settings do we have?
sysctl -a

Maybe you have enabled that unsigned kernel modules are forbidden or something like that.
 
boot with kernel 6.8.12-9-pve:

root@pbs-offsite:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.8.12-9-pve root=/dev/mapper/pbs-root ro quiet intel_iommu=off
...
see attachement, messages gets to long for forum software


boot with kernel 6.8.12-8-pve:
root@pbs-offsite:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.8.12-8-pve root=/dev/mapper/pbs-root ro quiet intel_iommu=off
...
see attachement, messages gets to long for forum software


This seem to be the only differences.
kernel_diff.png
 

Attachments

Wesumat said:
boot with kernel 6.8.12-9-pve:

root@pbs-offsite:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.8.12-9-pve root=/dev/mapper/pbs-root ro quiet intel_iommu=off
...
see attachement, messages gets to long for forum software


boot with kernel 6.8.12-8-pve:
root@pbs-offsite:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.8.12-8-pve root=/dev/mapper/pbs-root ro quiet intel_iommu=off
...
see attachement, messages gets to long for forum software


This seem to be the only differences.
View attachment 84720
Click to expand...
Can't say if it matters, but I recently ran into the same issue on our pbs. This presented itself only after I had moved the OS disk to different storage. Debian booted, but had all the same issues with modules not loading. I was fortunate enough to not delete the old disk when I moved(copied) it. After putting the old disk in place, it booted fine. Then I was able to make a backup from that and then restore it to the storage I wanted it on.
 
the combination of key-error (which might point to a secure-boot issue) and the hp-g8 series (not sure if they do support EFI-boot/Secure boot at all?)
is indeed odd.

on a hunch - did you maybe at some point install some DKMS modules? (maybe even ones for ZFS)?

else the following outputs might help:
* `zfs version`
* `zpool import`
* `modinfo zfs`
* `dkms status`
* `mount |grep efi`
* `dmesg | grep -i secure`
* `pveversion -v` (or `proxmox-backup-manager version --verbose`)
 
Last edited:
Wesumat said:
Apr 05 08:52:02 pbs-offsite kernel: DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x00000000000e8000-0x00000000000e8fff], contact BIOS vendor for fixes
Click to expand...

unrelated to the ZFS pool not getting imported - if I remember correctly - `intremap=off` on the kernel commandline can help with some issues on HP G8 series hardware (at least DL380)
 
Stoiko Ivanov said:
the combination of key-error (which might point to a secure-boot issue) and the hp-g8 series (not sure if they do support EFI-boot/Secure boot at all?)
is indeed odd.

on a hunch - did you maybe at some point install some DKMS modules? (maybe even ones for ZFS)?

else the following outputs might help:
* `zfs version`
* `zpool import`
* `modinfo zfs`
* `dkms status`
* `mount |grep efi`
* `dmesg | grep -i secure`
* `pveversion -v` (or `proxmox-backup-manager version --verbose`)
Click to expand...
The weirdest part for us was even though we have zfs installed by default, there are no zfs pools. Our backup storage is nfs and the local disk is ext4. Seems like I should consider removing zfs to prevent this in the future.
 
Hi,

no i did not manualy install any special dkms modules, just normal PBS updates via GUI.

The hp gen8 Microserver is not EFI capable, just legacy BIOS, which is the also the actual boot mode.


Code: 
root@pbs-offsite:~# zfs version
Failed to initialize the libzfs library.

root@pbs-offsite:~# zfs import
Failed to initialize the libzfs library.

root@pbs-offsite:~# modinfo zfs
filename:       /lib/modules/6.8.12-9-pve/zfs/zfs.ko
version:        2.2.7-pve2
license:        CDDL
license:        Dual BSD/GPL
license:        Dual MIT/GPL
author:         OpenZFS
description:    ZFS
alias:          zzstd
alias:          zcommon
alias:          zunicode
alias:          znvpair
alias:          zlua
alias:          icp
alias:          zavl
alias:          devname:zfs
alias:          char-major-10-249
srcversion:     5048CA0AD18BE2D2F9020C5
depends:        spl
retpoline:      Y
name:           zfs
vermagic:       6.8.12-9-pve SMP preempt mod_unload modversions
sig_id:         PKCS#7
signer:         Build time autogenerated kernel key
sig_key:        79:70:BC:20:D2:5B:45:80:F7:B8:40:85:F3:27:17:FD:F6:A8:CA:6F
sig_hashalgo:   sha512
signature:      83:0C:0C:2A:34:37:B9:D9:E5:3E:85:63:C8:7B:12:A4:0F:FA:F4:C0:
                83:32:38:8D:B0:81:81:89:4F:EC:83:77:3F:D0:82:AB:D1:5A:4C:92:
                80:9A:CB:9E:84:97:15:F6:F4:23:77:36:76:14:BE:48:8C:3C:D1:38:
                5A:A1:75:1E:24:6D:7F:C0:ED:CF:22:7D:43:04:2F:E0:DE:65:8B:7E:
                63:A0:DC:C6:5B:4D:8D:49:17:88:18:F4:E4:05:FB:96:4D:72:D1:DC:
                1E:A9:90:09:DE:B8:0C:42:ED:C8:97:B4:6A:80:8E:DB:09:24:91:EE:
                88:74:A1:6D:6A:F0:C7:29:0F:6C:6E:69:6F:1A:7B:0E:96:7A:A4:43:
                91:52:2F:5B:16:8E:FB:4C:A9:F2:DA:39:66:17:74:02:9C:84:B0:F7:
                81:5E:B2:24:91:4E:8A:BF:16:71:11:98:A6:03:0F:5F:9E:2E:F9:62:
                C5:B0:7D:97:49:4C:4A:BD:A0:4F:04:F2:12:D8:B5:E5:D5:CE:65:01:
                F2:77:3E:FD:CE:AA:C6:6B:3E:38:2B:2F:E5:97:30:09:BE:CC:F5:4B:
                60:49:24:F9:6F:56:00:5E:EF:03:8F:A3:8D:64:5C:1D:DB:9E:15:02:
                F4:A3:05:2E:9E:C7:2D:39:13:E9:8E:DB:B2:93:67:87:9F:FC:29:E0:
                A5:67:2A:81:89:D3:B5:88:46:18:10:9D:B5:87:37:F5:20:9A:85:7B:
                3A:74:D5:2A:2E:93:0D:58:D6:A4:E5:78:84:2B:96:05:4F:A9:50:94:
                78:CC:18:29:5D:1B:01:62:33:E1:F0:F4:6D:17:EA:C7:07:9A:ED:15:
                C7:B4:B9:60:DE:03:B5:3E:F5:D5:1F:8E:13:5E:D0:79:72:BF:99:D2:
                F5:C5:D5:8F:8B:97:4E:39:71:39:56:2A:53:98:0C:C7:3E:E9:30:F5:
                87:E5:FC:41:14:B2:93:5C:B6:02:5F:CB:9D:A3:96:A9:0C:6F:59:C2:
                FF:49:13:CE:27:35:A5:65:DB:12:9A:4B:EE:42:DD:22:FB:4B:36:E7:
                86:DD:85:AD:06:48:67:2F:DB:E6:2A:CD:0C:A6:E8:89:02:75:85:46:
                C9:52:38:D4:D8:9F:44:3E:1D:BC:CB:DD:06:97:8E:DF:87:DF:FB:C2:
                37:6F:D2:2B:28:9F:FA:7A:1A:2B:10:E9:00:96:C8:FB:43:98:D0:98:
                B9:45:22:05:31:5A:6B:AD:33:0A:AE:DF:21:02:4A:0E:3C:2E:3B:F7:
                DD:97:6B:B3:A8:90:77:76:A7:91:48:1C:96:05:E5:74:A2:94:A7:72:
                01:53:26:01:0A:00:27:CC:1F:44:D6:75
...
see attachement for full text, messages gets to long for forum software


root@pbs-offsite:~# dkms status
-bash: dkms: command not found

root@pbs-offsite:~# mount | grep efi

root@pbs-offsite:~# dmesg | grep -i secure

root@pbs-offsite:~# proxmox-backup-manager version --verbose
proxmox-backup                     3.3.0         running kernel: 6.8.12-9-pve
proxmox-backup-server              3.3.6-1       running version: 3.3.6
proxmox-kernel-helper              8.1.1
proxmox-kernel-6.8                 6.8.12-9
proxmox-kernel-6.8.12-9-pve-signed 6.8.12-9
proxmox-kernel-6.8.12-8-pve-signed 6.8.12-8
proxmox-kernel-6.8.12-4-pve-signed 6.8.12-4
ifupdown2                          3.2.0-1+pmx11
libjs-extjs                        7.0.0-5
proxmox-backup-docs                3.3.6-1
proxmox-backup-client              3.3.6-1
proxmox-mail-forward               0.3.1
proxmox-mini-journalreader         1.4.0
proxmox-offline-mirror-helper      0.6.7
proxmox-widget-toolkit             4.3.7
pve-xtermjs                        5.5.0-1
smartmontools                      7.3-pve1
zfsutils-linux                     2.2.7-pve2
root@pbs-offsite:~#
 

Attachments

Wesumat said:
Failed to initialize the libzfs library.
Click to expand...
hm - that's unexpected - and might be a hint.

please provide:
* `dpkg -l |grep -Ei '(zfs|zpool|2.2.*-pve)'`
* `strace -yyftt -o ./zfs-version.trace zfs version`
(the last one will create a file ./zfs-version.trace - please attach that one as well)

Thanks!
 
Code: 
root@pbs-offsite:~# dpkg -l |grep -Ei '(zfs|zpool|2.2.*-pve)'
ii  libjs-qrcodejs                     1.20230525-pve1                     all          Cross-Browser JavaScript library for making QRCodes
ii  libnvpair3linux                    2.2.7-pve2                          amd64        Solaris name-value library for Linux
ii  libuutil3linux                     2.2.7-pve2                          amd64        Solaris userland utility library for Linux
ii  libzfs4linux                       2.2.7-pve2                          amd64        OpenZFS filesystem library for Linux - general support
ii  libzpool5linux                     2.2.7-pve2                          amd64        OpenZFS pool library for Linux
ii  spl                                2.2.7-pve2                          all          Solaris Porting Layer user-space utilities for Linux (dummy)
ii  zfs-initramfs                      2.2.7-pve2                          all          OpenZFS root filesystem capabilities for Linux - initramfs
ii  zfs-zed                            2.2.7-pve2                          amd64        OpenZFS Event Daemon
ii  zfsutils-linux                     2.2.7-pve2                          amd64        command-line tools to manage OpenZFS filesystems
root@pbs-offsite:~# strace -yyftt -o ./zfs-version.trace zfs version
Failed to initialize the libzfs library.
 

Attachments

I knew i missed something - sorry :oops:

added the trace in prior post

Code: 
root@pbs-offsite:/usr/share/locale# locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=


root@pbs-offsite:/usr/share/locale# locale -a
C
C.utf8
en_US.utf8
POSIX
 
Last edited:
  • Like
Reactions: Stoiko Ivanov
Thanks for the trace-file!

I think the issue might be related to the following lines:
Code: 
1109  14:24:25.334871 access("/dev/zfs", F_OK) = 0
...
1109  14:24:25.335151 openat(AT_FDCWD</root>, "/dev/zfs", O_RDWR|O_EXCL|O_CLOEXEC) = -1 ENODEV (No such device)

which I cannot reproduce locally.

* Is the zfs module loaded? (`lsmod |grep -i zfs`)
* what's the output of `stat /dev/zfs` and `ls -la /dev/ |grep zfs`?
* can you remove it (potentially after stopping zfs-zed): `rmmod zfs spl`
please try doing so and running `zfs version` after unloading the module
afterwards check if the journal has any information - if possible simply attach the complete journal since boot (`journalctl -b`) - I hope there would be some indication of why /dev/zfs cannot be opened ...
 
I think with the 6.8.12-9-pve kernel the zfs module isn't active because of problems during the boot process at all.


Code: 
root@pbs-offsite:~# lsmod | grep -i zfs
root@pbs-offsite:~# stat /dev/zfs
  File: /dev/zfs
  Size: 0               Blocks: 0          IO Block: 4096   character special file
Device: 0,5     Inode: 532         Links: 1     Device type: 10,249
Access: (0666/crw-rw-rw-)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2025-04-14 17:50:11.168138140 +0200
Modify: 2025-04-14 17:50:11.168138140 +0200
Change: 2025-04-14 17:50:11.168138140 +0200
 Birth: 2025-04-14 17:50:11.071138144 +0200
root@pbs-offsite:~# ls -la /dev/ | grep zfs
crw-rw-rw-  1 root root     10, 249 Apr 14 17:50 zfs
root@pbs-offsite:~# rmmod zfs spl
rmmod: ERROR: Module zfs is not currently loaded
root@pbs-offsite:~# zfs version
Failed to initialize the libzfs library.
root@pbs-offsite:~#
 

Attachments

Thanks! - my mistake - the module-load fails, thus i guess that zfs version fails.

Do you get any more output from `modprobe zfs` ?
 
does loading other modules work? could you post the output of "sha256sum /usr/lib/modules/6.8.12-9-pve/zfs/zfs.ko"?
 
Code: 
root@pbs-offsite:/usr/lib/modules/6.8.12-9-pve/zfs# ls -altr
total 9884
-rw-r--r-- 1 root root 9845321 Mar 16 20:18 zfs.ko
-rw-r--r-- 1 root root  265385 Mar 16 20:18 spl.ko
drwxr-xr-x 2 root root    4096 Mar 29 07:44 .
drwxr-xr-x 4 root root    4096 Mar 29 07:44 ..
root@pbs-offsite:/usr/lib/modules/6.8.12-9-pve/zfs# sha256sum spl.ko
17549146a4cb4065442ce82d152afcc04722a39c67fc39e5b80888b47cd631c7  spl.ko
root@pbs-offsite:/usr/lib/modules/6.8.12-9-pve/zfs# sha256sum zfs.ko
13fc12e4735e734a9f2bc42135198d33b6363301e42f24e0f595495a68f24f86  zfs.ko
root@pbs-offsite:/usr/lib/modules/6.8.12-9-pve/zfs#


In comparison to my working pbs system with running 6.8.12-9-pve
root@pbs:/usr/lib/modules/6.8.12-9-pve/zfs# ls -altr
total 4094
-rw-r--r-- 1 root root 9845321 Mar 16 20:18 zfs.ko
-rw-r--r-- 1 root root  265385 Mar 16 20:18 spl.ko
drwxr-xr-x 2 root root       4 Mar 26 19:09 .
drwxr-xr-x 4 root root      17 Mar 26 19:09 ..
root@pbs:/usr/lib/modules/6.8.12-9-pve/zfs# sha256sum spl.ko
17549146a4cb4065442ce82d152afcc04722a39c67fc39e5b80888b47cd631c7  spl.ko
root@pbs:/usr/lib/modules/6.8.12-9-pve/zfs# sha256sum zfs.ko
863a6dfa35b04f6eda014e9342b71024253306f068b2572df25fbd80fa82a3ef  zfs.ko
root@pbs:/usr/lib/modules/6.8.12-9-pve/zfs#


The zfs.ko differs. Testing it from the working machine leads to:

root@pbs-offsite:/usr/lib/modules/6.8.12-9-pve/zfs# modprobe zfs
root@pbs-offsite:/usr/lib/modules/6.8.12-9-pve/zfs# lsmod | grep zfs
zfs                  6168576  0
spl                   143360  1 zfs
root@pbs-offsite:/usr/lib/modules/6.8.12-9-pve/zfs# zfs version
zfs-2.2.7-pve2
zfs-kmod-2.2.7-pve2
root@pbs-offsite:/usr/lib/modules/6.8.12-9-pve/zfs# ls -altr
total 29116
-rw-r--r-- 1 root root  265385 Mar 16 20:18 spl.ko
drwxr-xr-x 4 root root    4096 Mar 29 07:44 ..
-rw-r--r-- 1 root root 9845321 Apr 15 19:38 zfs.ko_pbs-ok
drwxr-xr-x 2 root root    4096 Apr 15 19:38 .
-rw-r--r-- 1 root root 9845321 Apr 15 19:38 zfs.ko-nok
-rw-r--r-- 1 root root 9845321 Apr 15 19:39 zfs.ko
root@pbs-offsite:/usr/lib/modules/6.8.12-9-pve/zfs#

Tested reboot with replaced zfs.ko - kernel 6.8.12-9-pve is working now with running zfs and imported pools.

Any idea, what could have caused this?
 
Ranging from best case scenario to worst case scenario:
Bit flipped by an event that came from space
Corrupted/replaced during a failed upgrade and you ignored the error
Drive/controller/RAM/CPU is failing and randomly corrupting your data
You screwed something up during another operation
You got hacked and someone is attempting to replace kernel modules
Your network is compromised and Proxmox packages are being replaced in transit
Proxmox build servers/repos are/were failing
Proxmox build server/repos were compromised

Given your event is just a sample of 1 and the age of your hardware I would suggest looking at one of the 4 first options.
 
Last edited:
if you still have the original corrupt module, you could check how much it differs (e.g. using vbindiff or a similar tool).
 
  • Like
Reactions: Stoiko Ivanov
You must log in or register to reply here.
Top Bottom