zfs version mismatch after dist-upgrade

[Kingneutron]

Applied latest patches to get zfs 2.2.6 and now my versions are mismatched after rebooting:


zfs -V

zfs-2.2.6-pve1
zfs-kmod-2.2.4-pve1



dpkg -l|grep zfs


ii libzfs4linux 2.2.6-pve1 amd64 OpenZFS filesystem library for Linux - general support
ii libzfs5-devel 2.2.2-1 amd64 Development headers
ii python3-pyzfs 2.2.6-pve1 amd64 wrapper for libzfs_core C library
ii zfs-dkms 2.2.5-1~bpo12+1 all OpenZFS filesystem kernel modules for Linux
ii zfs-initramfs 2.2.6-pve1 all OpenZFS root filesystem capabilities for Linux - initramfs
ii zfs-test 2.2.6-pve1 amd64 OpenZFS test infrastructure and support scripts
ii zfs-zed 2.2.6-pve1 amd64 OpenZFS Event Daemon
ii zfsutils-linux 2.2.6-pve1 amd64 command-line tools to manage OpenZFS filesystems

apt-get is saying the latest version of zfs-dkms is already installed, does someone on the Proxmox packaging end need to fix something?

 

[billy999]

Proxmox does not use (by default) or distribute zfs-dkms. Im assuming youre actively using the dkms module instead of the one shipped with the pve kernel so you need to update it to 2.2.6 from where you initially got it. Its definitely not from pve repos. I assume debian unstable

 

[Neobin]

ZFS 2.2.6 is currently only in pvetest. Make sure, you also have the corresponding PVE kernel 6.8.12-2 from there installed and booted.

And like @billy999 already said, you should really get rid of the zfs-dkms stuff from Debian bookworm-backports...

 

[Kingneutron]

FYI issue was fixed today by applying latest upgrades + reboot - just had to wait about a week

 

[Neobin]

FYI issue was fixed today by applying latest upgrades + reboot - just had to wait about a week

Afaict, there were no further / very recent updates regarding ZFS and/or the kernel on the Proxmox side: [1]; but instead Debian released ZFS 2.2.6 on 2024-09-12 in bookworm-backports: [2], which, again, would indicate that your problem stemmed solely from your, not at all recommended, installation of the zfs-dkms stuff...

Proxmox products come with ZFS already integrated.
With zfs-dkms being also installed, you have two competing sources of ZFS, which one obviously does not want...

But without further informations, especially which packages, including their versions, exactly you updated recently, one can only assume...

[1] http://download.proxmox.com/debian/pve/dists/bookworm/pvetest/binary-amd64
[2] https://tracker.debian.org/pkg/zfs-linux

 

[waltar]

Proxmox does not use (by default) or distribute zfs-dkms. Im assuming youre actively using the dkms module instead of the one shipped with the pve kernel so you need to update it to 2.2.6 from where you initially got it. Its definitely not from pve repos. I assume debian unstable

Afaict, there were no further / very recent updates regarding ZFS and/or the kernel on the Proxmox side: [1]; but instead Debian released ZFS 2.2.6 on 2024-09-12 in bookworm-backports: [2], which, again, would indicate that your problem stemmed solely from your, not at all recommended, installation of the zfs-dkms stuff...

Proxmox products come with ZFS already integrated.
With zfs-dkms being also installed, you have two competing sources of ZFS, which one obviously does not want...

I cannot confirm that - having 4 orig pve's with 8.3.2 (6.8.12-6) in cluster without any other repo definitions as default saying now
zfs -V
zfs-2.2.6-pve1
zfs-kmod-2.2.6-pve1 <- Never ever installed manually, just all pve default repos !!
dpkg --list|grep zfs
ii libzfs4linux 2.2.6-pve1 amd64 OpenZFS filesystem library for Linux - general support
ii zfs-initramfs 2.2.6-pve1 all OpenZFS root filesystem capabilities for Linux - initramfs
ii zfs-zed 2.2.6-pve1 amd64 OpenZFS Event Daemon
ii zfsutils-linux 2.2.6-pve1 amd64 command-line tools to manage OpenZFS filesystems

Having a 5th node with debian+pve with 8.3.3 (6.8.12-7) saying
zfs -V
zfs-2.2.7-pve1
zfs-kmod-2.2.6-pve1 <-- What's that ??
dpkg --list|grep zfs
ii libzfs4linux 2.2.7-pve1 amd64 OpenZFS filesystem library for Linux - general support
ii zfs-initramfs 2.2.7-pve1 all OpenZFS root filesystem capabilities for Linux - initramfs
ii zfs-zed 2.2.7-pve1 amd64 OpenZFS Event Daemon
ii zfsutils-linux 2.2.7-pve1 amd64 command-line tools to manage OpenZFS filesystems

 

[billy999]

I cannot confirm that - having 4 orig pve's with 8.3.2 (6.8.12-6) in cluster without any other repo definitions as default saying now

Cannot confirm what? Your list shows that your PVE doesnt have zfs-dkms installed and that it uses Proxmox' own, integrated kernel module zfs-kmod , thus proving our point

 

[waltar]

Why do we have zfs-kmod-2.2.6-pve1 with 8.3.3 (6.8.12-7) while zfs is 2.2.7 (and no zfs-dkms package listed in dpkg) ?

 

[leesteken]

Why do we have zfs-kmod-2.2.6-pve1 with 8.3.3 (6.8.12-7) while zfs is 2.2.7 (and no zfs-dkms package listed in dpkg) ?

Because there was no new kernel released by Proxmox. Maybe there were no changes in the kernel module and only in user space tools for 2.2.7. Or maybe the kernel update will come soon (as the 2.2.6 and 2.2.7 are compatible anyway) with some other non-ZFS updates as well.

 

[Neobin]

Having a 5th node with debian+pve with 8.3.3 (6.8.12-7) saying
zfs -V
zfs-2.2.7-pve1
zfs-kmod-2.2.6-pve1 <-- What's that ??

I am only seeing this behavior, if the node is not (yet) booted with the new/corresponding kernel:
Before update:

# uname -r
6.11.0-2-pve
# zfs version
zfs-2.2.6-pve1
zfs-kmod-2.2.6-pve1

After update, but before reboot:

# uname -r
6.11.0-2-pve
# zfs version
zfs-2.2.7-pve1
zfs-kmod-2.2.6-pve1

After reboot:

# uname -r
6.11.11-1-pve
# zfs version
zfs-2.2.7-pve1
zfs-kmod-2.2.7-pve1

Because there was no new kernel released by Proxmox. Maybe there were no changes in the kernel module and only in user space tools for 2.2.7. Or maybe the kernel update will come soon (as the 2.2.6 and 2.2.7 are compatible anyway) with some other non-ZFS updates as well.

I am not using the 6.8 kernel, but from the changelog, 6.8.12-7 should have 2.2.7:
https://git.proxmox.com/?p=pve-kern...;hpb=0cc0030692793b1f447996e0da6f402873b3f03b

 

[leesteken]

I am not using the 6.8 kernel, but from the changelog, 6.8.12-7 should have 2.2.7:
https://git.proxmox.com/?p=pve-kern...;hpb=0cc0030692793b1f447996e0da6f402873b3f03b

I guess something went wrong. I also see a 2.2.6 kernel module with kernel 6.8.12-7. I hope a staff member notices this.

 

[Neobin]

Why do we have zfs-kmod-2.2.6-pve1 with 8.3.3 (6.8.12-7) while zfs is 2.2.7 (and no zfs-dkms package listed in dpkg) ?

I guess something went wrong. I also see a 2.2.6 kernel module with kernel 6.8.12-7. I hope a staff member notices this.

Maybe @Stoiko Ivanov can have a look here, please?

 

[fiona]

Hi,
thank you for the reports! From a first look, it seems like a version bump is missing for the top-level zfs submodule for the bookworm-6.8 branch in our kernel repository, but the version for ZFS that is checked out is actually 2.2.7. So this should only be cosmetic and will be fixed in an upcoming version.