Linux Kernel 5.13, ZFS 2.1 for Proxmox VE

[t.lamprecht]

The upcoming Proxmox VE 7.1 (Q4/2021) will use a 5.13 based Linux kernel including ZFS 2.1. You can test this kernel now, install with:

apt update && apt install pve-kernel-5.13

This version will replace the currently used 5.11 kernel and will be maintained until the Proxmox VE 7.2 release.
The kernel is available on the pve-no-subscription repository at time of writing, so it's not required to enable the pvetest repository.

We invite you to test your hardware with this kernel, and we are thankful for receiving your feedback (please include HW & setup details, especially on issues).

A note regarding ZFS versions:
ZFS is separated in user space tools and kernel module, both have a separate version. The user space tooling is backward compatible to older ZFS kernel module versions, so it's fine if you get 2.1 based user space tools but still use the PVE 5.11 kernel (which is build with ZFS 2.0). You can check both versions with zfs -V, for example:

ZFS 2.1 user space, 5.11 kernel with 2.0 ZFS module:

zfs -V
zfs-2.1.1-pve1
zfs-kmod-2.0.6-pve1

ZFS 2.1 user space, 5.13 kernel with 2.1 ZFS module:

zfs -V
zfs-2.1.1-pve1
zfs-kmod-2.1.1-pve1

 

[MDE86]

Just installed and all working. Will update with any issues.

 

[liberodark]

Have update All PBS & PVE Cluster too

 

[ness1602]

Working okay with EPYC here. Thanks for the kernel.

 

[vesalius]

A bit off topic, but I am seeing ZFS 2.1.1 as an upgrade on PVE 7.0 from the no-subscription repository with the stock 5.11 kernel. I never saw 2.0.6. Is that expected? or does this new kernel upgrade come with the zfs-kmod-2.0.6-pve1 and ZFS-2.1.1 user space?

 

[t.lamprecht]

A bit off topic, but I am seeing ZFS 2.1.1 as an upgrade on PVE 7.0 from the no-subscription repository with the stock 5.11 kernel. I never saw 2.0.6. Is that expected? or does this new kernel upgrade come with the zfs-kmod-2.0.6-pve1 and ZFS-2.1.1 user space?

As said:

ZFS is separated in user space tools and kernel module, both have a separate version. The user space tooling is backward compatible to older ZFS kernel module versions, so it's fine if you get 2.1 based user space tools but still use the PVE 5.11 kernel (which is build with ZFS 2.0). You can check both versions with zfs -V, for example:

 

[jasonsansone]

Warning to anyone thinking about updating to kernel 5.13 - Nvidia drivers are not yet compatible with a kernel past 5.11. I had to roll back to 5.11 because 5.13 broke hardware transcoding support. Otherwise it worked great. I am still running 5.13 on another cluster that does not require GPU's.

 

[Taylan]

Also a warning, see https://forum.proxmox.com/threads/update-to-5-11-22-7-pve-causes-zfs-issues.99401/

 

[t.lamprecht]

Also a warning, see https://forum.proxmox.com/threads/update-to-5-11-22-7-pve-causes-zfs-issues.99401/

This has nothing to do with the 5.13 kernel though, ARC stat and summary tools work just fine with that..

 

[BuFf0k]

2007-2009 Intel E5000 CPU's have a potential compatibility issue with this upgrade. System fails to boot. (So far replicated on two Intel S5000VSA based servers, looking at the forum it appears to also affect Dell and HP servers with this generation CPU)
https://forum.proxmox.com/threads/updated-to-7-1-and-having-boot-issues.99908/#post-431535

 

[olidal]

The upcoming Proxmox VE 7.1 (Q4/2021) will use a 5.13 based Linux kernel including ZFS 2.1. You can test this kernel now, install with:

apt update && apt install pve-kernel-5.13

This version will replace the currently used 5.11 kernel and will be maintained until the Proxmox VE 7.2 release.
The kernel is available on the pve-no-subscription repository at time of writing, so it's not required to enable the pvetest repository.

We invite you to test your hardware with this kernel, and we are thankful for receiving your feedback (please include HW & setup details, especially on issues).

A note regarding ZFS versions:
ZFS is separated in user space tools and kernel module, both have a separate version. The user space tooling is backward compatible to older ZFS kernel module versions, so it's fine if you get 2.1 based user space tools but still use the PVE 5.11 kernel (which is build with ZFS 2.0). You can check both versions with zfs -V, for example:

ZFS 2.1 user space, 5.11 kernel with 2.0 ZFS module:

zfs -V
zfs-2.1.1-pve1
zfs-kmod-2.0.6-pve1

ZFS 2.1 user space, 5.13 kernel with 2.1 ZFS module:

zfs -V
zfs-2.1.1-pve1
zfs-kmod-2.1.1-pve1

Thanks for this great news!

Does it come with AUFS module support?
(see my post here: https://forum.proxmox.com/threads/l...ailed-driver-not-supported.97851/#post-429892)

Rationale for supporting AUFS: OVERLAYFS is not compatible with ZFS, but AUFS is, and AUFS also comes with unique additional features (eg. writes can be directed to covered FS instead of all going to upper file system)

Thanks

 

[t.lamprecht]

Does it come with AUFS module support?

No, I'm afraid not, see https://forum.proxmox.com/threads/kernel-5-11.86225/page-4#post-390667 for a bit more info.

Rationale for supporting AUFS: OVERLAYFS is not compatible with ZFS, but AUFS is, and AUFS also comes with unique additional features (eg. writes can be directed to covered FS instead of all going to upper file system)

The overlayfs situation is not ideal, but, there's some semi-active work going on to support OverlayFS in ZFS, e.g. see this comment for describing current efforts somewhat: https://github.com/openzfs/zfs/pull/9414#issuecomment-939201415

 

[olidal]

No, I'm afraid not, see https://forum.proxmox.com/threads/kernel-5-11.86225/page-4#post-390667 for a bit more info.


The overlayfs situation is not ideal, but, there's some semi-active work going on to support OverlayFS in ZFS, e.g. see this comment for describing current efforts somewhat: https://github.com/openzfs/zfs/pull/9414#issuecomment-939201415

Thank you Thomas.

This is a very bad news on my side as we are developing a Proxmox-based product that depends on some of aufs' unique features.

I guess I am left with the only option of maintaining my own promox kernel variant and release it through our own repo (which does not exist yet). Unless we could find a way to distribute this alternate kernel build through the Proxmox repo?

I would be very happy to share that effort with the Proxmox community, and that would save me quite some efforts trying to maintain a debian repo just for that kernel.

Just let me know,

Kindest regards,
Olivier.

 

[t.lamprecht]

I guess I am left with the only option of maintaining my own promox kernel variant and release it through our own repo (which does not exist yet).

Hmm, I mean you could go for the dkms or "build-only the module" approach, but you'd need to maintain that yourself - but it may be less work required than rebuilding every kernel (albeit I'm guesstimating this one).

Unless we could find a way to distribute this alternate kernel build through the Proxmox repo?

Well, our repos are not accessible from the public and the signing key is locked down on an HSM.
Contributing the changes required to add aufs (again) and maintain that could be a possibility on the other hand, it would be contained in our kernel package then. But, I'd like to wait out for Q2'22 to make a decision for accepting such a thing:

• as Proxmox VE 6.4 is still supported till 2022-07 and it has still aufs support (so you could use that in the meantime)
• ZFS may get overlayfs support until then (knocks wood), making such extra "maintain-aufs" effort superfluous

 

[olidal]

Hmm, I mean you could go for the dkms or "build-only the module" approach, but you'd need to maintain that yourself - but it may be less work required than rebuilding every kernel (albeit I'm guesstimating this one).

Can I "build only the module"? I tried but was never able to insmod it, because the module was not matching with the kernel. I'd be happy to try this option if you can help or give some tips on how to match the two parts.

Well, our repos are not accessible from the public and the signing key is locked down on an HSM.

The safe way for sure.

Contributing the changes required to add aufs (again) and maintain that could be a possibility on the other hand, it would be contained in our kernel package then. But, I'd like to wait out for Q2'22 to make a decision for accepting such a thing:

• as Proxmox VE 6.4 is still supported till 2022-07 and it has still aufs support (so you could use that in the meantime)

The other issue I have is that older 5.4 kernel do not support some H/W such as the I219-V intel NIC, and you dont include AUFS in the 5.11 backport for PVE6.4, so I guess moving on now to PVE7 is still my best option.

• ZFS may get overlayfs support until then (knocks wood), making such extra "maintain-aufs" effort superfluous

Sadly, that will never solve my problem as I am using aufs for some of its unique features, not planned for inclusion in overlayfs.

Thanks again, please let me know if we can try to work together on the "build-only the module" option, which I would be happy to maintain and contribute for the Proxmox community.

Olivier.

 

[BuFf0k]

2007-2009 Intel E5000 CPU's have a potential compatibility issue with this upgrade. System fails to boot. (So far replicated on two Intel S5000VSA based servers, looking at the forum it appears to also affect Dell and HP servers with this generation CPU)
https://forum.proxmox.com/threads/updated-to-7-1-and-having-boot-issues.99908/#post-431535

This issue seems fixed with pve-kernel-5.13.19-1-pve (5.13.19-3)

 

[gerby]

First gen EPYC with a Windows 10 Guest and HOST cpu type; guest failed to boot. Setting CPU type EPYC resolved the issue.