Proxmox ZFS and Intel QAT

[Doridian]

Greetings,

It seems that Proxmox ships with ZFS baked into the kernel package (the zfs kernel modules are not their own package) and they are also not compiled with Intel QAT enabled.
I have proceeded to install zfs-dkms from buster-backports to get a compile going with QAT support enabled, but this seems a bit annoying to maintain as DKMS (correctly) sees the Proxmox native modules as the same version that it itself provides and therefor I have to
dkms remove/add --force to force it to still put the modules in the right directory for the kernel to load them.

Is there any way I could either get Proxmox without its ZFS kernel modules or a version with QAT enabled from the get go?

Or a more maintainable solution for this that I can't think of somehow? I'd much rather not have to mess around every time I do a kernel update, because if I forget once and reboot, I'll have to reboot again to get the things in.

- Doridian

 

[LnxBil]

I think opening a bug report/feature request for Intel QAT is the best way to go.

 

[ensnare]

Greetings,

It seems that Proxmox ships with ZFS baked into the kernel package (the zfs kernel modules are not their own package) and they are also not compiled with Intel QAT enabled.
I have proceeded to install zfs-dkms from buster-backports to get a compile going with QAT support enabled, but this seems a bit annoying to maintain as DKMS (correctly) sees the Proxmox native modules as the same version that it itself provides and therefor I have to
dkms remove/add --force to force it to still put the modules in the right directory for the kernel to load them.

Is there any way I could either get Proxmox without its ZFS kernel modules or a version with QAT enabled from the get go?

Or a more maintainable solution for this that I can't think of somehow? I'd much rather not have to mess around every time I do a kernel update, because if I forget once and reboot, I'll have to reboot again to get the things in.

- Doridian

Hi — Did you ever figure out a solution to this? It would be even better if Proxmox had full ZFS QAT support built in.

 

[Doridian]

I filed a feature request: https://bugzilla.proxmox.com/show_bug.cgi?id=2619

As for solution? You can install zfs-dkms on Proxmox, just have to carefully navigate the versions (make sure the ZFS tools are also from the same version as zfs-dkms). zfs-dkms supports QAT if you set the environment variables right (as documented).

I roughly followed this post (except I made sure to get the latest version of the QAT API/driver and not just the linked one)
https://forums.servethehome.com/ind...x-qat-acceleration-in-0-7-0.16566/post-217944

 

[Obmawo]

Hi! Any News for the QAT feature?
ZFS hasn't it in the official proxmox module...

The zfs-dkms solution isn't working anaymore for the latest proxmox kernel...

edit:

runs now. Just installed the zfs-dkms from the debian testing branch.

But: This is not a perfect solution, because it doesn't has official support. hopefully the proxmox team will adress this issue.

 

[Ramalama]

Hi! Any News for the QAT feature?
ZFS hasn't it in the official proxmox module...

The zfs-dkms solution isn't working anaymore for the latest proxmox kernel...

edit:

runs now. Just installed the zfs-dkms from the debian testing branch.

But: This is not a perfect solution, because it doesn't has official support. hopefully the proxmox team will adress this issue.

As soon as zfs merges the ZIA merge request, there will be easy ways to add qat support, bill till then I don't believe that anything changes.

But tbh, with latest improvements everywhere (kernel/zfs/newer cpus/etc...), i don't believe that qat will actually improve anything tbh.
Neither in less CPU consumption or faster checksumming or compression.

Only gzip compression, sure, but if you need a better compression as LZ4, there is Zstd.
For checksumming there is aes-ni support by default.

Tbh, zstd might not be as fast as lz4, but way faster as gzip without qat with slightly better compression.
Lz4 should be nowadays faster as gzip with qat, while using same resources.
So the only difference in the end is the compression ratio, but are 5-10% slightly better compression worth the headaches to maintain qzfs in proxmox? I doubt that.

Cheers

 

[Obmawo]

Ok, I see.
Then, what can be a good purpuse for the Intel QAT Chip?

 

[Ramalama]

Ok, I see.
Then, what can be a good purpuse for the Intel QAT Chip?

As far i know you can virtualize it into multiple functions and passthrough.
So probably for pfsense? nginx, anything openssl related?
opnsense has no support for qat, as far i know.

Otherwise not much.
I readed a bit and i was probably half correct at least.
If it comes to those QAT Accelerator Cards 8950/8960, they are indeed not worth anymore in any manner, except if you have an old generation CPU, like Xeon E3/E5 v2/v3/v4 then it may be worth it.

If it comes to newer generation Xeons with built-in QAT in CPU, the QAT here seems to be much more efficient/performant. So it seems like definitively worth using QAT there.
I think the reason is simply because an PCIe Card needs to copy all the necessary blocks to the Card itself to Compress etc... and send back...
While the CPU built-in QAT has direct access to everything (nvme or memory), so its more efficient by design.

But thats what i assume.
However it doesn't change the fact, that it's a pain in the ass to get ZFS with QAT Support on Proxmox, especially with Secure boot nowadays it might be even more painful.

In theory you could build an Compiler-VM for this, with a script that Checks out the Proxmox Kernel + OpenZFS and links it agains QAT librarys, then builds a new Kernel.
So you could use the script to automate the Process, because if you do it by hand, you'll forget some steps after time and you will loose a lot of your time to build weekly or every 2 weeks a new Kernel.

Otherwise you could do the easy route of dkms, but i personally don't like dkms, you will blow the Proxmox Host with Kernel Headers and build environment anyway with this way.
The dkms approach is somewhat only easier in terms of, that you can update the Kernel afterwards, once ZFS+QAT exists as dkms module, but in my opinion dkms is not reliable, it will break after some updates, because the dkms modules gets after some point incompatible.

However, there is still the secure boot signing, i don't know if a custom kernel will work at all with secure boot these days, but on other hand, we can disable secure boot, since i don't see any real reasons where it provides any benefit at all in Proxmox.

In the end how to make a script and compile ZFS+QAT, you're alone, cause i don't believe that anyone will afford the time of testing and writing something for you. Its time consuming. So youll have to google or chatgpt yourself through.

Cheers

 

[Chanabra]

as far as i know they are still viable, the whole point of the feature was that the card was mapped on the cpu rather than a pcie bus, which allows for the quick memory transfers etc that are advertised with QAT//DMA

 

[Ramalama]

as far as i know they are still viable, the whole point of the feature was that the card was mapped on the cpu rather than a pcie bus, which allows for the quick memory transfers etc that are advertised with QAT//DMA

Thanks for the info, but that still wouldnt work with amd plattforms. And intel CPU's have QAT already integrated (some/most of them).