Compression accelerator for PVE Backups

[Tmanok]

Hey Everyone,

PCIe accelerator cards of all shapes and sizes interest me, mostly because CPUs are these hulking beasts that handle significant amounts of processing and they are constantly taxed from "side" or "auxiliary" workloads outside of their primary tasks.

Take a hypervisor, proxmox for example. It's primary purpose is to run virtual machines. It's secondary purposes will be networking (VLANs, firewall, bonding, tunneling, etc), network traffic encryption, VM graphics, storage device management, compression of backups or local storage, and on and on...

With the X5600 series of CPUs being the top dog for a while in servers, many servers had E5400 series and AMD Opteron garbage, while also processing new and exciting virtualization workloads. During that time there was an evident lack of CPU power and FPGAs and ASICs in servers were becoming a very popular idea (from my limited perspective). That died down for a while as CPUs became more powerful (basically X5600 series and later) but now we're back at pushing the limits of storage and networking and CPU performance with 100Gbps-400Gbps networking per interface, spawning Data Processing Units (NVIDIA Bluefield DPUs) for boosted network and edge of server storage performance. We use graphics cards for graphical workload performance on Proxmox all the time- to the point where there are more Gaming on Proxmox guides than I can be bothered to read...

Many cards still exist to reduce CPU taxation caused by compression, encryption, firewall tasks, etc as we also begin to hyperconverge many many once different roles into a single cluster of machines. So here's my question...

What is the support status in Linux and Proxmox for cards that utilize GZIP compression acceleration such as the IBM card found here: https://www.ibm.com/docs/en/linux-o...i-pcie3-fpga-compression-accelerator-adapters ? There are cheaper cards available online that also interest me. The specific use case that I see, is with Proxmox Backup Server and Proxmox Virtual Environment backups in general as they utilize excellent compression to reduce storage consumption. Would it not reduce CPU taxation to use an accelerator and how feasible would it be to install this card and have Proxmox utilize it for those purposes? Perhaps my use case is not even the best, so what about local storage compression (e.g. ZFS Lz4 which is likely incompatible with the card mentioned but might be with another FPGA).

Thanks, hope this is an interesting question for the developers and the community. Also worth putting it on the Google Search map for other administrators and engineers to find when they have a similar question.


Tmanok

 

[RolandK]

gzip is dead. use zstd.

In never used gzip again since zstd exists.

trying to answer your question:

regarding status of linux and proxmox support for compression add-on-card, i think it's a difficult theme....as this cards are not mainstream and afaik, there is no real standard with these....

with vzdump and with pbs, compression is implemented in a very different way, too. with vzdump it's the linux compression binary, with pbs, afaik it's in the rust libs.

using accelerated compression with vzdump (i.e. full backups) should probably be easier to implement, imho and that should work without support of proxmox development. with pbs, i think you're out of luck without proxmox devs.

 

[Tmanok]

gzip is dead. use zstd.

In never used gzip again since zstd exists.

trying to answer your question:

regarding status of linux and proxmox support for compression add-on-card, i think it's a difficult theme....as this cards are not mainstream and afaik, there is no real standard with these....

with vzdump and with pbs, compression is implemented in a very different way, too. with vzdump it's the linux compression binary, with pbs, afaik it's in the rust libs.

using accelerated compression with vzdump (i.e. full backups) should probably be easier to implement, imho and that should work without support of proxmox development. with pbs, i think you're out of luck without proxmox devs.

Hey Roland,

Personally, I use zstd everywhere I can. When I listed gzip and lz4 it was merely because the FPGA supported it and ZFS uses LZ4 by default ¯\_(ツ)_/¯
Completely agree with the sentiment. Worst case use pgzip but even then....

Hmm, I suppose you're right in that many aspects of Proxmox VE are Proxmox Dev driven and not Linux Kernel or Red Hat driven, but they're able to upstream some requests of course. I actually had some thoughts on this in the meantime, however.

Given that the compression / accelerator card merely uses another (supposedly) compatible executable to gzip, perhaps there are two solutions:

1. Short Term Solution: symlink /usr/bin/gzip to /usr/bin/genwqe_gzip
   1. Perhaps there are alternatives for multiple compression algorithms etc.
   2. I can't see this working nicely given that it requires an algorithm argument (e.g. -AGENWQE -B0) but perhaps a wrapper could be written or a "default" can be specified in a .conf file somewhere.
2. Submit a feature request (or better yet some commits) to compression software such as gzip and zstd to start the process of supporting these accelerator cards... gzip --device /dev/ibm_fpga0 -c file > output.gz
   1. May require an unaffiliated developer to push ahead as their agenda;
   2. May require hardware purchases on behalf of existing maintainers or remote access to a test bench system.

Thanks, eager to hear other thoughts and replies.

Tmanok