Offloading emulation with DPUs

[Tmanok]

Hi PVE Devs,

For the last three years, I've kept a keen eye on new high performance computing hardware and modern virtualization techniques. In recent years, previously inaccessible technologies such as DPUs have become more readily available, enabling even medium size businesses to acquire NICs that can run operating systems. We approach an Era where offloading functions like firewalls, compression, encryption, and emulation become far more realistic while simultaneously more important with higher throughput NICs.

Given DPUs with multiple ARM cores, 2-16GB of RAM and local storage, when will PVE begin implementing future virtualization tools and architectures that leverage running functions or the entire hypervisor on hardware such as a DPU? Has there been any exploration in this field? AWS is about a decade ahead in this regard, but the hardware to DIY is now available as commodity (e.g. NVIDIA Bluefield).

Thanks, I'm eager to hear exciting news on this front.


Tmanok

 

[spirit]

They are some interesting presentation from nvidia/cumulus at last netdev conf about evpn offload in dpu

https://www.youtube.com/watch?v=yNuDREQ2b1g

So, technically, it'll be really possible to push bridges in the dpu (and also routing).

Maybe firewall and other things should be possible too.

But I don't have it's they are already some standards, or if you need to implement specific code for each dpu model.

(I'm a mellanox/nvidia user, I think they are the most advanced currently).

Maybe I could give it a try in proxmox sdn. (but I really lack of time ^_^)

 

[Tmanok]

They are some interesting presentation from nvidia/cumulus at last netdev conf about evpn offload in dpu

https://www.youtube.com/watch?v=yNuDREQ2b1g

So, technically, it'll be really possible to push bridges in the dpu (and also routing).

Maybe firewall and other things should be possible too.

But I don't have it's they are already some standards, or if you need to implement specific code for each dpu model.

(I'm a mellanox/nvidia user, I think they are the most advanced currently).

Maybe I could give it a try in proxmox sdn. (but I really lack of time ^_^)

Hi Spirit,

Keep in mind some DPUs include the ability to run the entire hypervisor or a specialized sub-OS on them. Meaning you can run RouterOS directly on a Mikrotik DPU or Ubuntu with iptables on an NVidia Bluefield card.

So the options for PVE seem endless, not just for standards that enable kernel modules to offload specific functions like routing, but a PVE sub-OS that receives firewall configuration from the main host OS, or PVE running entirely on the NIC(s).

Cheers,


Tmanok

 

[spirit]

Hi Spirit,

Keep in mind some DPUs include the ability to run the entire hypervisor or a specialized sub-OS on them. Meaning you can run RouterOS directly on a Mikrotik DPU or Ubuntu with iptables on an NVidia Bluefield card.

So the options for PVE seem endless, not just for standards that enable kernel modules to offload specific functions like routing, but a PVE sub-OS that receives firewall configuration from the main host OS, or PVE running entirely on the NIC(s).

Cheers,


Tmanok

yes sure. I'm sure to see performance of netfilter for firewalling for example. Why I don't known is how you interact with this sub-os.
I'll try to buy 1 card this year and do some tests.

 

[Tmanok]

Hey Spirit,

I'm pretty sure if you did it yourself you'd install Ubuntu (supported OS) on the DPU and then modify PVE to send all of the networking rules for that card directly to the Ubuntu OS. In the case where it become supported by PVE it could look like: Create a bridge in the PVE GUI? PVE backend sends the config file to the Ubuntu DPU. Create a VM and tied it to the bridge? PVE sends that directly to the DPU. Create firewall rules in the PVE GUI? PVE will simply send all of the IPTables rules it automatically generates to the DPU.

When I became PVE Certified, the developer training my group described the ability to tie in a lot of integration into the IPTables rules, but I'm pretty sure that is costly. Perhaps a DPU could add enough performance to make it worthwhile. If it becomes API driven then other developers could likely tie into it nicely so that metrics, IDS, IPS, WAF, etc all become tied in directly to the DPU. OR like I mentioned earlier, PVE installs to the DPU and the baremetal is managed the other way around.

Cheers,


Tmanok

 

[auranext]

Hi,

we begin Bluefield2 DPU testing with proxmox.
The goal is to move the vxlan/EVPN controlplane into the DPU and to accelerate the packet processing with hw capabilities
First I'll try native HBN service , based on Cumulus network OS into a container
Then I will test a custom debian system and play with SDK (doca)
I am ready to discuss other vxlan/EVPN scenarios

 

[Nuke Bloodaxe]

Hmmmm... maybe I should break out my M1 Killer NIC again: https://en.wikipedia.org/wiki/Killer_NIC#:~:text=The card was first introduced in 2006.

 

[Tmanok]

We've also been in communication with ML SIEM makers that talk about port mirroring and rules they would generate in PVE to stop malicious attacks from things like Ransomware. Again this could be costly on the host CPU to process those restrictions and port mirroring. I think the best solution would have PVE offload those newly (and automatically) configured rules that protect the hypervisor and VM Workloads to the DPU.

Auranext, I'm keen to see some of your cumulus configurations if you are willing to share.

Cheers,

Tmanok