Hallo,
EDIT: Thought I was in a German speaking forum. Now translated into English.
Status Quo
On my Minisforum MS-01 (Intel 13900H, 96GB RAM), I run Proxmox with a virtualized router. On the router, only the basic firewalling rules are active, no IDS / IPS or any additional filters / proxies.
Both, the ISP uplink as well as my switch are fully 10G. My virtualized router has currently 8 cores and 8GB memory assigned (also tested with higher resources) without success.
I tried independent OPNsense (what I had used so far) and VyOS.
My regular setup is having a VLAN aware bridge (vmbr1) and then assigning for each VLAN a virtual interface. But to test out, I also passed the vmbr1 as a trunk and let the router handle the VLAN with sub-interfaces (eth1.10, ...).
This is a sketch of my network setup:
Tests
I did several tests with Iper3. I wanted to test the uplink to my ISP from the router, as well ass from the VMs. Later I started testing the bandwidth between the VMs. Short, if the router has to do NATting at most, I get my desired speed. Which confirms, that my switch as well as the ISP uplink is fully functional and delivers the contracted bandwith.
1. Proxmox Node --> ISP (9.3 Gbps)
2. PC --> PC (9.4 Gbps)
3. VM1 --> LXC1 / VM2 --> LXC2 (48-50 Gbps)
4. Router eth1 --> VM1 (48-50 Gbps)
---
5. VM1 --> VM2 / VM1 --> PC / VM1 --> ISP (6-7 Gbps)
But as soon as the connections are across VLANs, the bandwith drops. Since my Proxmox node itself is also in a VLAN and the connection to the ISP is excellent, I'm asking what it is. Then according to the list above, VM to ISP is not.
Root causes
I tried to rule out all the possible causes, that's why I played with two router OS, different resource settings, ...
All the recommended tuning parameters (Internet, ChatGPT, ...) such as playing with hardware-offloading, multiqueue on the bridge, have worsened the result (~5Gbps).
Just observing the general CPU usage doesn't reveal any hardware limitations. The entire Proxmox node as well as single VMs do not exceed 5-15%.
My last guess is, that my machines utilizes just one core or cores, which are efficiency-cores, with not enough power. Would in this case, the utilization in a VM still display 10%?
Do you have experience with such a home lab setup? Any suggestions, what I could test next?
Thank you a lot!
Cheers Danny
EDIT: Thought I was in a German speaking forum. Now translated into English.
Status Quo
On my Minisforum MS-01 (Intel 13900H, 96GB RAM), I run Proxmox with a virtualized router. On the router, only the basic firewalling rules are active, no IDS / IPS or any additional filters / proxies.
Both, the ISP uplink as well as my switch are fully 10G. My virtualized router has currently 8 cores and 8GB memory assigned (also tested with higher resources) without success.
I tried independent OPNsense (what I had used so far) and VyOS.
My regular setup is having a VLAN aware bridge (vmbr1) and then assigning for each VLAN a virtual interface. But to test out, I also passed the vmbr1 as a trunk and let the router handle the VLAN with sub-interfaces (eth1.10, ...).
This is a sketch of my network setup:
Tests
I did several tests with Iper3. I wanted to test the uplink to my ISP from the router, as well ass from the VMs. Later I started testing the bandwidth between the VMs. Short, if the router has to do NATting at most, I get my desired speed. Which confirms, that my switch as well as the ISP uplink is fully functional and delivers the contracted bandwith.
1. Proxmox Node --> ISP (9.3 Gbps)
2. PC --> PC (9.4 Gbps)
3. VM1 --> LXC1 / VM2 --> LXC2 (48-50 Gbps)
4. Router eth1 --> VM1 (48-50 Gbps)
---
5. VM1 --> VM2 / VM1 --> PC / VM1 --> ISP (6-7 Gbps)
But as soon as the connections are across VLANs, the bandwith drops. Since my Proxmox node itself is also in a VLAN and the connection to the ISP is excellent, I'm asking what it is. Then according to the list above, VM to ISP is not.
Root causes
I tried to rule out all the possible causes, that's why I played with two router OS, different resource settings, ...
All the recommended tuning parameters (Internet, ChatGPT, ...) such as playing with hardware-offloading, multiqueue on the bridge, have worsened the result (~5Gbps).
Just observing the general CPU usage doesn't reveal any hardware limitations. The entire Proxmox node as well as single VMs do not exceed 5-15%.
My last guess is, that my machines utilizes just one core or cores, which are efficiency-cores, with not enough power. Would in this case, the utilization in a VM still display 10%?
Do you have experience with such a home lab setup? Any suggestions, what I could test next?
Thank you a lot!
Cheers Danny
Last edited: