linux bridge vs ovs Bridge

[Keeper of the Keys]

Thomas - thanks that is very cool.

hvisage - Yes I could create seperate vlans for everything but that would also make management way more complicated, port isolation/micro-segmentation would make my life a lot easier.

 

[jaminmc]

I just converted my test Proxmox running on my iMac mid 2011 that has pfSense running in a VM with virtIO network driver. On my Proxmox, I ran 'iperf -s' and in pfSense running in a console, I connected to the ip of Proxmox (Just as a client, not routing anything)

With Linux Bridge:

Accepted connection from 192.168.1.188, port 41433
[  5] local 192.168.1.147 port 5201 connected to 192.168.1.188 port 32115
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   344 MBytes  2.89 Gbits/sec
[  5]   1.00-2.00   sec   381 MBytes  3.20 Gbits/sec
[  5]   2.00-3.00   sec   358 MBytes  3.00 Gbits/sec
[  5]   3.00-4.00   sec   365 MBytes  3.06 Gbits/sec
[  5]   4.00-5.00   sec   405 MBytes  3.40 Gbits/sec
[  5]   5.00-6.00   sec   378 MBytes  3.17 Gbits/sec
[  5]   6.00-7.00   sec   353 MBytes  2.96 Gbits/sec
[  5]   7.00-8.00   sec   367 MBytes  3.08 Gbits/sec
[  5]   8.00-9.00   sec   362 MBytes  3.04 Gbits/sec
[  5]   9.00-10.00  sec   416 MBytes  3.49 Gbits/sec
[  5]  10.00-10.26  sec  88.7 MBytes  2.87 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.26  sec  3.73 GBytes  3.12 Gbits/sec                  receiver

and OpenVSwitch:

Accepted connection from 192.168.1.188, port 48286
[  5] local 192.168.1.147 port 5201 connected to 192.168.1.188 port 48287
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   396 MBytes  3.32 Gbits/sec
[  5]   1.00-2.00   sec   554 MBytes  4.65 Gbits/sec
[  5]   2.00-3.00   sec   478 MBytes  4.01 Gbits/sec
[  5]   3.00-4.00   sec   288 MBytes  2.42 Gbits/sec
[  5]   4.00-5.00   sec   289 MBytes  2.42 Gbits/sec
[  5]   5.00-6.00   sec   291 MBytes  2.44 Gbits/sec
[  5]   6.00-7.00   sec   455 MBytes  3.82 Gbits/sec
[  5]   7.00-8.00   sec   549 MBytes  4.60 Gbits/sec
[  5]   8.00-9.00   sec   562 MBytes  4.71 Gbits/sec
[  5]   9.00-10.00  sec   556 MBytes  4.67 Gbits/sec
[  5]  10.00-10.26  sec   145 MBytes  4.72 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.26  sec  4.46 GBytes  3.73 Gbits/sec                  receiver
-

So, I seem to be getting better throughput through the OpenVSwitch bridge.
CPU(s) 4 x Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz (1 Socket)
Kernel Version Linux 5.18.15-edge #1 SMP PREEMPT_DYNAMIC PVE Edge 5.18.15-1 (2022-07-30)
PVE Manager Version pve-manager/7.2-7/d0dd0e85

Obviously, this hardware isn't the best, as it doesn't seem to even support 10Gbps speed. But the Ethernet is 1Gbps, and both bridges are better than the NIC in the computer.


EDIT: Above is with PfSense configured as default. If I enable hardware acceleration in pfSense for the NIC's Which are the VirtIO, Linux bridge wins, and the bitrate is much higher!

Linux Bridge:

Accepted connection from 192.168.1.188, port 33349
[  5] local 192.168.1.147 port 5201 connected to 192.168.1.188 port 33350
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.75 GBytes  15.0 Gbits/sec
[  5]   1.00-2.00   sec  2.33 GBytes  20.0 Gbits/sec
[  5]   2.00-3.00   sec  2.33 GBytes  20.0 Gbits/sec
[  5]   3.00-4.00   sec  2.32 GBytes  19.9 Gbits/sec
[  5]   4.00-5.00   sec  2.31 GBytes  19.9 Gbits/sec
[  5]   5.00-6.00   sec  2.31 GBytes  19.8 Gbits/sec
[  5]   6.00-7.00   sec  2.30 GBytes  19.8 Gbits/sec
[  5]   7.00-8.00   sec  2.31 GBytes  19.8 Gbits/sec
[  5]   8.00-9.00   sec  2.27 GBytes  19.5 Gbits/sec
[  5]   9.00-10.00  sec  2.24 GBytes  19.3 Gbits/sec
[  5]  10.00-10.25  sec   586 MBytes  19.9 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.25  sec  23.0 GBytes  19.3 Gbits/sec                  receiver

Open vSwitch:

Accepted connection from 192.168.1.188, port 37474
[  5] local 192.168.1.147 port 5201 connected to 192.168.1.188 port 3281
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.32 GBytes  11.4 Gbits/sec
[  5]   1.00-2.00   sec  2.09 GBytes  18.0 Gbits/sec
[  5]   2.00-3.00   sec  2.09 GBytes  18.0 Gbits/sec
[  5]   3.00-4.00   sec  2.09 GBytes  18.0 Gbits/sec
[  5]   4.00-5.00   sec  2.10 GBytes  18.0 Gbits/sec
[  5]   5.00-6.00   sec  2.09 GBytes  18.0 Gbits/sec
[  5]   6.00-7.00   sec  2.10 GBytes  18.0 Gbits/sec
[  5]   7.00-8.00   sec  2.10 GBytes  18.1 Gbits/sec
[  5]   8.00-9.00   sec  2.09 GBytes  18.0 Gbits/sec
[  5]   9.00-10.00  sec  2.09 GBytes  18.0 Gbits/sec
[  5]  10.00-10.36  sec   724 MBytes  17.0 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.36  sec  20.9 GBytes  17.3 Gbits/sec                  receiver

So, now it does well over 10Gbits/sec

 

[cjangrist]

EDIT: Above is with PfSense configured as default. If I enable hardware acceleration in pfSense for the NIC's Which are the VirtIO, Linux bridge wins, and the bitrate is much higher!

I thought pfsense on proxmox guide says explicitly to disable hardware based nic acceleration?

 

[CharlieCortial]

I thought pfsense on proxmox guide says explicitly to disable hardware based nic acceleration?

Hello everyone,

I can see @cjangrist never got an answer to his questions, and I’m pretty curious about it too.

I’m pondering to use an OPNsense VM on Promox, using Linux bridges, and I wonder is the hardware acceleration is now okay to use, after seeing @jaminmc messages.
Or is the requirement specific to using OpenVSwitch and pfSense?

Thanks everyone, cheers!

 

[hvisage]

Hello everyone,

I can see @cjangrist never got an answer to his questions, and I’m pretty curious about it too.

I’m pondering to use an OPNsense VM on Promox, using Linux bridges, and I wonder is the hardware acceleration is now okay to use, after seeing @jaminmc messages.
Or is the requirement specific to using OpenVSwitch and pfSense?

Thanks everyone, cheers!

The last time *I* used pfSense/OPNsense, the reason you disabled the hardware acceleration and did not want to use the virtIO nics, was that DHCP/UDP had troubles with the lack of checksums added to the packets, and then certain parts would drop the packets, so the need was to stick with E1000 no-HW acceleration options.

That is typically the case to test/check first and take it from there

 

[CharlieCortial]

The last time *I* used pfSense/OPNsense, the reason you disabled the hardware acceleration and did not want to use the virtIO nics, was that DHCP/UDP had troubles with the lack of checksums added to the packets, and then certain parts would drop the packets, so the need was to stick with E1000 no-HW acceleration options.

That is typically the case to test/check first and take it from there

Okay, thank you very much for your answer. I'm quite the beginner, learning a lot, and every new word opens a door to another unknown subject, and sometimes it's quite overwhelming. But still, there's some fun involved.
Thanks again!

 

[ExitDust]

i have a small pc with 2 network adapters ne0 and ne1. ne0 is bridged to vmbr0 and ne1 is bridged to vmbr1.

ne0 is connected to a modem/router as 10.0.0.1.
ne1 is connected to a switch for the rest of my network

so i configure vmbr0 as 10.0.0.2/24 and gateway to 10.0.0.1 on proxmox. vmbr1 gets 192.168.0,2 and no gateway

all is well and proxmox can reach the internet fine.

now i install a vm for pfsense. i bind both vmbrs to the vm and i tell pfsense which one is for wan and which for lan. I configure the wan interface with a static ip of 10.0.0.3 and gateway 10.0.0.1. lan interface gets 192.168.0.1/24 and no gateway.

Now i found out that with a linux bridge, the pfsense cannot reach the router/modem on 10.0.0.1 so gets no internet. It can reach 10.0.0.2 (the proxmox host).

And here comes the kicker. When i remove the vmbr0 and remake it as a ovs bridge with the exact same specs, pfsense can reach the router/modem.

No idea why, but that was a good 5 hours of my life wasted figuring that out.