10Gig card using Linux Bridge

[fahadshery]

Hi,

I have a 10Gig card with 2 NICs
I created the Linux Bond using these 2 NICs and then created a Linux bridge on top of this bond as shown below:

I am using this bond0 as a LAN in pfSense by creating the Linux Bridge (vmbr1).
pfSense recognises this interface (vmbr1) as 10 gig as shown below:

I created 2 VMs in this pfSense LAN and here are the iperf3 results:
Server:


and here is the client:


why am I not getting more than 1.95 GBytes?
What is wrong?

 

[_gabriel]

NICs are emulated in VMs, even with 100 Mbps physical nic card you'll get the same iperf speed because it's limited by the cpu.
you need to passthrough the physical card to bypass the cpu.

 

[Dunuin]

It also helps to work with jumbo frames. When working with bridges you are limited on how fast your CPU can process those packets. With a MTU of 9000 instead of 1500 your packets will be 6 times the size so you get 6 times the throughput. Without it my 10 Gbit NIC capped at 3-4 Gbit/s. With it it reaches its 10 Gbit/s.

 

[fahadshery]

It also helps to work with jumbo frames. When working with bridges you are limited on how fast your CPU can process those packets. With a MTU of 9000 instead of 1500 your packets will be 6 times the size so you get 6 times the throughput. Without it my 10 Gbit NIC capped at 3-4 Gbit/s. With it it reaches its 10 Gbit/s.

is this how you set it up?

 

[fahadshery]

NICs are emulated in VMs, even with 100 Mbps physical nic card you'll get the same iperf speed because it's limited by the cpu.
you need to passthrough the physical card to bypass the cpu.

This makes sense. I will try to pass 1 x NIC (enp68s0f0) port to the VM instead of creating a bond and see if that makes a difference. Thanks very much for your help

 

[Dunuin]

is this how you set it up?
View attachment 60975

Yes, but your Switch and the other clients also need to use an MTU of 9000 and support jumbo frames.

 

[fahadshery]

Yes, but your Switch and the other clients also need to use an MTU of 9000 and support jumbo frames.

I had problems with setting up the MTU of 9000. I enabled the jumbo frames on my Cisco switch and then pinged it using the following:

ping -s 8184 192.168.10.254 (-s is to specify the size of the data packaget to be sent)

the response shows that it can only carry the data packets of size 8192 (8184 + 8 bytes of ICMP header size). So this means that I could set the MTU size of 8184?


because if I increase just a byte and do the following:
ping -s 8185 192.168.10.254

I get the following error:

ping -s 8185 192.168.10.254
PING 192.168.10.254 (192.168.10.254): 8185 data bytes
ping: sendto: Message too long
ping: sendto: Message too long
Request timeout for icmp_seq 0
ping: sendto: Message too long
Request timeout for icmp_seq 1
^C
--- 192.168.10.254 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss