I have a bridge configured and working, it's just no faster than the 1gig NIC?

[RobGranger]

Even though the Bridge and Nic shows 10Gb, I only seem to get 1Gb speed? Am I just missing something? I wanted to remount my VM storage share to load over the Bridge thinking it would be faster and maybe safer knowing that my HP Z4 has the network interface bug.
Maybe I am chasing Unicorns. Any help in solving or suggestion to just abandon this would be appreciated
Thanks in advance,
Rob

Network Config on PVE

 cat interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eno1 inet manual
    post-up /usr/sbin/ethtool -K $IFACE tso off gro off gso off

iface eno2 inet manual

iface enp45s0 inet static
    address 10.20.0.16/24
    post-up /usr/sbin/ethtool -K $IFACE tso off gro off gso off

auto vmbr0
iface vmbr0 inet static
    address 10.20.0.50/24
    gateway 10.20.0.1
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0
    post-up /usr/sbin/ethtool -K $IFACE tso off gro off gso off


## this is the bridge between PVE and Truenas
auto vmbr1
iface vmbr1 inet static
    address 10.20.11.50/24
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    mtu 9000
#    post-up /usr/sbin/ethtool -K $IFACE tso off gro off gso off

Bridge Status on PVE

root@proxmox1:/etc/network# ethtool vmbr1
Settings for vmbr1:
    Supported ports: [  ]
    Supported link modes:   Not reported
    Supported pause frame use: No
    Supports auto-negotiation: No
    Supported FEC modes: Not reported
    Advertised link modes:  Not reported
    Advertised pause frame use: No
    Advertised auto-negotiation: No
    Advertised FEC modes: Not reported
    Speed: 10Mb/s **** 
    Duplex: Unknown! (255)
    Auto-negotiation: off
    Port: Other
    PHYAD: 0
    Transceiver: internal
    Link detected: yes

Network Status on Truenas VM

 ifconfig
vtnet0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>
    ether 0e:a7:cb:48:81:7a
    inet 10.20.0.15 netmask 0xffffff00 broadcast 10.20.0.255
    media: Ethernet autoselect (10Gbase-T <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
    groups: pflog
vtnet1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>
    ether b2:d1:04:7b:73:f7
    inet 10.20.11.15 netmask 0xffffff00 broadcast 10.20.11.255
    media: Ethernet autoselect (10Gbase-T <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>

Network Speed Test with Netcat

root@proxmox1:/etc/network# dd if=/dev/zero bs=16000 count=500000 | nc -v -n  10.20.11.15 12345
(UNKNOWN) [10.20.11.15] 12345 (?) open
500000+0 records in
500000+0 records out
8000000000 bytes (8.0 GB, 7.5 GiB) copied, 7.88025 s, 1.0 GB/s

 

[wbk]

Hi RobGranger,

Even though the Bridge and Nic shows 10Gb, I only seem to get 1Gb speed?

Is that a statement or a question?

You posted enough data, but missing a helping hand in what you want to show, I only noticed this bit:

Speed: 10Mb/s ****

Did you put the asterixes there to attract our attention, or is that part of the output? Anyway, with that configuration I would be surprised with 1 Gb/s speed as well :-D

 

[RobGranger]

Hi RobGranger,




Is that a statement or a question?

You posted enough data, but missing a helping hand in what you want to show, I only noticed this bit:



Did you put the asterixes there to attract our attention, or is that part of the output? Anyway, with that configuration I would be surprised with 1 Gb/s speed as well :-D

Yes, Asterix for bolding, but can't bold in the code block. I know it was a lot, those TrueNas guys broke me about not posting enough info. I think I have everything set for 10gig bridge, I don't know why it is only proving out to be 1gig.

 

[Dunuin]

8000000000 bytes (8.0 GB, 7.5 GiB) copied, 7.88025 s, 1.0 GB/s

Thats 1 GByte/s so 8 Gbit/s.

I would recommend benchmarking with a proper tool like "iperf3". This will also read/write to RAM to rule out the storage bottlenecking.

 

[RobGranger]

Thats 1 GByte/s so 8 Gbit/s.

I would recommend benchmarking with a proper tool like "iperf3". This will also read/write to RAM to rule out the storage bottlenecking.

Thanks, will try that. but netcat shouldn't be disk bound. Did I just misread the results. Od that I get the same either over the 1gig or 10gig side? maybe proxmox isn't sending the traffic over the network card after all.

 

[spirit]

bridge have unlimited speed, it's a pure in memory virtual construction. The cpu speed is the limiting factor.

 

[Dunuin]

maybe proxmox isn't sending the traffic over the network card after all.

There is no NIC attached to your vmbr1 and there is no NAT defined. If you didn't enable routing via sysctl.conf no physical NIC will be used and the throughput will be limited by the performance of your CPU.

 

[RobGranger]

Thanks, I think I had just been staring at the problem too long. I had been fighting the NIC issue in the Z4 workstation for days (hard NIC hangs).
Thanks for letting me bounce the info off you!