Networking speed issues: 1Gb/s instead of 2.5Gb/s

[notarobot]

Hello everyone

I have a 4 ports QNAP 2.5Gbits NIC that I passthrough my Opnsense VM and through that this VM is able to get an Internet speed of around 2.2/2.3Gbp/s so this is OK.

On my Proxmox hypervisor i have another 2.5Gbp NIC that is used as a bridge for all LAN VMs. If i check with ethtool on Proxmox the card is indeed at a negotiated speed of 2500Mbps Full Duplex.
The problem is all the VM on this NIC can't get Internet speed above 1Gb/s (tested with speedtest and fast.com). Even if I force the speed with ethtool i got 1Gbp.
All the NIC on the VMs are VirtIO.

Am I missing something here ? How can I have 2,5gbe speed everywhere ?

Thanks for the help !

 

[Axl_fzetafs]

What is the OS of the VMs?

 

[notarobot]

What is the OS of the VMs?

Mainly Debian 11

 

[Axl_fzetafs]

Maybe I missed it, but what's the output of ethtool <device> within the OS guest?

BTW: what's the command you use to force the interface to a higher speed?

 

[bobmc]

what kind of cpu/chipset is your host running on? could be limitations on the pcie lanes

 

[notarobot]

Maybe I missed it, but what's the output of ethtool <device> within the OS guest?

ethtool ens18
Settings for ens18:
    Supported ports: [  ]
    Supported link modes:   Not reported
    Supported pause frame use: No
    Supports auto-negotiation: No
    Supported FEC modes: Not reported
    Advertised link modes:  Not reported
    Advertised pause frame use: No
    Advertised auto-negotiation: No
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: off
    Port: Other
    PHYAD: 0
    Transceiver: internal
netlink error: Operation not permitted
    Link detected: yes

BTW: what's the command you sue to force the interface to a higher speed?

ethtool -s ens18 speed 2500 duplex full

 

[notarobot]

what kind of cpu/chipset is your host running on? could be limitations on the pcie lanes

The NICs on the hypervisor are functionning correctly at 2.5GbE though ?

 

[Axl_fzetafs]

ethtool ens18
Settings for ens18:
    Supported ports: [  ]
    Supported link modes:   Not reported
    Supported pause frame use: No
    Supports auto-negotiation: No
    Supported FEC modes: Not reported
    Advertised link modes:  Not reported
    Advertised pause frame use: No
    Advertised auto-negotiation: No
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: off
    Port: Other
    PHYAD: 0
    Transceiver: internal
netlink error: Operation not permitted
    Link detected: yes

ethtool -s ens18 speed 2500 duplex full

Which driver are you using for the guest? Can you paste the conf file?

I found this

"Make txqueuelen of virbr0 10000 and mtu 9000 using ifconfigAnd in guest change mtu (jumbo packet) to 9000 which is under ethernet card device settings. It hits 10Gbit/s speeds with virto drivers of red hat on Win10."

on

https://www.reddit.com/r/Proxmox/comments/j2t3or/can_i_set_the_virtio_network_device_to_a_faster/

You might want to give it a try.

 

[notarobot]

Which driver are you using for the guest? Can you paste the conf file?

I didn't use a specific driver I just use the OS as is

 

[crc-error-79]

Have you tried to increase the mtu to 9000 as suggested?

 

[notarobot]

Have you tried to increase the mtu to 9000 as suggested?

If I try on the host It gives me

TASK ERROR: netdev net0: MTU '9000' is bigger than the bridge MTU '1500'

which is normal but I'm afraid to lose connectivity if I change it on the bridge

 

[crc-error-79]

If I try on the host It gives me

TASK ERROR: netdev net0: MTU '9000' is bigger than the bridge MTU '1500'

which is normal but I'm afraid to lose connectivity if I change it on the bridge

You have to modify that even in your host -> network otherwise it will not work

 

[notarobot]

I'm not willing to change MTU size to 9000 as all my network won't be complian and I'm not sure I need jumbo frames.
I wonder if the problem isn't elsewhere. On a Windows VM on this bridge the NIC says 10Gb/s, but Iperf or speedtest still show 1Gb/s speed

 

[Axl_fzetafs]

You may take a capture on the outgoing interface on the host for the two scearios, from the host when you get a throughout close to 10Gbps and and another capture with test from the host and have a look at what the captures differ by.

 

[bobmc]

The NICs on the hypervisor are functionning correctly at 2.5GbE though ?

try swapping the nics between slots and see what happens

 

[notarobot]

I ran a few test to see where there might be a problem:
Just to explain: my server contains a motherboard with
- Integrated 2.5Gbe NIC that I use for the bridge used by my LAN
- Another 4 port 2.5GbE NIC in a PCI slot dedicated to my Opnsense VM
- a 1GbE NIC dedicated to the bridge used by my DMZ.

If I do a speedtest from my OpnSense machine to the Net: I have 2.5GbE
if I do a speedtest between my OPnsense machine to the Proxmox server: i have 2.5GbE
If I do a speedtest from Proxmox server to the Internet: Only 1GbE speed