[SOLVED] Network Degradation between proxmox install

[jaimarti]

My Setup is HP z620 with dual e5-2670, 96G ram, Broadcom 57810S Dual Port 10Gb NIC RJ45 Proxmox VE 8.2.2 and Main desktop HP z620 with dual e5-2670,96 G ram, Broadcom 57810S Dual Port 10Gb NIC RJ45 Debian Bookworm.

When I was transferring some large files I noticed a Speed degradation. Upon investigating I narrowed it down to tons packet retransmissions. I have done all the usual troubleshooting changing cables and cards with no change. I then decided to Boot the Proxmox machine to a live CD, Linux Mint. No retransmission issue. I had recently changed out my Proxmox drive to an SSD, approximately 4 months ago. I reinstalled the old drive no retransmissions. So I believe this narrows the issue down to my current Proxmox install on the SSD.

In Iperf3 my typical transfer speeds are 9Gb/s+ on a 10 Gig interface with maybe a few retransmission as seen on my old install illustrated below. With my MTU set at 9000 I was barely getting 3Gb/s and changing it to 1500 Brought me up to 6Gb/s which is my current setting



Old Install

iperf3 -c xxx.xxx.xxx.xxx

Transfer Bitrate Retr Cwnd

[ 5] 0.00-1.00 sec 1.08 GBytes 9.24 Gbits/sec 0 1.59 MBytes
[ 5] 1.00-2.00 sec 1.08 GBytes 9.27 Gbits/sec 0 1.59 MBytes
[ 5] 2.00-3.00 sec 1.08 GBytes 9.25 Gbits/sec 0 1.67 MBytes
[ 5] 3.00-4.00 sec 1.08 GBytes 9.29 Gbits/sec 0 1.76 MBytes
[ 5] 4.00-5.00 sec 1.09 GBytes 9.33 Gbits/sec 0 1.76 MBytes
[ 5] 5.00-6.00 sec 1.09 GBytes 9.32 Gbits/sec 0 1.76 MBytes
[ 5] 6.00-7.00 sec 1.08 GBytes 9.31 Gbits/sec 0 1.76 MBytes
[ 5] 7.00-8.00 sec 1.08 GBytes 9.29 Gbits/sec 0 1.76 MBytes
[ 5] 8.00-9.00 sec 1.08 GBytes 9.26 Gbits/sec 0 1.76 MBytes
[ 5] 9.00-10.00 sec 1.08 GBytes 9.24 Gbits/sec 0 1.88 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.8 GBytes 9.28 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 10.8 GBytes 9.28 Gbits/sec receiver

iperf3 -c xxx.xxx.xxx.xxx -R

Transfer Bitrate
[ 5] 0.00-1.00 sec 1.08 GBytes 9.25 Gbits/sec
[ 5] 1.00-2.00 sec 1.08 GBytes 9.24 Gbits/sec
[ 5] 2.00-3.00 sec 1.06 GBytes 9.10 Gbits/sec
[ 5] 3.00-4.00 sec 1.07 GBytes 9.22 Gbits/sec
[ 5] 4.00-5.00 sec 1.07 GBytes 9.19 Gbits/sec
[ 5] 5.00-6.00 sec 1.07 GBytes 9.22 Gbits/sec
[ 5] 6.00-7.00 sec 1.07 GBytes 9.18 Gbits/sec
[ 5] 7.00-8.00 sec 1.07 GBytes 9.20 Gbits/sec
[ 5] 8.00-9.00 sec 1.07 GBytes 9.16 Gbits/sec
[ 5] 9.00-10.00 sec 1.07 GBytes 9.15 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.7 GBytes 9.19 Gbits/sec 22 sender
[ 5] 0.00-10.00 sec 10.7 GBytes 9.19 Gbits/sec receiver

Current install

iperf3 -c xxx.xxx.xxx.xxx

[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 715 MBytes 6.00 Gbits/sec 974 585 KBytes
[ 5] 1.00-2.00 sec 606 MBytes 5.09 Gbits/sec 101 612 KBytes
[ 5] 2.00-3.00 sec 580 MBytes 4.87 Gbits/sec 73 602 KBytes
[ 5] 3.00-4.00 sec 619 MBytes 5.19 Gbits/sec 298 612 KBytes
[ 5] 4.00-5.00 sec 639 MBytes 5.36 Gbits/sec 295 629 KBytes
[ 5] 5.00-6.00 sec 600 MBytes 5.03 Gbits/sec 336 464 KBytes
[ 5] 6.00-7.00 sec 585 MBytes 4.91 Gbits/sec 83 615 KBytes
[ 5] 7.00-8.00 sec 582 MBytes 4.89 Gbits/sec 65 601 KBytes
[ 5] 8.00-9.00 sec 566 MBytes 4.75 Gbits/sec 100 612 KBytes
[ 5] 9.00-10.00 sec 682 MBytes 5.73 Gbits/sec 945 561 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 6.03 GBytes 5.18 Gbits/sec 3270 sender
[ 5] 0.00-10.00 sec 6.03 GBytes 5.18 Gbits/sec receiver

iperf3 -c xxx.xxx.xxx.xxx -R

[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.06 GBytes 9.12 Gbits/sec
[ 5] 1.00-2.00 sec 1.05 GBytes 9.04 Gbits/sec
[ 5] 2.00-3.00 sec 1.06 GBytes 9.13 Gbits/sec
[ 5] 3.00-4.00 sec 1.06 GBytes 9.15 Gbits/sec
[ 5] 4.00-5.00 sec 1.07 GBytes 9.17 Gbits/sec
[ 5] 5.00-6.00 sec 1.07 GBytes 9.18 Gbits/sec
[ 5] 6.00-7.00 sec 1.08 GBytes 9.26 Gbits/sec
[ 5] 7.00-8.00 sec 1.05 GBytes 9.04 Gbits/sec
[ 5] 8.00-9.00 sec 1.08 GBytes 9.24 Gbits/sec
[ 5] 9.00-10.00 sec 1.07 GBytes 9.16 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.7 GBytes 9.16 Gbits/sec 56 sender
[ 5] 0.00-10.00 sec 10.7 GBytes 9.15 Gbits/sec receiver

I did try rolling back to the same kernel as the install with no issue and degradation was still there.
Linux 6.5.11-8-pve x86_64

uname -srm

Old install
Linux 6.5.11-8-pve x86_64
Current install
Linux 6.8.4-3-pve x86_64

when I did a stare and compare of the following commands

lspci -v

only difference
Old install
Flags: bus master, fast devsel, latency 0, IRQ 42
current install
Flags: bus master, fast devsel, latency 0, IRQ 86, NUMA node 0, IOMMU group 35

ethtool --driver ens5f0

no differance

ethtool ens5f0

no differance

ethtool vmbr0

no difference


I am not sure what else to check so I am reaching out for suggestions. I have been working on this in my spare time for a couple of months now.

 

[jaimarti]

Reverting to old backjup

 

[jaimarti]

I have Cloned the old image tested it, everthing worked fine no degradation. After I updated proxmox I now have the degradation again Biterate has gone from 9.33 Gbits/sec, 106 Retr, Cwnd 2.14 MBytes to 5.68 Gbites/sec and 9604 Retr Cwnd 561. Any sugestion as to how to resolve this is appreciated.

 

[_gabriel]

boot back with 6.5 kernel to confirm if a 6.8 Kernel issue with your conf.

 

[jaimarti]

Thanks for the response. I rolled back to 6.5 kernel and I am good again with 9.3 Gbits/sec 237 Retr and Cwnd 1.79. Is there a recomendation on next steps. Guidence is always appreciated.