One nic card fails to transfer data?

[{c}guy_123]

I'm running Proxmox 8.03 and it's been working great for a long time now. Last week all of a sudden my backup failed, upon researching it, the 10GB nic card which connects that server to my nas (on another machine) wouldn't transfer any data. I tested with iperf3 and here's the results:

Connecting to host 192.168.0.108, port 5201
[  5] local 192.168.0.27 port 60862 connected to 192.168.0.108 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   323 KBytes  2.65 Mbits/sec    3   8.74 KBytes
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    1   8.74 KBytes
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0   8.74 KBytes
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    1   8.74 KBytes
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    0   8.74 KBytes
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    0   8.74 KBytes
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    1   8.74 KBytes
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   8.74 KBytes
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0   8.74 KBytes
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   8.74 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   323 KBytes   265 Kbits/sec    6             sender
[  5]   0.00-10.04  sec  0.00 Bytes  0.00 bits/sec                  receiver

So I replaced the nic card with a new one, but the new one is doing the exact same thing? If I disable that nic and let everything run over the 1G management interface nic, all works as expected.

I'm at a loss to figure out where to look next? NFS won't mount my backup share, but it's nic related issue ( I think ) because it works if I disable that nic.

I've done no updates, or anything - it just quit working one day.

I should mention that I switched network ports on my switch, and that the green lights indicating a connection are lit. Also, I can't find any networking errors in the log. Ping also works as expected without issue.

Edit: Some more troubleshooting... I changed the nic into a different pci slot - same issue
I tried systemctl restart networking - same issue
I ran iper3 with --udp and that did work, so it seems this is a tcp/ip issue...

I ran lspci -nnvvv and then grep | ixgbe and here's the info on that:

root@pve:~# dmesg | grep ixgbe [ 3.117322] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver [ 3.117329] ixgbe: Copyright (c) 1999-2016 Intel Corporation. [ 3.403601] ixgbe 0000:0a:00.0: Multiqueue Enabled: Rx Queue count = 16, Tx Queue count = 16 XDP Queue count = 0 [ 3.488829] ixgbe 0000:0a:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x4 link at 0000:00:03.2 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link) [ 3.513417] ixgbe 0000:0a:00.0: MAC: 3, PHY: 0, PBA No: 000000-000 [ 3.513423] ixgbe 0000:0a:00.0: 98:b7:85:00:8c:16 [ 3.664460] ixgbe 0000:0a:00.0: Intel(R) 10 Gigabit Network Connection [ 3.665989] ixgbe 0000:0a:00.0 enp10s0: renamed from eth0 [ 11.022934] ixgbe 0000:0a:00.0: registered PHC device on enp10s0 [ 16.093486] ixgbe 0000:0a:00.0 enp10s0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

I should also mention that I have another proxmox server that connects to my nas server (truenas scale) without issue at full 10GB. so I can't see how this would be a Truenas issue.

I'm stumped!

Help Please!

Thanks in advance.

 

[{c}guy_123]

Anyone? Thoughts? Not sure what to check next...