Slow upload speed from node to node

daves_nt_here · 2026-01-27T16:40:00+0100

So, I have two Dell power edge R720's, both specked out identically.
PVE-1 is running 7.4-16
PVE-2 is running 9.1.4

I'm moving the backups for the VMs from PVE1 to PVE2 using SSH rsync in preparation of wiping out PV-1 for updates. I'm going to recreate the LXC's from scratch on PVE-2 so I only need the VMs.

A 60 gig transfer is taking almost 5 hours.
Started doing some speed tests this morning and I've discovered that uploading from PVE-1 to PVE-2 has a bottleneck.

I've done some testing using a 5 gig file between the two machines as well as using my desktop computer and filezilla. From what I found,
uploading and downloading <> PVE1 runs at full speed both ways.
Downloading from PVE2 goes at full speed but uploading a 5 gig to PVE2 starts out at full speed (110 MiB/s) but then around 40% to 50% it slows down to about 20 MiB/s.

I've tried changing out network cables and switches.
My Dell has a 4-port nic on the back so I also tried changing ports and reconfiguring Proxmox to the new interface. That didn't change anything either.

The only difference between the two machines is the Proxmox version.

The only clue that is popping up is from dmesg on PVE2. It keep looping this message as soon as the speed drops:

Code:

[  316.544317] tg3 0000:02:00.0 nic2: Link is down
[  316.877604] vmbr3: port 1(nic2) entered disabled state
[  319.563731] tg3 0000:02:00.0 nic2: Link is up at 1000 Mbps, full duplex
[  319.563780] vmbr3: port 1(nic2) entered blocking state
[  319.564714] tg3 0000:02:00.0 nic2: Flow control is on for TX and on for RX
[  319.565651] vmbr3: port 1(nic2) entered forwarding state
[  319.566578] tg3 0000:02:00.0 nic2: EEE is enabled

Code:

root@pve:~# pveversion -v
proxmox-ve: 7.4-1 (running kernel: 5.15.111-1-pve)

root@pve2:~# pveversion -v
proxmox-ve: 9.1.0 (running kernel: 6.17.4-2-pve)

Code:

root@pve:~# iperf3 -c 192.168.40.35
Connecting to host 192.168.40.35, port 5201
[  5] local 192.168.40.15 port 45924 connected to 192.168.40.35 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   114 MBytes   956 Mbits/sec    0    416 KBytes
[  5]   1.00-2.00   sec   112 MBytes   940 Mbits/sec    0    416 KBytes
[  5]   2.00-3.00   sec   113 MBytes   946 Mbits/sec    0    491 KBytes
[  5]   3.00-4.00   sec   112 MBytes   939 Mbits/sec    0    491 KBytes
[  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec    0    491 KBytes
[  5]   5.00-6.00   sec   112 MBytes   944 Mbits/sec    0    518 KBytes
[  5]   6.00-7.00   sec   113 MBytes   948 Mbits/sec    0    518 KBytes
[  5]   7.00-8.00   sec   112 MBytes   937 Mbits/sec    0    518 KBytes
[  5]   8.00-9.00   sec   113 MBytes   946 Mbits/sec    0    518 KBytes
[  5]   9.00-10.00  sec   112 MBytes   938 Mbits/sec    0    518 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   944 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec                  receiver


root@pve2:~# iperf3 -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 192.168.40.15, port 45912
[  5] local 192.168.40.35 port 5201 connected to 192.168.40.15 port 45924
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   112 MBytes   939 Mbits/sec
[  5]   1.00-2.00   sec   112 MBytes   942 Mbits/sec
[  5]   2.00-3.00   sec   112 MBytes   942 Mbits/sec
[  5]   3.00-4.00   sec   112 MBytes   941 Mbits/sec
[  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec
[  5]   5.00-6.00   sec   112 MBytes   942 Mbits/sec
[  5]   6.00-7.00   sec   112 MBytes   942 Mbits/sec
[  5]   7.00-8.00   sec   112 MBytes   941 Mbits/sec
[  5]   8.00-9.00   sec   112 MBytes   942 Mbits/sec
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec
[  5]  10.00-10.00  sec   256 KBytes  1.11 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec                  receiver

Code:

root@pve:~# iperf3 -c 192.168.40.35 -R
Connecting to host 192.168.40.35, port 5201
Reverse mode, remote host 192.168.40.35 is sending
[  5] local 192.168.40.15 port 60874 connected to 192.168.40.35 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   112 MBytes   942 Mbits/sec
[  5]   1.00-2.00   sec   112 MBytes   941 Mbits/sec
[  5]   2.00-3.00   sec   112 MBytes   941 Mbits/sec
[  5]   3.00-4.00   sec   112 MBytes   942 Mbits/sec
[  5]   4.00-5.00   sec   112 MBytes   942 Mbits/sec
[  5]   5.00-6.00   sec   112 MBytes   941 Mbits/sec
[  5]   6.00-7.00   sec   112 MBytes   940 Mbits/sec
[  5]   7.00-8.00   sec   112 MBytes   941 Mbits/sec
[  5]   8.00-9.00   sec   112 MBytes   942 Mbits/sec
[  5]   9.00-10.00  sec   112 MBytes   942 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec                  receiver

Server listening on 5201 (test #2)
-----------------------------------------------------------
Accepted connection from 192.168.40.15, port 60858
[  5] local 192.168.40.35 port 5201 connected to 192.168.40.15 port 60874
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   114 MBytes   957 Mbits/sec    0    365 KBytes
[  5]   1.00-2.00   sec   112 MBytes   943 Mbits/sec    0    416 KBytes
[  5]   2.00-3.00   sec   112 MBytes   943 Mbits/sec    0    464 KBytes
[  5]   3.00-4.00   sec   112 MBytes   938 Mbits/sec    0    491 KBytes
[  5]   4.00-5.00   sec   113 MBytes   948 Mbits/sec    0    491 KBytes
[  5]   5.00-6.00   sec   112 MBytes   940 Mbits/sec    0    491 KBytes
[  5]   6.00-7.00   sec   112 MBytes   937 Mbits/sec    0    516 KBytes
[  5]   7.00-8.00   sec   112 MBytes   943 Mbits/sec    0    546 KBytes
[  5]   8.00-9.00   sec   112 MBytes   943 Mbits/sec    0    546 KBytes
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec    0    546 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec    0            sender

Code:

root@pve:~# ethtool eno4
Settings for eno4:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Full
        Link partner advertised pause frame use: Symmetric
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 2
        Transceiver: internal
        MDI-X: off
        Supports Wake-on: g
        Wake-on: d
        Current message level: 0x000000ff (255)
                               drv probe link timer ifdown ifup rx_err tx_err
        Link detected: yes


root@pve2:~# ethtool nic2
Settings for nic2:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Half 1000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Full
        Link partner advertised pause frame use: Symmetric
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 2
        Transceiver: internal
        MDI-X: off
        Supports Wake-on: g
        Wake-on: d
        Current message level: 0x000000ff (255)
                               drv probe link timer ifdown ifup rx_err tx_err
        Link detected: yes

bbgeek17 · 2026-01-27T16:46:43+0100

daves_nt_here said:
The only clue that is popping up is from dmesg on PVE2. It keep looping this message as soon as the speed drops:

You have looping messages that the link goes down and up? That will certainly reset the sessions and slow things down.
At a high level it sounds like a hardware issue, perhaps a bug in NIC firmware, or something similar.
If this is a one time even - grab a USB drive and transfer the data that way.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

PwrBank · 2026-01-27T18:05:04+0100

bbgeek17 said:
You have looping messages that the link goes down and up? That will certainly reset the sessions and slow things down.
At a high level it sounds like a hardware issue, perhaps a bug in NIC firmware, or something similar.
If this is a one time even - grab a USB drive and transfer the data that way.

Maybe the hosts are physically plugged into each other with the same bridge the management interface is using?

That would certainly cause a loop.

Search

Search

Slow upload speed from node to node

daves_nt_here

Member

bbgeek17

Distinguished Member

PwrBank

Active Member

We value your privacy