Problem with vmbr0, slow upload, PVE 8.2.2

gmce · Oct 2, 2024

Hail to everyone, i have a strange problem with a new proxmox ve 8.2.2 server.
The machine is a Supermicro server with 2 Intel 1Gbe.

Everything is fine but the upload speed, it is very slow. My wan is FTTC with 80Mbit down 40Mbit up.

Here is some info:

Linux pve8 6.8.4-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.4-2 (2024-04-10T17:36Z) x86_64 GNU/Linux
Output of ethotool eno1
Settings for eno1:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Auto-negotiation: on
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
MDI-X: off (auto)
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

Output of ethtool mvbr0
Settings for vmbr0:
Supported ports: [ ]
Supported link modes: Not reported
Supported pause frame use: No
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Unknown! (255)
Auto-negotiation: off
Port: Other
PHYAD: 0
Transceiver: internal
Link detected: yes

All the Windows base VM, with vrtio, shows ethernet card with 10000 speed

The upload to the external PBS is unusable.

Regards, Marco Mascheroni

gmce · Oct 2, 2024

Here some extra info:

Code:

lspci | grep -i 'ethernet'
03:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
04:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

I've tried to upload a single 54MB file to a external server via scp, it has taken more than 5 minutes instead of less than 30 seconds

Another info:
Every VM on proxmox host (2 linux, 2 Windows Server 2019, 1 Windows 10) have normal internet performances, as expected for my wan, 80 Mbit down 30/40 Mbit up

Only Proxmox VE 8.2.2 Host has poor internet performance.

_gabriel · Oct 2, 2024

gmce said:
The upload to the external PBS is unusable.

have you checked with iperf3 to the external PBS ?

gmce · Oct 2, 2024

Here the result between Proxmox host and Pbs on the cloud

[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 758 KBytes 6.21 Mbits/sec 37 9.90 KBytes
[ 5] 1.00-2.00 sec 636 KBytes 5.21 Mbits/sec 7 7.07 KBytes
[ 5] 2.00-3.00 sec 509 KBytes 4.17 Mbits/sec 6 14.1 KBytes
[ 5] 3.00-4.00 sec 255 KBytes 2.08 Mbits/sec 13 7.07 KBytes
[ 5] 4.00-5.00 sec 382 KBytes 3.13 Mbits/sec 8 8.48 KBytes
[ 5] 5.00-6.00 sec 382 KBytes 3.13 Mbits/sec 9 8.48 KBytes
[ 5] 6.00-7.00 sec 382 KBytes 3.13 Mbits/sec 8 9.90 KBytes
[ 5] 7.00-8.00 sec 382 KBytes 3.13 Mbits/sec 6 12.7 KBytes
[ 5] 8.00-9.00 sec 382 KBytes 3.13 Mbits/sec 7 9.90 KBytes
[ 5] 9.00-10.00 sec 382 KBytes 3.13 Mbits/sec 10 5.66 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 4.34 MBytes 3.64 Mbits/sec 111 sender
[ 5] 0.00-10.04 sec 4.17 MBytes 3.49 Mbits/sec receiver

This instead is proxmox in my office (with slower wan connection 42M/20M) to the same PBS:
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 2.20 MBytes 18.4 Mbits/sec 2 66.1 KBytes
[ 5] 1.00-2.00 sec 1.98 MBytes 16.6 Mbits/sec 0 85.8 KBytes
[ 5] 2.00-3.00 sec 1.73 MBytes 14.5 Mbits/sec 1 77.3 KBytes
[ 5] 3.00-4.00 sec 2.04 MBytes 17.1 Mbits/sec 1 70.3 KBytes
[ 5] 4.00-5.00 sec 1.73 MBytes 14.5 Mbits/sec 0 87.2 KBytes
[ 5] 5.00-6.00 sec 1.98 MBytes 16.6 Mbits/sec 1 78.8 KBytes
[ 5] 6.00-7.00 sec 1.73 MBytes 14.5 Mbits/sec 1 67.5 KBytes
[ 5] 7.00-8.00 sec 1.98 MBytes 16.6 Mbits/sec 1 59.1 KBytes
[ 5] 8.00-9.00 sec 1.98 MBytes 16.6 Mbits/sec 0 78.8 KBytes
[ 5] 9.00-10.00 sec 1.73 MBytes 14.5 Mbits/sec 1 70.3 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 19.1 MBytes 16.0 Mbits/sec 8 sender
[ 5] 0.00-10.03 sec 18.6 MBytes 15.6 Mbits/sec receiver

_gabriel · Oct 2, 2024

so, what about iperf3 upload speed from your slow PVE host to another host ?

gmce · Oct 3, 2024

This is a ubuntu server hosted in germany

[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 516 KBytes 4.23 Mbits/sec 25 5.66 KBytes
[ 5] 1.00-2.00 sec 318 KBytes 2.61 Mbits/sec 11 5.66 KBytes
[ 5] 2.00-3.00 sec 255 KBytes 2.08 Mbits/sec 9 9.90 KBytes
[ 5] 3.00-4.00 sec 318 KBytes 2.61 Mbits/sec 9 7.07 KBytes
[ 5] 4.00-5.00 sec 382 KBytes 3.13 Mbits/sec 9 7.07 KBytes
[ 5] 5.00-6.00 sec 382 KBytes 3.13 Mbits/sec 6 8.48 KBytes
[ 5] 6.00-7.00 sec 318 KBytes 2.61 Mbits/sec 8 9.90 KBytes
[ 5] 7.00-8.00 sec 318 KBytes 2.61 Mbits/sec 10 7.07 KBytes
[ 5] 8.00-9.00 sec 445 KBytes 3.65 Mbits/sec 9 5.66 KBytes
[ 5] 9.00-10.00 sec 318 KBytes 2.61 Mbits/sec 11 8.48 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 3.49 MBytes 2.92 Mbits/sec 107 sender
[ 5] 0.00-10.04 sec 3.32 MBytes 2.78 Mbits/sec receiver

_gabriel · Oct 3, 2024

what about iperf3 within guests ?

gmce · Oct 3, 2024

Hallo Grabriel.
This iperf3 results between host and linux client:

Code:

[  5] local 192.168.0.31 port 45930 connected to 192.168.0.32 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   344 MBytes  2.88 Gbits/sec   65   1.44 MBytes     
[  5]   1.00-2.00   sec   344 MBytes  2.88 Gbits/sec    0   1.61 MBytes     
[  5]   2.00-3.00   sec   342 MBytes  2.87 Gbits/sec    3   1.24 MBytes     
[  5]   3.00-4.00   sec   345 MBytes  2.89 Gbits/sec    0   1.43 MBytes     
[  5]   4.00-5.00   sec   344 MBytes  2.88 Gbits/sec    0   1.60 MBytes     
[  5]   5.00-6.00   sec   339 MBytes  2.84 Gbits/sec    6   1.23 MBytes     
[  5]   6.00-7.00   sec   338 MBytes  2.83 Gbits/sec    0   1.42 MBytes     
[  5]   7.00-8.00   sec   340 MBytes  2.85 Gbits/sec    0   1.58 MBytes     
[  5]   8.00-9.00   sec   345 MBytes  2.89 Gbits/sec    6   1.23 MBytes     
[  5]   9.00-10.00  sec   344 MBytes  2.88 Gbits/sec    0   1.42 MBytes     
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.34 GBytes  2.87 Gbits/sec   80             sender
[  5]   0.00-10.00  sec  3.34 GBytes  2.87 Gbits/sec                  receiver

This is the result of the proxmox-client-benchmark between host and the internal PBS:

Code:

Uploaded 250 chunks in 5 seconds.
Time per request: 20366 microseconds.
TLS speed: 205.94 MB/s
SHA256 speed: 1689.19 MB/s
Compression speed: 551.53 MB/s
Decompress speed: 809.44 MB/s
AES256/GCM speed: 1948.72 MB/s
Verify speed: 558.46 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 205.94 MB/s (17%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 1689.19 MB/s (84%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 551.53 MB/s (73%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 809.44 MB/s (68%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 558.46 MB/s (74%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1948.72 MB/s (53%) │
└───────────────────────────────────┴────────────────────┘

This is the result of the proxmox-client-benchmark between host and the external PBS:

Code:

Uploaded 10 chunks in 127 seconds.
Time per request: 12790602 microseconds.
TLS speed: 0.33 MB/s
SHA256 speed: 1677.26 MB/s
Compression speed: 567.55 MB/s
Decompress speed: 793.59 MB/s
AES256/GCM speed: 1958.45 MB/s
Verify speed: 543.59 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 0.33 MB/s (0%)     │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 1677.26 MB/s (83%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 567.55 MB/s (75%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 793.59 MB/s (66%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 543.59 MB/s (72%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 1958.45 MB/s (54%) │
└───────────────────────────────────┴────────────────────┘

Some additional info on the hardware:

CPU(s) 40 x Intel(R) Xeon(R) Silver 4416+ (1 Socket)

Kernel Version Linux 6.8.4-2-pve (2024-04-10T17:36Z)

Boot Mode EFI

Manager Version pve-manager/8.2.2/9355359cd7afbae4

_gabriel · Oct 3, 2024

please put results into CODE tags.
it's already hard to understand your problem.

I mean guest to your external PBS !

gmce · Oct 3, 2024

VM guest to external pbs:

Code:

[  5] local 192.168.0.32 port 60680 connected to 104.152.49.123 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   486 KBytes  3.98 Mbits/sec   16   8.48 KBytes       
[  5]   1.00-2.00   sec   255 KBytes  2.08 Mbits/sec   11   9.90 KBytes       
[  5]   2.00-3.00   sec   318 KBytes  2.61 Mbits/sec    7   7.07 KBytes       
[  5]   3.00-4.00   sec   191 KBytes  1.56 Mbits/sec    9   5.66 KBytes       
[  5]   4.00-5.00   sec   191 KBytes  1.56 Mbits/sec    6   8.48 KBytes       
[  5]   5.00-6.00   sec   255 KBytes  2.09 Mbits/sec    9   9.90 KBytes       
[  5]   6.00-7.00   sec   318 KBytes  2.61 Mbits/sec    8   12.7 KBytes       
[  5]   7.00-8.00   sec   255 KBytes  2.09 Mbits/sec    9   7.07 KBytes       
[  5]   8.00-9.00   sec   255 KBytes  2.09 Mbits/sec    7   5.66 KBytes       
[  5]   9.00-10.00  sec   255 KBytes  2.08 Mbits/sec    9   4.24 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.71 MBytes  2.28 Mbits/sec   91             sender
[  5]   0.00-10.02  sec  2.58 MBytes  2.16 Mbits/sec                  receiver

_gabriel · Oct 4, 2024

so there is throttle between your host and your external PBS,
it's out of scope of Proxmox.

ucholak · Oct 4, 2024

gmce said:
Only Proxmox VE 8.2.2 Host has poor internet performance.

check MTUs

Search

Search

Problem with vmbr0, slow upload, PVE 8.2.2

gmce

New Member

gmce

New Member

_gabriel

Renowned Member

gmce

New Member

_gabriel

Renowned Member

gmce

New Member

_gabriel

Renowned Member

gmce

New Member

_gabriel

Renowned Member

gmce

New Member

_gabriel

Renowned Member

ucholak

Member