Rate limit strange

[Aleksej]

Hello.
I faced the problem that speed between VMs is very slow.
Almost all VMs have rate limit of 12 (approx 100mbps).
speedtest from any VM shows 100mbps,but iperf3 between them shows exactly 12mbps (2,6MB/s)

Maybe rate limit can have any other values?

Spoiler: pveversion --verbose

pveversion --verbose
proxmox-ve: 6.3-1 (running kernel: 5.4.78-2-pve)
pve-manager: 6.3-6 (running version: 6.3-6/2184247e)
pve-kernel-5.4: 6.3-8
pve-kernel-helper: 6.3-8
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.103-1-pve: 5.4.103-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.0-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.8
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-5
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.13-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-9
pve-cluster: 6.2-1
pve-container: 3.3-4
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-5
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-10
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1

 

[Stefan_R]

Which network interface are you using for your VMs? Can you post your VM config? (qm config <vmid>)

 

[Aleksej]

Which network interface are you using for your VMs? Can you post your VM config? (qm config <vmid>)

Spoiler: qm config

boot: order=scsi0
cores: 2
cpu: host,flags=+virt-ssbd;+amd-ssbd;+hv-tlbflush
cpuunits: 16384
memory: 4096
name: kovaleva
net0: virtio=0A:4D:90:8E:16:01,bridge=vmbr0,,rate=12
numa: 0
onboot: 1
ostype: win10
protection: 1
scsi0: ssd_b:vm-109-disk-1,cache=writeback,format=raw,size=50G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=098a91f8-8893-4fcf-87a7-3d9a0dfb7eb4
sockets: 1
vmgenid: 871b7721-0e83-4d44-b5cf-9d84fdcf9563

also tried with E1000 and RTL8139. situation is the same. firewall or multiqueue has no effect.
it is very strange that in\out traffic is limited in megabytes per sec but internal in megabits per sec...

with this config - speedtest - 91mbps, iperf (with local VM (or host - 25mbps)

without @rate=12": speedtest - 970mbps
iperf3 - 8.8Gbps

 

[Stefan_R]

When you say ratelimit, do you mean you are actively rate-limiting, or are you just referring to the fact that the connection is slower than it could be?

What kind of performance do you get between a VM and the PVE host?

 

[Aleksej]

What kind of performance do you get between a VM and the PVE host?

Between VM and host or different VM even on other host in cluster - 8.8-9.4Gbps

In fact i want to limit VM to 100mbps (approx 12MB/s?)

 

[Stefan_R]

If you have two rate-limited VMs that communicate with each other, the rate-limiting is applied twice. This might lead to such effects as you are seeing.

 

[Aleksej]

If you have two rate-limited VMs that communicate with each other, the rate-limiting is applied twice. This might lead to such effects as you are seeing.

No. "test" VM is not rate limited.
host actually too)

 

[Stefan_R]

There's not a lot of information to go on then, potentially try and narrow down where the issue occurs with iftop or similar performance metric tools. Maybe also check the other load on the machine, CPU and memory, etc...

 

[Aleksej]

without rate=20

Spoiler: iperf3

[ 4] local 6.6.4.2 port 33528 connected to 192.168.88.141 port 10251
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 1.45 GBytes 12.4 Gbits/sec 0 3.01 MBytes
[ 4] 1.00-2.00 sec 1.45 GBytes 12.5 Gbits/sec 0 3.01 MBytes
[ 4] 2.00-3.00 sec 1.48 GBytes 12.7 Gbits/sec 0 3.01 MBytes
[ 4] 3.00-4.00 sec 1.45 GBytes 12.4 Gbits/sec 0 3.01 MBytes
[ 4] 4.00-5.00 sec 1.40 GBytes 12.0 Gbits/sec 0 3.01 MBytes
[ 4] 5.00-6.00 sec 1.39 GBytes 11.9 Gbits/sec 0 3.01 MBytes
[ 4] 6.00-7.00 sec 1.39 GBytes 11.9 Gbits/sec 0 3.01 MBytes
[ 4] 7.00-8.00 sec 1.39 GBytes 11.9 Gbits/sec 0 3.01 MBytes
[ 4] 8.00-9.00 sec 1.43 GBytes 12.3 Gbits/sec 0 3.01 MBytes
[ 4] 9.00-10.00 sec 1.43 GBytes 12.3 Gbits/sec 0 3.01 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 14.2 GBytes 12.2 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 14.2 GBytes 12.2 Gbits/sec receiver

iperf Done.

Spoiler: iftop

6.6.4.2 => 192.168.88.141 11,3Gb 8,24Gb 2,06Gb
<= 9,67Mb 7,04Mb 1,76Mb

with rate=20

Spoiler: iperf3

[ 4] local 6.6.4.2 port 33540 connected to 192.168.88.141 port 10251
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 7.50 MBytes 62.9 Mbits/sec 521 102 KBytes
[ 4] 1.00-2.00 sec 5.41 MBytes 45.3 Mbits/sec 681 1.41 KBytes
[ 4] 2.00-3.00 sec 4.91 MBytes 41.2 Mbits/sec 306 1.41 KBytes
[ 4] 3.00-4.00 sec 4.91 MBytes 41.2 Mbits/sec 588 22.6 KBytes
[ 4] 4.00-5.00 sec 4.72 MBytes 39.6 Mbits/sec 431 5.66 KBytes
[ 4] 5.00-6.00 sec 6.09 MBytes 51.1 Mbits/sec 424 62.2 KBytes
[ 4] 6.00-7.00 sec 4.41 MBytes 37.0 Mbits/sec 422 62.2 KBytes
[ 4] 7.00-8.00 sec 5.53 MBytes 46.4 Mbits/sec 453 76.4 KBytes
[ 4] 8.00-9.00 sec 4.85 MBytes 40.7 Mbits/sec 345 109 KBytes
[ 4] 9.00-10.00 sec 6.71 MBytes 56.3 Mbits/sec 911 36.8 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 55.0 MBytes 46.2 Mbits/sec 5082 sender
[ 4] 0.00-10.00 sec 53.7 MBytes 45.1 Mbits/sec receiver

iperf Done.

Spoiler: iftop

6.6.4.2 => 192.168.88.141 62,0Mb 27,1Mb 6,78Mb
<= 546Kb 240Kb 59,9Kb

this tests was done one by one. (in approx 5 minutes...)

Real memory

223.95 MiB used / 728.79 MiB cached / 1.95 GiB total

Virtual memory

0 bytes used / 1.99 GiB total

CPU load averages

0.00 (1 min) 0.00 (5 mins) 0.00 (15 mins)

AMD Ryzen 9 3950X 16-Core Processor, 1 cores

Local disk space

29.52 GiB used / 56.94 GiB free / 86.46 GiB total

anything else?
tests was made with different NICs on VM. results are the same, so i decided not to post them separately.


Also i can add that speed limits with "rate=20"(or any number) causes speed limits in 2xrate mbits to any VM or host in a cluster.
To different cluster VMs or nodes speed is limited in Megabytes (rate=20 is the same as rate=190mbps)
Sorry for my bad english