Eth rate limit -> unstable bandwidth

Sakis · May 28, 2015

I found that using rate limit at ethernet devices causes bandwidth to be unstable.

Here is a iperf example of a vm with no limits. The iperf server has a 1Gbit limit and the result is expected

Code:

[root@localhost ~]# iperf -c x.x.x.x -t 60 -i 1
------------------------------------------------------------
Client connecting to x.x.x.x, TCP port 5001
TCP window size: 19.3 KByte (default)
------------------------------------------------------------
[  3] local y.y.y.y port 35179 connected with x.x.x.x port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec   102 MBytes   858 Mbits/sec
[  3]  1.0- 2.0 sec   112 MBytes   941 Mbits/sec
[  3]  2.0- 3.0 sec   112 MBytes   936 Mbits/sec
[  3]  3.0- 4.0 sec  97.2 MBytes   816 Mbits/sec
[  3]  4.0- 5.0 sec   107 MBytes   899 Mbits/sec
[  3]  5.0- 6.0 sec   112 MBytes   935 Mbits/sec
[  3]  6.0- 7.0 sec   111 MBytes   931 Mbits/sec
[  3]  7.0- 8.0 sec   113 MBytes   947 Mbits/sec
[  3]  8.0- 9.0 sec   105 MBytes   880 Mbits/sec
[  3]  9.0-10.0 sec   112 MBytes   935 Mbits/sec
[  3] 10.0-11.0 sec   112 MBytes   935 Mbits/sec
[  3] 11.0-12.0 sec   112 MBytes   937 Mbits/sec
[  3] 12.0-13.0 sec   112 MBytes   940 Mbits/sec
[  3] 13.0-14.0 sec   111 MBytes   929 Mbits/sec
[  3] 14.0-15.0 sec   112 MBytes   943 Mbits/sec
[  3] 15.0-16.0 sec   111 MBytes   928 Mbits/sec
[  3] 16.0-17.0 sec   112 MBytes   940 Mbits/sec
[  3] 17.0-18.0 sec   112 MBytes   940 Mbits/sec
[  3] 18.0-19.0 sec   112 MBytes   936 Mbits/sec
[  3] 19.0-20.0 sec   112 MBytes   937 Mbits/sec
[  3] 20.0-21.0 sec   112 MBytes   941 Mbits/sec
[  3] 21.0-22.0 sec   111 MBytes   928 Mbits/sec
[  3] 22.0-23.0 sec   112 MBytes   940 Mbits/sec
[  3] 23.0-24.0 sec   110 MBytes   925 Mbits/sec
[  3] 24.0-25.0 sec   112 MBytes   935 Mbits/sec
[  3] 25.0-26.0 sec   111 MBytes   931 Mbits/sec
[  3] 26.0-27.0 sec   110 MBytes   923 Mbits/sec
[  3] 27.0-28.0 sec   111 MBytes   932 Mbits/sec
[  3] 28.0-29.0 sec   112 MBytes   935 Mbits/sec
[  3] 29.0-30.0 sec   110 MBytes   921 Mbits/sec
[  3] 30.0-31.0 sec   110 MBytes   926 Mbits/sec
[  3] 31.0-32.0 sec   111 MBytes   930 Mbits/sec
[  3] 32.0-33.0 sec   110 MBytes   920 Mbits/sec
[  3] 33.0-34.0 sec   112 MBytes   941 Mbits/sec
[  3] 34.0-35.0 sec   111 MBytes   928 Mbits/sec
[  3] 35.0-36.0 sec   112 MBytes   942 Mbits/sec
[  3] 36.0-37.0 sec   112 MBytes   940 Mbits/sec
[  3] 37.0-38.0 sec   112 MBytes   936 Mbits/sec
[  3] 38.0-39.0 sec   111 MBytes   928 Mbits/sec
[  3] 39.0-40.0 sec   111 MBytes   929 Mbits/sec
[  3] 40.0-41.0 sec   112 MBytes   943 Mbits/sec
[  3] 41.0-42.0 sec   111 MBytes   930 Mbits/sec
[  3] 42.0-43.0 sec   113 MBytes   947 Mbits/sec
[  3] 43.0-44.0 sec   112 MBytes   936 Mbits/sec
[  3] 44.0-45.0 sec   111 MBytes   930 Mbits/sec
[  3] 45.0-46.0 sec   107 MBytes   899 Mbits/sec
[  3] 46.0-47.0 sec   109 MBytes   918 Mbits/sec
[  3] 47.0-48.0 sec   111 MBytes   933 Mbits/sec
[  3] 48.0-49.0 sec   112 MBytes   935 Mbits/sec
[  3] 49.0-50.0 sec   112 MBytes   940 Mbits/sec
[  3] 50.0-51.0 sec   112 MBytes   938 Mbits/sec
[  3] 51.0-52.0 sec   112 MBytes   937 Mbits/sec
[  3] 52.0-53.0 sec   112 MBytes   943 Mbits/sec
[  3] 53.0-54.0 sec   111 MBytes   930 Mbits/sec
[  3] 54.0-55.0 sec   113 MBytes   948 Mbits/sec
[  3] 55.0-56.0 sec   111 MBytes   930 Mbits/sec
[  3] 56.0-57.0 sec   112 MBytes   943 Mbits/sec
[  3] 57.0-58.0 sec   112 MBytes   936 Mbits/sec
[  3] 58.0-59.0 sec   110 MBytes   924 Mbits/sec
[  3] 59.0-60.0 sec   112 MBytes   940 Mbits/sec
[  3]  0.0-60.0 sec  6.49 GBytes   929 Mbits/sec

I stop the vm and set rate=12 (close to 100Mbit/s), boot and check again (it works on the fly too, but i wanted to be sure cause i am not familiar the mechanism used to rate limit)

Here is the output now

Code:

[root@localhost ~]# iperf -c x.x.x.x -t 60 -i 1
------------------------------------------------------------
Client connecting to x.x.x.x, TCP port 5001
TCP window size: 19.3 KByte (default)
------------------------------------------------------------
[  3] local y.y.y.y port 57898 connected with x.x.x.x port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec   100 MBytes   839 Mbits/sec
[  3]  1.0- 2.0 sec  74.8 MBytes   627 Mbits/sec
[  3]  2.0- 3.0 sec  0.00 Bytes  0.00 bits/sec
[  3]  3.0- 4.0 sec  0.00 Bytes  0.00 bits/sec
[  3]  4.0- 5.0 sec  0.00 Bytes  0.00 bits/sec
[  3]  5.0- 6.0 sec  0.00 Bytes  0.00 bits/sec
[  3]  6.0- 7.0 sec  0.00 Bytes  0.00 bits/sec
[  3]  7.0- 8.0 sec  0.00 Bytes  0.00 bits/sec
[  3]  8.0- 9.0 sec  55.8 MBytes   468 Mbits/sec
[  3]  9.0-10.0 sec  66.5 MBytes   558 Mbits/sec
[  3] 10.0-11.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 11.0-12.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 12.0-13.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 13.0-14.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 14.0-15.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 15.0-16.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 16.0-17.0 sec  60.0 MBytes   503 Mbits/sec
[  3] 17.0-18.0 sec  73.4 MBytes   616 Mbits/sec
[  3] 18.0-19.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 19.0-20.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 20.0-21.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 21.0-22.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 22.0-23.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 23.0-24.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 24.0-25.0 sec  27.9 MBytes   234 Mbits/sec
[  3] 25.0-26.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 26.0-27.0 sec  25.2 MBytes   212 Mbits/sec
[  3] 27.0-28.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 28.0-29.0 sec  5.38 MBytes  45.1 Mbits/sec
[  3] 29.0-30.0 sec  62.8 MBytes   526 Mbits/sec
[  3] 30.0-31.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 31.0-32.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 32.0-33.0 sec  1.38 MBytes  11.5 Mbits/sec
[  3] 33.0-34.0 sec  65.8 MBytes   552 Mbits/sec
[  3] 34.0-35.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 35.0-36.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 36.0-37.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 37.0-38.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 38.0-39.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 39.0-40.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 40.0-41.0 sec  69.1 MBytes   580 Mbits/sec
[  3] 41.0-42.0 sec  73.4 MBytes   616 Mbits/sec
[  3] 42.0-43.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 43.0-44.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 44.0-45.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 45.0-46.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 46.0-47.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 47.0-48.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 48.0-49.0 sec  26.2 MBytes   220 Mbits/sec
[  3] 49.0-50.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 50.0-51.0 sec  31.2 MBytes   262 Mbits/sec
[  3] 51.0-52.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 52.0-53.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 53.0-54.0 sec  1.38 MBytes  11.5 Mbits/sec
[  3] 54.0-55.0 sec  67.9 MBytes   569 Mbits/sec
[  3] 55.0-56.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 56.0-57.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 57.0-58.0 sec  1.38 MBytes  11.5 Mbits/sec
[  3] 58.0-59.0 sec  67.2 MBytes   564 Mbits/sec
[  3] 59.0-60.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 60.0-61.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 61.0-62.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 62.0-63.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 63.0-64.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 64.0-65.0 sec  0.00 Bytes  0.00 bits/sec
[  3]  0.0-65.2 sec   957 MBytes   123 Mbits/sec

Checking with various cli bandwidth tools i can confirm that the bandwidth is not stable at all. Ofc changing it back to unlimited works again. The node used for iperf client test dont have any other machines running or doing any other bandwidth hungry process.

Is this behaviour expected? Does this effect real bandwidth?

pveversion

Code:

proxmox-ve-2.6.32: 3.4-150 (running kernel: 2.6.32-37-pve)
pve-manager: 3.4-3 (running version: 3.4-3/2fc72fee)
pve-kernel-2.6.32-32-pve: 2.6.32-136
pve-kernel-2.6.32-37-pve: 2.6.32-150
pve-kernel-2.6.32-29-pve: 2.6.32-126
pve-kernel-2.6.32-31-pve: 2.6.32-132
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-2
pve-cluster: 3.0-16
qemu-server: 3.4-3
pve-firmware: 1.1-4
libpve-common-perl: 3.0-24
libpve-access-control: 3.0-16
libpve-storage-perl: 3.0-32
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.2-10
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1

kvm conf

Code:

bootdisk: virtio0
cores: 2
cpu: qemu64
ide2: none,media=cdrom
memory: 4096
name: eth-test
net0: virtio=8E:19:A2:5E:24:B8,bridge=vmbr0,rate=12
numa: 0
ostype: l26
smbios1: uuid=e49feb90-75d5-4036-9693-1ceaa81c651b
sockets: 1
virtio0: pool:vm-150-disk-1,size=20G

udo · May 28, 2015

Hi,
never used bandwith limitation before... but an short test with kernel 3.10 shows the same behavior.

Udo

Sakis · May 29, 2015

Should we report a bug?

Sakis · Jun 3, 2015

Do anyone else have the same problem? Its easy to test.

spirit · Jun 3, 2015

The code which managed rate limit is in

/usr/share/perl5/PVE/Network.pm

Code:

sub setup_tc_rate_limit {
    my ($iface, $rate, $burst, $debug) = @_;


    system("/sbin/tc class del dev $iface parent 1: classid 1:1 >/dev/null 2>&1");
    system("/sbin/tc filter del dev $iface parent ffff: protocol ip prio 50 estimator 1sec 8sec >/dev/null 2>&1");
    system("/sbin/tc qdisc del dev $iface ingress >/dev/null 2>&1");
    system("/sbin/tc qdisc del dev $iface root >/dev/null 2>&1");


    return if !$rate;


    run_command("/sbin/tc qdisc add dev $iface handle ffff: ingress");


    # this does not work wit virtio - don't know why (setting "mtu 64kb" does not help)
    #run_command("/sbin/tc filter add dev $iface parent ffff: protocol ip prio 50 u32 match ip src 0.0.0.0/0 police rate ${rate}bps burst ${burst}b drop flowid :1");
    # so we use avrate instead
    run_command("/sbin/tc filter add dev $iface parent ffff: " .
                "protocol ip prio 50 estimator 1sec 8sec " .
                "u32 match ip src 0.0.0.0/0 police avrate ${rate}bps drop flowid :1");


    # tbf does not work for unknown reason
    #$TC qdisc add dev $DEV root tbf rate $RATE latency 100ms burst $BURST
    # so we use htb instead
    run_command("/sbin/tc qdisc add dev $iface root handle 1: htb default 1");
    run_command("/sbin/tc class add dev $iface parent 1: classid 1:1 " .
                "htb rate ${rate}bps burst ${burst}b");


    if ($debug) {
        print "DEBUG tc settings\n";
        system("/sbin/tc qdisc ls dev $iface");
        system("/sbin/tc class ls dev $iface");
        system("/sbin/tc filter ls dev $iface parent ffff:");
    }
}

Maybe somebody expert with tc can improve that ?

spirit · Jun 3, 2015

They are also good infos in libvirt code, in src/util/virnetdevbandwidth.c

Sakis · Jun 5, 2015

Should we report a bug?

Sakis · Jun 8, 2015

I believe this potential bug its essential. Its strange that no one cares.

spirit · Jun 8, 2015

Sakis said:
I believe this potential bug its essential. Its strange that no one cares.

you can report to bugzilla.proxmox.com.

I'll try to check this in coming weeks

Sakis · Jun 9, 2015

I report it. Thank you.
https://bugzilla.proxmox.com/show_bug.cgi?id=639

giner · Jun 14, 2015

Hi Sakis,

What if you try another direction?

Stanislav

Sakis · Jun 15, 2015

I believe the problem is the same weather traffic is incoming or outgoing. I will update with logs to confirm it.

giner · Jun 15, 2015

Sakis said:
I believe the problem is the same weather traffic is incoming or outgoing. I will update with logs to confirm it.

iperf is doing half-duplex test by default. As I remember from times when I played around with traffic shapers it was easy to slow down outgoing traffic but not easy for incoming. And the only way to limit incoming traffic was to drop some packets.

Nemesiz · Jun 15, 2015

Sometimes I have crazy ideas in my mind. One of them are network connection rate. VirtIO Ethernet support 10M/100M/1G/10G. VM user can change it. But how about not standard rate mode and fixed rate (to make user not accessible to change or changes with no effect). Rates like 50M (full/half duplex). Or does OS will not talk at all ?

Sakis · Jun 19, 2015

libvrt seems to have already this functionality
http://libvirt.org/formatdomain.html#elementQoS

Do we have any update from proxmox team regarding this potential bug / feature?

spirit · Jun 19, 2015

Sakis said:
libvrt seems to have already this functionality
http://libvirt.org/formatdomain.html#elementQoS

yes, in libvirt code
src/util/virnetdevbandwidth.c

Do we have any update from proxmox team regarding this potential bug / feature?

Currently working on proxmox 4.0, I think they don't have yet looked at it.

Sakis · Sep 4, 2015

I would love and update in this

https://bugzilla.proxmox.com/show_bug.cgi?id=639

I have a proxmox 4.0 installation. I will try to find some time and respond if this bug is present in 4.0 Beta

Sakis · Oct 15, 2015

Same problem in Proxmox 4 stable release.
Does anyone working on this?

wolfgang · Oct 15, 2015

Hi,

It is known and we are working on the problem but it is not easy to fix it.
https://bugzilla.proxmox.com/show_bug.cgi?id=639

Sakis · Oct 17, 2015

I am happy to know that you are working in it.
I opened this bug and i thought that no one pay attention to it.
Thanks!

Eth rate limit -> unstable bandwidth

Active Member

Distinguished Member

Active Member

Active Member

Distinguished Member

Distinguished Member

Active Member

Active Member

Distinguished Member

Active Member

Member

Active Member

Member

Renowned Member

Active Member

Distinguished Member

Active Member

Active Member

Proxmox Retired Staff

Active Member