I found that using rate limit at ethernet devices causes bandwidth to be unstable.
Here is a iperf example of a vm with no limits. The iperf server has a 1Gbit limit and the result is expected
I stop the vm and set rate=12 (close to 100Mbit/s), boot and check again (it works on the fly too, but i wanted to be sure cause i am not familiar the mechanism used to rate limit)
Here is the output now
Checking with various cli bandwidth tools i can confirm that the bandwidth is not stable at all. Ofc changing it back to unlimited works again. The node used for iperf client test dont have any other machines running or doing any other bandwidth hungry process.
Is this behaviour expected? Does this effect real bandwidth?
pveversion
kvm conf
Here is a iperf example of a vm with no limits. The iperf server has a 1Gbit limit and the result is expected
Code:
[root@localhost ~]# iperf -c x.x.x.x -t 60 -i 1
------------------------------------------------------------
Client connecting to x.x.x.x, TCP port 5001
TCP window size: 19.3 KByte (default)
------------------------------------------------------------
[ 3] local y.y.y.y port 35179 connected with x.x.x.x port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 102 MBytes 858 Mbits/sec
[ 3] 1.0- 2.0 sec 112 MBytes 941 Mbits/sec
[ 3] 2.0- 3.0 sec 112 MBytes 936 Mbits/sec
[ 3] 3.0- 4.0 sec 97.2 MBytes 816 Mbits/sec
[ 3] 4.0- 5.0 sec 107 MBytes 899 Mbits/sec
[ 3] 5.0- 6.0 sec 112 MBytes 935 Mbits/sec
[ 3] 6.0- 7.0 sec 111 MBytes 931 Mbits/sec
[ 3] 7.0- 8.0 sec 113 MBytes 947 Mbits/sec
[ 3] 8.0- 9.0 sec 105 MBytes 880 Mbits/sec
[ 3] 9.0-10.0 sec 112 MBytes 935 Mbits/sec
[ 3] 10.0-11.0 sec 112 MBytes 935 Mbits/sec
[ 3] 11.0-12.0 sec 112 MBytes 937 Mbits/sec
[ 3] 12.0-13.0 sec 112 MBytes 940 Mbits/sec
[ 3] 13.0-14.0 sec 111 MBytes 929 Mbits/sec
[ 3] 14.0-15.0 sec 112 MBytes 943 Mbits/sec
[ 3] 15.0-16.0 sec 111 MBytes 928 Mbits/sec
[ 3] 16.0-17.0 sec 112 MBytes 940 Mbits/sec
[ 3] 17.0-18.0 sec 112 MBytes 940 Mbits/sec
[ 3] 18.0-19.0 sec 112 MBytes 936 Mbits/sec
[ 3] 19.0-20.0 sec 112 MBytes 937 Mbits/sec
[ 3] 20.0-21.0 sec 112 MBytes 941 Mbits/sec
[ 3] 21.0-22.0 sec 111 MBytes 928 Mbits/sec
[ 3] 22.0-23.0 sec 112 MBytes 940 Mbits/sec
[ 3] 23.0-24.0 sec 110 MBytes 925 Mbits/sec
[ 3] 24.0-25.0 sec 112 MBytes 935 Mbits/sec
[ 3] 25.0-26.0 sec 111 MBytes 931 Mbits/sec
[ 3] 26.0-27.0 sec 110 MBytes 923 Mbits/sec
[ 3] 27.0-28.0 sec 111 MBytes 932 Mbits/sec
[ 3] 28.0-29.0 sec 112 MBytes 935 Mbits/sec
[ 3] 29.0-30.0 sec 110 MBytes 921 Mbits/sec
[ 3] 30.0-31.0 sec 110 MBytes 926 Mbits/sec
[ 3] 31.0-32.0 sec 111 MBytes 930 Mbits/sec
[ 3] 32.0-33.0 sec 110 MBytes 920 Mbits/sec
[ 3] 33.0-34.0 sec 112 MBytes 941 Mbits/sec
[ 3] 34.0-35.0 sec 111 MBytes 928 Mbits/sec
[ 3] 35.0-36.0 sec 112 MBytes 942 Mbits/sec
[ 3] 36.0-37.0 sec 112 MBytes 940 Mbits/sec
[ 3] 37.0-38.0 sec 112 MBytes 936 Mbits/sec
[ 3] 38.0-39.0 sec 111 MBytes 928 Mbits/sec
[ 3] 39.0-40.0 sec 111 MBytes 929 Mbits/sec
[ 3] 40.0-41.0 sec 112 MBytes 943 Mbits/sec
[ 3] 41.0-42.0 sec 111 MBytes 930 Mbits/sec
[ 3] 42.0-43.0 sec 113 MBytes 947 Mbits/sec
[ 3] 43.0-44.0 sec 112 MBytes 936 Mbits/sec
[ 3] 44.0-45.0 sec 111 MBytes 930 Mbits/sec
[ 3] 45.0-46.0 sec 107 MBytes 899 Mbits/sec
[ 3] 46.0-47.0 sec 109 MBytes 918 Mbits/sec
[ 3] 47.0-48.0 sec 111 MBytes 933 Mbits/sec
[ 3] 48.0-49.0 sec 112 MBytes 935 Mbits/sec
[ 3] 49.0-50.0 sec 112 MBytes 940 Mbits/sec
[ 3] 50.0-51.0 sec 112 MBytes 938 Mbits/sec
[ 3] 51.0-52.0 sec 112 MBytes 937 Mbits/sec
[ 3] 52.0-53.0 sec 112 MBytes 943 Mbits/sec
[ 3] 53.0-54.0 sec 111 MBytes 930 Mbits/sec
[ 3] 54.0-55.0 sec 113 MBytes 948 Mbits/sec
[ 3] 55.0-56.0 sec 111 MBytes 930 Mbits/sec
[ 3] 56.0-57.0 sec 112 MBytes 943 Mbits/sec
[ 3] 57.0-58.0 sec 112 MBytes 936 Mbits/sec
[ 3] 58.0-59.0 sec 110 MBytes 924 Mbits/sec
[ 3] 59.0-60.0 sec 112 MBytes 940 Mbits/sec
[ 3] 0.0-60.0 sec 6.49 GBytes 929 Mbits/sec
I stop the vm and set rate=12 (close to 100Mbit/s), boot and check again (it works on the fly too, but i wanted to be sure cause i am not familiar the mechanism used to rate limit)
Here is the output now
Code:
[root@localhost ~]# iperf -c x.x.x.x -t 60 -i 1
------------------------------------------------------------
Client connecting to x.x.x.x, TCP port 5001
TCP window size: 19.3 KByte (default)
------------------------------------------------------------
[ 3] local y.y.y.y port 57898 connected with x.x.x.x port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 100 MBytes 839 Mbits/sec
[ 3] 1.0- 2.0 sec 74.8 MBytes 627 Mbits/sec
[ 3] 2.0- 3.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 3.0- 4.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 4.0- 5.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 5.0- 6.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 6.0- 7.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 7.0- 8.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 8.0- 9.0 sec 55.8 MBytes 468 Mbits/sec
[ 3] 9.0-10.0 sec 66.5 MBytes 558 Mbits/sec
[ 3] 10.0-11.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 11.0-12.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 12.0-13.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 13.0-14.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 14.0-15.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 15.0-16.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 16.0-17.0 sec 60.0 MBytes 503 Mbits/sec
[ 3] 17.0-18.0 sec 73.4 MBytes 616 Mbits/sec
[ 3] 18.0-19.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 19.0-20.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 20.0-21.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 21.0-22.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 22.0-23.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 23.0-24.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 24.0-25.0 sec 27.9 MBytes 234 Mbits/sec
[ 3] 25.0-26.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 26.0-27.0 sec 25.2 MBytes 212 Mbits/sec
[ 3] 27.0-28.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 28.0-29.0 sec 5.38 MBytes 45.1 Mbits/sec
[ 3] 29.0-30.0 sec 62.8 MBytes 526 Mbits/sec
[ 3] 30.0-31.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 31.0-32.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 32.0-33.0 sec 1.38 MBytes 11.5 Mbits/sec
[ 3] 33.0-34.0 sec 65.8 MBytes 552 Mbits/sec
[ 3] 34.0-35.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 35.0-36.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 36.0-37.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 37.0-38.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 38.0-39.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 39.0-40.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 40.0-41.0 sec 69.1 MBytes 580 Mbits/sec
[ 3] 41.0-42.0 sec 73.4 MBytes 616 Mbits/sec
[ 3] 42.0-43.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 43.0-44.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 44.0-45.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 45.0-46.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 46.0-47.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 47.0-48.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 48.0-49.0 sec 26.2 MBytes 220 Mbits/sec
[ 3] 49.0-50.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 50.0-51.0 sec 31.2 MBytes 262 Mbits/sec
[ 3] 51.0-52.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 52.0-53.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 53.0-54.0 sec 1.38 MBytes 11.5 Mbits/sec
[ 3] 54.0-55.0 sec 67.9 MBytes 569 Mbits/sec
[ 3] 55.0-56.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 56.0-57.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 57.0-58.0 sec 1.38 MBytes 11.5 Mbits/sec
[ 3] 58.0-59.0 sec 67.2 MBytes 564 Mbits/sec
[ 3] 59.0-60.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 60.0-61.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 61.0-62.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 62.0-63.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 63.0-64.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 64.0-65.0 sec 0.00 Bytes 0.00 bits/sec
[ 3] 0.0-65.2 sec 957 MBytes 123 Mbits/sec
Checking with various cli bandwidth tools i can confirm that the bandwidth is not stable at all. Ofc changing it back to unlimited works again. The node used for iperf client test dont have any other machines running or doing any other bandwidth hungry process.
Is this behaviour expected? Does this effect real bandwidth?
pveversion
Code:
proxmox-ve-2.6.32: 3.4-150 (running kernel: 2.6.32-37-pve)
pve-manager: 3.4-3 (running version: 3.4-3/2fc72fee)
pve-kernel-2.6.32-32-pve: 2.6.32-136
pve-kernel-2.6.32-37-pve: 2.6.32-150
pve-kernel-2.6.32-29-pve: 2.6.32-126
pve-kernel-2.6.32-31-pve: 2.6.32-132
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-2
pve-cluster: 3.0-16
qemu-server: 3.4-3
pve-firmware: 1.1-4
libpve-common-perl: 3.0-24
libpve-access-control: 3.0-16
libpve-storage-perl: 3.0-32
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.2-10
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1
kvm conf
Code:
bootdisk: virtio0
cores: 2
cpu: qemu64
ide2: none,media=cdrom
memory: 4096
name: eth-test
net0: virtio=8E:19:A2:5E:24:B8,bridge=vmbr0,rate=12
numa: 0
ostype: l26
smbios1: uuid=e49feb90-75d5-4036-9693-1ceaa81c651b
sockets: 1
virtio0: pool:vm-150-disk-1,size=20G