BUG: Bandwith Limit does not work after Migration of VM

eude · Jul 13, 2021

Hi everyone,
we are using a Bandwith Limit for some of our Customers set viá the Proxmox UI. We discovered, that some of our Customers have way higher Download/Upload rates then they should be. After some digging, we found out, that if you Migrate a VM that has a Bandwith rate limit set to lets say 13 MB/s to another Node, the Ratelimit does not work anymore.

Here a Image of an iperf, at Hop 24 you see the succesful Migration to another node

The vm.conf file on the nodes keep the rate limit (rate=13) but it does not affect the migratet machine anymore:

if you make a change to the Bandwith Limit, it works again.

running on:
pve-manager/6.4-9/5f5c0e3f (running kernel: 5.4.119-1-pve)

fabian · Jul 13, 2021

is the limit correctly applied when you stop and then start the VM again? could you include

pveversion -v from both nodes?

eude · Jul 13, 2021

if i migrate the vm, turn it off, turn it back on the Bandwith Limit does NOT work.

Code:

root@KDALNPPX008:~# pveversion -v
proxmox-ve: 6.4-1 (running kernel: 5.4.119-1-pve)
pve-manager: 6.4-9 (running version: 6.4-9/5f5c0e3f)
pve-kernel-5.4: 6.4-3
pve-kernel-helper: 6.4-3
pve-kernel-5.4.119-1-pve: 5.4.119-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-3
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-3
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
openvswitch-switch: 2.12.3-1
proxmox-backup-client: 1.1.10-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-6
pve-cluster: 6.4-1
pve-container: 3.3-5
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.2-4
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1

Code:

root@prx12:~# pveversion -v
proxmox-ve: 6.4-1 (running kernel: 5.4.119-1-pve)
pve-manager: 6.4-9 (running version: 6.4-9/5f5c0e3f)
pve-kernel-5.4: 6.4-3
pve-kernel-helper: 6.4-3
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.119-1-pve: 5.4.119-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.73-1-pve: 5.4.73-1
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-4.15: 5.4-12
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-4.15.18-24-pve: 4.15.18-52
pve-kernel-4.15.18-10-pve: 4.15.18-32
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ifupdown2: residual config
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-3
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-3
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
openvswitch-switch: 2.12.3-1
proxmox-backup-client: 1.1.10-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-6
pve-cluster: 6.4-1
pve-container: 3.3-5
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.2-4
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1

(we have 16 Nodes, tried with 5 of em and 3 different VMs)

fabian · Jul 13, 2021

works here.. could you post the output of tc qdisk | grep VMID for a non working VMID? anything special about your network setup?

Odie02 · Jul 13, 2021

fabian said:
works here.. could you post the output of tc qdisk | grep VMID for a non working VMID? anything special about your network setup?

tc qdisc ;-)

eude · Jul 13, 2021

Code:

root@KDALNPPX008:~# tc qdisc | grep 175
qdisc htb 1: dev tap175i0 root refcnt 2 r2q 10 default 0x1 direct_packets_stat 0 direct_qlen 1000

Bash:

auto lo
iface lo inet loopback

auto ens1f0
iface ens1f0 inet manual

auto ens1f1
iface ens1f1 inet manual

auto ens3f0
iface ens3f0 inet manual

auto ens3f1
iface ens3f1 inet manual

iface eno1 inet manual
iface eno2 inet manual
iface eno3 inet manual
iface eno4 inet manual

#20Gb Uplink FlexFabric
allow-vmbr0 bond0
iface bond0 inet manual
        ovs_bridge vmbr0
        ovs_type OVSBond
        ovs_bonds ens3f0 ens3f1
        pre-up (ifconfig ens3f0 && ifconfig ens3f1)
        ovs_options lacp=active other_config:lacp-time=fast bond_mode=balance-tcp

# OVS - Bridge
auto vmbr0
allow-vmbr0 vmbr0
allow-ovs vmbr0
iface vmbr0 inet manual
        ovs_type OVSBridge
        ovs_ports bond0 vlan20 vlan128 vlan197 vlan276 vlan277 vlan1010 vlan1005 vlan1025 vlan1060 vlan1070 vlan1100 vlan2001 vlan2002 vlan2003 $
# OVS - Bridge; Kunde Regler vlan2030-2039
auto vmbr2030
allow-vmbr2030 vmbr2030
allow-ovs vmbr2030
iface vmbr2030 inet manual
        ovs_type OVSBridge
        ovs_ports bond0 vlan2030 vlan2031 vlan2032 vlan2033 vlan2034 vlan2035 vlan2036 vlan2037 vlan2038 vlan2039

#Proxmox Management vLan
allow-vmbr0 vlan1100
iface vlan1100 inet static
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options tag=1100
        ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif
        address  10.10.100.42
        netmask  255.255.255.0
        gateway 10.10.100.254
      
#Proxmox cluster com vlan
allow-vmbr0 vlan1010
iface vlan1010 inet static
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options tag=1010
        ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif
        address  10.10.10.42
        netmask  255.255.255.0
        gateway  10.10.10.254

auto bond1
iface bond1 inet static
        address  10.10.5.42
        netmask  255.255.255.0
        bond-slaves ens1f0 ens1f1
        bond-mode 4
        bond-miimon 100
        bond-updelay 200
        bond-downdelay 200
        bond-lacp-rate 1
        pre-up ethtool -s ens1f0 speed 10000 duplex full autoneg off && ethtool -s ens1f1 speed 10000 duplex full autoneg off
        post-up ifup ens1f0 && ifup ens1f1

Bash:

auto lo
iface lo inet loopback

iface eno1 inet manual
iface eno2 inet manual

auto eno33np0
iface eno33np0 inet manual
auto eno34np1
iface eno34np1 inet manual

auto ens2f0np0
iface ens2f0np0 inet manual
auto ens2f1np1
iface ens2f1np1 inet manual

#proxmox config 09.06.2021

#20gb uplink
allow-vmbr0 bond0
iface bond0 inet manual
        ovs_bridge vmbr0
        ovs_type OVSBond
        ovs_bonds eno33np0 eno34np1
        pre-up (ifconfig eno33np0 && ifconfig eno34np1)
        ovs_options bond_mode=balance-tcp lacp=active other_config:lacp-time=fast

#ovs-bridge
auto vmbr0
allow-vmbr0 vmbr0
allow-ovs vmbr0
iface vmbr0 inet manual
        ovs_type OVSBridge
        ovs_ports bond0 vlan20 vlan128 vlan197 vlan276 vlan277 vlan1025 vlan1010 vlan1005 vlan1060 vlan1070 vlan1100 vlan2001 vlan2002 vlan2003 vlan2004 vlan2005 vlan2006 vlan2007 vlan$
      
#prx-mngmt
allow-vmbr0 vlan1100
iface vlan1100 inet static
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options tag=1100
        ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif
        address 10.10.100.38
        netmask 255.255.255.0
        gateway 10.10.100.254

#prx-cluster-com
allow-vmbr0 vlan1010
iface vlan1010 inet static
        ovs_type OVSIntPort
        ovs_bridge vmbr0
        ovs_options tag=1010
        ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif
        address 10.10.10.38
        netmask 255.255.255.0
        gateway 10.10.10.254
      
#ceph-client-nwk
auto bond2
iface bond2 inet static
        address 10.10.5.38
        netmask 255.255.255.0
        bond_slaves ens2f0np0 ens2f1np1
        bond-mode 4
        bon-miimon 100
        bond-updelay 200
        bond-downdelay 200
        bond-lacp-rate 1
        pre-up ethtool -s ens2f0np0 speed 10000 duplex full autoneg off && ethtool -s ens2f1np1 speed 10000 duplex full autoneg off
        post-up ifup ens2f0np0 && ifup ens2f1np1

fabian · Jul 13, 2021

could you post the output of the tc command after changing the ratelimit while the VM is running?

fabian · Jul 13, 2021

also tc class show dev tap175i0 and tc filter show dev tap175i0 root would be interesting after starting VM 175 and after changing the ratelimit while it's running.

eude · Jul 13, 2021

no rate change (unlimited BW):

and after the change (limit to 12.5MB/s)

iperf on the vm:

fabian · Jul 14, 2021

that really looks quite weird.. could you do the full tc qdisc show dev tap175i0 as well before and after (instead of grep)? is there anything else on that system that might interfere/set its own limits?

fabian · Jul 14, 2021

so a colleague helpfully pointed out that this is an OVS issue - OVS clears/reset the rate limit when setting up the bridge port..

https://bugs.launchpad.net/ubuntu/+source/openvswitch/+bug/1840716

we'll see how we can handle this properly/better!

eude · Jul 14, 2021

ah, great - thank you!

fabian · Jul 14, 2021

filed https://bugzilla.proxmox.com/show_bug.cgi?id=3529 for tracking

Search

Search

BUG: Bandwith Limit does not work after Migration of VM

eude

New Member

fabian

Proxmox Staff Member

eude

New Member

fabian

Proxmox Staff Member

Odie02

New Member

eude

New Member

fabian

Proxmox Staff Member

fabian

Proxmox Staff Member

eude

New Member

Attachments

fabian

Proxmox Staff Member

fabian

Proxmox Staff Member

eude

New Member

fabian

Proxmox Staff Member