Performance issue with Proxmox 4 - high IO delay
- Thread starter alex3137
- Start date
No I did a reconfig of /etc/network/interfaces but I have to remove it with
# apt-get purge openvswitch-common openvswitch-switch
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet manual
mtu 8996
auto eth1
iface eth1 inet manual
mtu 8996
auto eth2
iface eth2 inet manual
mtu 8996
auto eth3
iface eth3 inet manual
mtu 8996
auto bond0
iface bond0 inet manual
slaves eth0 eth1 eth2 eth3
bond_miimon 100
bond_mode 802.3ad
bond_xmit_hash_policy layer3+4
mtu 8996
auto vlan122
iface vlan122 inet manual
vlan-raw-device bond0
mtu 8996
auto vmbr0
iface vmbr0 inet static
address 10.1.2.3
netmask 255.255.255.0
gateway 10.1.2.1
network 10.1.2.0
broadcast 10.1.2.255
bridge_ports bond0
bridge_stp off
bridge_fd 0
dns-nameservers 172.18.1.1
dns-search mgmt.domain.ch corp.domain.ch
bridge_fs 0
# dns-* options are implemented by the resolvconf package, if installed
auto vmbr1
iface vmbr1 inet static
address 10.2.1.131
netmask 255.255.255.0
bridge_ports vlan122
bridge_stp off
bridge_fd 0
Above does not do the required thing. Eg. not using vlan122. The proper way on Jessie is:
1) Skip the vlan-raw-device
2) Use only:
auto vmbr1
iface vmbr1 inet static
address 10.2.1.131
netmask 255.255.255.0
bridge_ports bond0.122
bridge_stp off
bridge_fd 0
Hi mir,
Well, see here: https://pve.proxmox.com/wiki/Network_Model#Create_VLAN
"First explicit method" and "I prefer the first one!" (??)
But I also changed now to the following config which is working stable and fast:
........
auto bond0
iface bond0 inet manual
slaves eth0 eth1 eth2 eth3
bond_miimon 100
bond_mode 802.3ad
bond_xmit_hash_policy layer3+4
mtu 8996
auto vmbr0
iface vmbr0 inet static
address 10.1.2.3
netmask 255.255.255.0
broadcast 10.1.2.255
bridge_ports bond0
bridge_stp off
bridge_fd 0
bridge_vlan_aware yes
network 10.1.2.0
dns-nameservers 1.1.1.1
dns-search mgmt.domain.ch corp.domain.ch
bridge_fs 0
auto vmbr122
iface vmbr122 inet static
address 10.2.2.3
netmask 255.255.255.0
bridge_ports bond0.122
bridge_stp off
bridge_fd 0
........
I don't trust Openvswitch anymore because of the speed problems...
And I almost forgot. vmbr0 is the LACP Trunk connected to a "General" Mode Port to a Cisco Switch. So I have on the PVID the Management Network (10.1.2.0/24) for the Proxmox Host.
Well, see here: https://pve.proxmox.com/wiki/Network_Model#Create_VLAN
"First explicit method" and "I prefer the first one!" (??)
But I also changed now to the following config which is working stable and fast:
........
auto bond0
iface bond0 inet manual
slaves eth0 eth1 eth2 eth3
bond_miimon 100
bond_mode 802.3ad
bond_xmit_hash_policy layer3+4
mtu 8996
auto vmbr0
iface vmbr0 inet static
address 10.1.2.3
netmask 255.255.255.0
broadcast 10.1.2.255
bridge_ports bond0
bridge_stp off
bridge_fd 0
bridge_vlan_aware yes
network 10.1.2.0
dns-nameservers 1.1.1.1
dns-search mgmt.domain.ch corp.domain.ch
bridge_fs 0
auto vmbr122
iface vmbr122 inet static
address 10.2.2.3
netmask 255.255.255.0
bridge_ports bond0.122
bridge_stp off
bridge_fd 0
........
I don't trust Openvswitch anymore because of the speed problems...
And I almost forgot. vmbr0 is the LACP Trunk connected to a "General" Mode Port to a Cisco Switch. So I have on the PVID the Management Network (10.1.2.0/24) for the Proxmox Host.
Last edited:
Changing to chroot format and fstab options worked form me.
The LXC VM is really faster now. An import of a dump sql takes 8 secs. Before changing this... 5 minutes.
I have a OVH server with Proxmox 4.2. LVM on ext3.
The LXC VM is really faster now. An import of a dump sql takes 8 secs. Before changing this... 5 minutes.
I have a OVH server with Proxmox 4.2. LVM on ext3.
Code:
pct restore 100 backup.tar.gz --rootfs local:0
Code:
...
/dev/vg1/lv1 /var/lib/vz ext3 defaults,noatime,nobarrier,data=ordered 0 2
/dev/sdb1 /backups ext3 defaults,noatime,nobarrier,data=ordered 0 3Is the problem with LVM Thin or RAW simply that these volumes are mounted with barriers=1 then? Clearly they are not mounted inside /etc/fstab so unable to change the mount options but could be why chroot is faster.
Is it possible to add an option like on KVM for the disks where write back could be selected as an option for the LXC disk? That in my head would suggest it would achieve the same as barriers=0 but could be selected on a per LXC basis.
Is it possible to add an option like on KVM for the disks where write back could be selected as an option for the LXC disk? That in my head would suggest it would achieve the same as barriers=0 but could be selected on a per LXC basis.
Hi. I am using Proxmox 3 for OpenVZ containers and moved to LXC in Proxmox 4. These containers are Debian8 mysql database servers with a lot of queries that are executed by PHP scripts (nginx + php5-fpm). There are 2000-5000 connections to PHP per second depending on time of a day. I am executing calculations basing on database entries (millions od records) periodicaly.
I have spent a lot of time to make decision that I have to downgrade to version 3.4. I will try to explain you why. But first, my configuration. I have tested on two identical dedicated machines in OVH:
Intel Xeon E3-1245v5 (8 x 3.50 GHz)
32 GB ram
2 x HGST HUS726020ALA610 SATA 2 TB
Software Raid 1 + LVM on ext3 filesystem
Latest Proxmox 3 or 4 versions with latest package updates
Have a look at FSYNC/SECOND in pveperf for Proxmox 3:
and the same for Proxmox 4:
The increase of IO delay in Proxmox panel was the first thing I have seen in 24h after upgrade to LXC on Promox 4. Yesterday, after a week I have downgraded Proxmox to version 3 and moved back to OpenVZ. See the difference in CPU load (which depends also on io delays):
I have also tested on Proxmox 4 the following configurations:
Maybe there are some kernel 4.x problems i Proxmox 4? I can't explain it but I see the facts above.
I have to say that I have tested Proxmox 4 on OVH machines with hardware LSI megaraid and on SSD disk with software raid. In both cases FSYNCS/SECOND was at the level of 5000 and the whole system performance was very good.
I have spent a lot of time to make decision that I have to downgrade to version 3.4. I will try to explain you why. But first, my configuration. I have tested on two identical dedicated machines in OVH:
Intel Xeon E3-1245v5 (8 x 3.50 GHz)
32 GB ram
2 x HGST HUS726020ALA610 SATA 2 TB
Software Raid 1 + LVM on ext3 filesystem
Latest Proxmox 3 or 4 versions with latest package updates
Have a look at FSYNC/SECOND in pveperf for Proxmox 3:
and the same for Proxmox 4:
The increase of IO delay in Proxmox panel was the first thing I have seen in 24h after upgrade to LXC on Promox 4. Yesterday, after a week I have downgraded Proxmox to version 3 and moved back to OpenVZ. See the difference in CPU load (which depends also on io delays):
I have also tested on Proxmox 4 the following configurations:
- No software Raid - only LVM
- No LVM - only software Raid
- No software Raid and no LVM
- ext3, ext4, zfs (with its raid 1) filesystems
Maybe there are some kernel 4.x problems i Proxmox 4? I can't explain it but I see the facts above.
I have to say that I have tested Proxmox 4 on OVH machines with hardware LSI megaraid and on SSD disk with software raid. In both cases FSYNCS/SECOND was at the level of 5000 and the whole system performance was very good.
Hello V-SPEED,
Could you please test on Proxmox 4 ext3 filesystem performance with mount option "barrier=0"
for ext4 fs i have great improvements on software raid1
default mount options:
CPU BOGOMIPS: 36269.76
REGEX/SECOND: 876545
HD SIZE: 458.20 GB (/dev/md0)
BUFFERED READS: 192.36 MB/sec
AVERAGE SEEK TIME: 19.20 ms
FSYNCS/SECOND: 58.10
DNS EXT: 101.53 ms
DNS INT: 127.28 ms
and with barrier=0
CPU BOGOMIPS: 36269.76
REGEX/SECOND: 1032417
HD SIZE: 458.20 GB (/dev/md0)
BUFFERED READS: 182.06 MB/sec
AVERAGE SEEK TIME: 18.33 ms
FSYNCS/SECOND: 594.19
DNS EXT: 86.55 ms
DNS INT: 108.55 ms
Could you please test on Proxmox 4 ext3 filesystem performance with mount option "barrier=0"
for ext4 fs i have great improvements on software raid1
default mount options:
CPU BOGOMIPS: 36269.76
REGEX/SECOND: 876545
HD SIZE: 458.20 GB (/dev/md0)
BUFFERED READS: 192.36 MB/sec
AVERAGE SEEK TIME: 19.20 ms
FSYNCS/SECOND: 58.10
DNS EXT: 101.53 ms
DNS INT: 127.28 ms
and with barrier=0
CPU BOGOMIPS: 36269.76
REGEX/SECOND: 1032417
HD SIZE: 458.20 GB (/dev/md0)
BUFFERED READS: 182.06 MB/sec
AVERAGE SEEK TIME: 18.33 ms
FSYNCS/SECOND: 594.19
DNS EXT: 86.55 ms
DNS INT: 108.55 ms
If can be useful, I have tested it to the same config and with or without barrier=0. No real difference.
disks settings (for 4.4):
Test on 3.4:
Test on 4.4:
Really interested to find a way to have decent performance with LXC.
Best regards, P.
disks settings (for 4.4):
Code:
/dev/md2 on / type ext3 (rw,relatime,nobarrier,errors=remount-ro,data=ordered)
/dev/mapper/pve-data on /var/lib/vz type ext3 (rw,relatime,nobarrier,data=ordered)Test on 3.4:
Code:
# pveperf /var/lib/vz
CPU BOGOMIPS: 42666.16
REGEX/SECOND: 1054378
HD SIZE: 874.50 GB (/dev/mapper/pve-data)
BUFFERED READS: 93.75 MB/sec
AVERAGE SEEK TIME: 11.18 ms
FSYNCS/SECOND: 704.72
DNS EXT: 31.62 ms
DNS INT: 38.30 msTest on 4.4:
Code:
# pveperf /var/lib/vz
CPU BOGOMIPS: 42667.28
REGEX/SECOND: 1260695
HD SIZE: 905.94 GB (/dev/mapper/pve-data)
BUFFERED READS: 49.48 MB/sec
AVERAGE SEEK TIME: 55.28 ms
FSYNCS/SECOND: 35.40
DNS EXT: 35.80 ms
DNS INT: 34.75 msReally interested to find a way to have decent performance with LXC.
Best regards, P.
PaulVM - in my case it helped. On the investigated machine I have two LVMs and one pure md on ext3. One LVMs mounted with barrier=0 and the rest without. The result:
Now I am looking for the same solution for ZFS. As we know, LXC snapshot mode in backups is not available for LVM (ext3, ext4) but works fine on ZFS. Performance for ZFS is as bad as without barrier=0 for ext3. Anyone can help with this?
Code:
root@ns3033026:/# pveversion
pve-manager/4.4-13/7ea56165 (running kernel: 4.4.49-1-pve)
root@ns3033026:/# mount | grep ext
/dev/md2 on / type ext3 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/pve-data on /var/lib/vz type ext3 (rw,relatime,nobarrier,data=ordered)
/dev/mapper/pve-backupsys on /home/backupsys type ext3 (rw,relatime,data=ordered)
root@ns3033026:/# pveperf /
CPU BOGOMIPS: 54280.00
REGEX/SECOND: 2923105
HD SIZE: 19.10 GB (/dev/md2)
BUFFERED READS: 129.28 MB/sec
AVERAGE SEEK TIME: 6.32 ms
FSYNCS/SECOND: 22.94
DNS EXT: 21.84 ms
DNS INT: 1.06 ms (ip-149-202-206.eu)
root@ns3033026:/# pveperf /var/lib/vz
CPU BOGOMIPS: 54280.00
REGEX/SECOND: 2938194
HD SIZE: 590.46 GB (/dev/mapper/pve-data)
BUFFERED READS: 147.97 MB/sec
AVERAGE SEEK TIME: 10.86 ms
FSYNCS/SECOND: 840.98
DNS EXT: 21.13 ms
DNS INT: 1.67 ms (ip-149-202-206.eu)
root@ns3033026:/# pveperf /home/backupsys/
CPU BOGOMIPS: 54280.00
REGEX/SECOND: 2937838
HD SIZE: 1167.70 GB (/dev/mapper/pve-backupsys)
BUFFERED READS: 95.07 MB/sec
AVERAGE SEEK TIME: 11.41 ms
FSYNCS/SECOND: 13.36
DNS EXT: 22.71 ms
DNS INT: 1.06 ms (ip-149-202-206.eu)Now I am looking for the same solution for ZFS. As we know, LXC snapshot mode in backups is not available for LVM (ext3, ext4) but works fine on ZFS. Performance for ZFS is as bad as without barrier=0 for ext3. Anyone can help with this?