Performance issue with Proxmox 4 - high IO delay

[alex3137]

Yes raw to chroot format made a huge difference in my configuration. Bear in mind you will loose the ability to set disk size from the GUI and container will see all the storage available.

To migrate a container, you need to back it up and restore it this way (replace 100 by CT's number):

pct restore 100 backup.tar.gz --rootfs local:0

Unfortunately it is not possible to create a container with chroot disks from the GUI. Now I use a command which is much quicker as I automate a lot of stuff at the same time: enable automatic security updates, add some packages, allow root login via SSH, configure default locales & vimrc etc...

Try to adapt the below with your configuration ;-) You mainly need to change the CT number (100) and IP address, hostname, etc...

pct create 100 /var/lib/vz/template/cache/debian-8.0-standard_8.0-1_amd64.tar.gz -net0 bridge=vmbr1,gw=10.10.10.254,ip=10.10.10.100/24,name=eth0,type=veth -hostname my.domain.com -password -rootfs local:0 -cpulimit 1 -memory 1024 -swap 1024 &&

pct start 100 &&

pct exec 100 -- bash -c "sed -i.bak s/'# en_US.UTF-8 UTF-8'/'en_US.UTF-8 UTF-8'/g /etc/locale.gen" &&
pct exec 100 -- bash -c "locale-gen" &&
pct exec 100 -- bash -c "sed -i.bak s/'PermitRootLogin without-password'/'PermitRootLogin yes'/g /etc/ssh/sshd_config" &&
pct exec 100 -- bash -c "cp /etc/vim/vimrc ~/.vimrc" &&
pct exec 100 -- bash -c "aptitude update && aptitude upgrade -y" &&
pct exec 100 -- bash -c "aptitude install unattended-upgrades bash-completion man htop iftop iotop bz2 -y"

pct exec 100 -- bash -c "dpkg-reconfigure -plow unattended-upgrades"

 

[Philipp Page]

oh okay, so chroot would be no option for renting vps because disk size is unlimited and the vps can see all storages right?

Did you try to only use kvm? In theory this should work as well or not? I simply cannot explain me the high io activity of the jbd2 process

and thank you for you fast feedback!

This is my current pveperf output

root@srv02:~# pveperf
CPU BOGOMIPS:      38315.12
REGEX/SECOND:      1923670
HD SIZE:           19.10 GB (/dev/md2)
BUFFERED READS:    78.31 MB/sec
AVERAGE SEEK TIME: 18.70 ms
FSYNCS/SECOND:     264.65
DNS EXT:           38.09 ms
DNS INT:           40.75 ms (vhost.pt-hosting.net)

Here is a screenshot of iotop:

I have 2x2tb sata in software raid 1 and ext3 file system

 

[alex3137]

oh okay, so chroot would be no option for renting vps because disk size is unlimited and the vps can see all storages right?

and thank you for you fast feedback!

Yes I believe so but I am not entirely sure.

@fabian do you know if we can set quotas on containers with chroot disks ?

 

[Philipp Page]

Was the chroot the only things that worked for you or do you know something different what could help? I have gameservers running on some vms and they are lagging like hell because of the io wait.

See my io delay

 

[alex3137]

Yes chroot disks improved performances the most but reducing journaling on the partition on the host helped too (data=writeback option in /etc/mount). Best way I found to troubleshoot is install iotop on the host and watch what processes consumes most of the IOps.

 

[Philipp Page]

Okay, thanks. I tried this command, it helped a bit but still not that good: mount -o remount,nobarrier,rw,noatime,errors=remount-ro,data=ordered /
I will try data=writeback

 

[alex3137]

Sorry didn't see your screenshot initially, you definitely need to try both. chroot will in overall increase performances but your issue is the journaling as well (jbd2 process). Try to use data=writeback and let us know

 

[Philipp Page]

i get this output for data=writeback :

root@srv02:~# mount -o remount,nobarrier,rw,noatime,errors=remount-ro,data=writeback /
mount: / not mounted or bad option

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

 

[alex3137]

Not sure how to do it without rebooting server, here how it is mounted in my /etc/mount file:

/dev/pve/data   /var/lib/vz     ext3    defaults,barrier=0,data=writeback               1       2

 

[Philipp Page]

I dont have the file /etc/mount should i just create it then?

 

[alex3137]

Are you sure ? This file is essential to the system. PM me if you want.

 

[mir]

The file is /etc/fstab

 

[Philipp Page]

Thanks,
barrier=0 and data=writeback does not improve the performance.
I will do some mores tests

 

[Philipp Page]

After testing i found out, that as soon as only one lxc container is started the general io performance of the whole host becomes bad due to hard io spikes of jbd2 process. There must be a bug with handling/opening lxc containers in proxmox i think.

Can anyone here tell me an other solution than converting to chroot because this is no option for me?

 

[charlesm1888]

I have the same issue. In my case only one CT (CPanel installed) has a high IO (99.99%) the major time.

So what about to move the CT raw disk to a ZFS partition?

 

[ned]

What is the performance with lxc on lvm thin pools as storage backend?

 

[sahostking]

Same problem. Moved from LXC with exact same hardware specs. Using ZFS. I even tried with the dangerous option of turning off zfs get sync and still very high io.

For me I'm sticking to proxmox 3. Honestly my kvm vms seemed fine but the LXC environment is killing me. Bugs and performance issues and I can't keep just pushing better hardware at it as it costs a arm and leg.

Anyway setting up a few more proxmox 3 servers now and have feeling they going to run smoothly like the first one.

 

[ApolloDS]

Hi all,
I just want to add for everyone that I have the exactly same problem with very high disk-io with the process "99.99 % [jbd2/dm-0-8]".
Very strange is that I use many LXC Containers on some hosts but only one with ext3 is showing this kind of problem.

So is someone know how to troubleshoot this I would be very thankful!

 

[Philipp Page]

I could improve the general io performance by adding noatime,nobarrier,data=ordered options to the filesystem where the vms are stored.
fsync before: 33,46
fsync after: 900-1200

 

[ApolloDS]

Hi all,
I found the problem with my machine, it's Openvswitch.
On this host I used Openvswitch and everything was getting really slow, my Backup runs about 17 hours (!!) and the vm/ct where not so responsive.
I had a standard config with LACP/Bond and VLANs on the Openvswitch.
I cannot recommend Openvswitch anymore, it seems to have it's issues.

Now with Linux Bridge and Bonding everything runs fast again.
Waiting for a backup this night and check the runtime then...

By the way: IO Delay is now also massive lower!