Problems with i/o on pve kernels

[KORG]

Hi,

We use Debian 6.0.7 with Proxmox kernel:

Linux 2.6.32-17-pve #1 SMP Wed Nov 28 07:15:55 CET 2012 x86_64 GNU/Linux

Servel normaly handles 247 OpenVZ containers. But.. when someone of containers use active I/O operations - for example extract 1Gb tar.gz archive - server load average goes to ~2000 la.

Used scheduler - CFQ.

Can you please give any advice?

 

[mmenaz]

See: http://forum.proxmox.com/threads/10...sponsiveness-even-if-on-different-SAS-storage

 

[gkovacs]

Use deadline I/O scheduler, regardless of what people (or even Proxmox devs) say. CFQ has been proven time and time again as not suitable for server environments, regardless of the features it offers.

You need to modify /boot/grub/grub.cfg, then run update-grub2 and reboot.

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet elevator=deadline"
GRUB_CMDLINE_LINUX="elevator=deadline"

The Red Hat based kernel (first seen in 2.6.32-6-pve) introduced serious I/O performance regressions, which combined with the sluggish CFQ scheduler result in exactly what you experienced: when a single process uses heavy I/O, all the other requests are starved. Deadline solves this but providing equal access to I/O resources, with much smaller overhead.

For example when snapshot backups start on our servers (using 6 and 8 disk Adaptec 6805 RAID10 arrays), using CFQ websites hosted under OpenVZ regularly timed out. With deadline you don't even notice the backups running.

The things we set up on Proxmox 2.3 to improve I/O performance:
- ext4 filesystem (much faster fsck, somewhat faster quota calculation after unscheduled reboot)
- deadline I/O scheduler
- noatime, nodiratime mount options on both pve-root and pve-data

Of course it helps to use a decent RAID controller, striped arrays (like RAID10) with large stripe sizes are preferred to parity-based ones (like RAID5).

 

[mmenaz]

- noatime, nodiratime mount options on both pve-root and pve-data

Just to add that noatime is a superset of nodiratime, so only the former is needed (if you specify both, nodiratime is redundant).