disk I/O wait on guest at 30% while I/O wait on host below 3%

[sub0]

Hello Proxmoxers,

We're seeing very high load averages in a CentOS 6.6 guest which is caused by I/O wait anywhere from 15-70%. The Proxmox 3.1-21 host I/O wait is typically well below 3%, occasionally spiking to 5%.

The guest virtual disk (500GB) is configured with virtio, qcow2 and cache set to Default (No cache). From reading http://pve.proxmox.com/wiki/Performance_Tweaks#Disk_Cache it seems like No Cache is the optimum setting.

What could be causing the bottleneck?

-paul

 

[sumsum]

What application do you run in the guest VM ?

 

[sub0]

What application do you run in the guest VM ?

NFS server. There's another guest that runs Asterisk and an NFS client. The Asterisk guest writes about 2Mbit/s worth of recordings to the NFS server through its NFS mount.

 

[sumsum]

How many CPU Cores have you assigned to the guest OS?

 

[sub0]

There was only one CPU core assigned to the guest OS. I added another core yesterday and that dropped the load average below 1. So that helped a lot... However, the guest OS I/O wait % remains high, frequently above 20%.

 

[mir]

How fast is your storage?

Run a fio and bonnie+ test and show the result.

 

[sub0]

I'll have to wait with that until off-peak hours, but I don't think the underlying storage is the bottleneck. The image shows host block I/O over time. The long bursts in throughput, beginning at midnight and 3pm, are due to backup jobs. The small ripples during daytime are due to activity by the NFS server guest and other guests. So there should be plenty of capacity... which is confirmed by the low host I/O wait % reported in the Proxmox UI.

 

[sub0]

Does anyone have any ideas what could be causing the poor guest I/O performance?

 

[wahmed]

This is just my opinion. I think the qcow2 disk image type might be causing the high I/O. Unlike .raw format, qcow2 does add some overhead causing higher I/O wait. If you could duplicate the server, try to change the disk image type to .raw. I think you will see big improvement.

 

[cesarpk]

@sub0:

According to IBM, the best practice is to use LVM as block device instead of a file system with a disk image file.

Please see this link:
http://www-01.ibm.com/support/knowledgecenter/#!/linuxonibm/liaat/liaatbpblock.htm

Best regards
Cesar

 

[sub0]

Thanks all for your replies. We bypassed NFS and created a raw disk for the asterisk guest on top of LVM. It appears to have solved the issue. I/O wait is now averaging below 5%.