VM performance problem.

[wipeout_dude]

Hi All..

I have been playing with Proxmox VE for a few weeks now but can't seem to solve a performance issue I am having in VM's..

I have a single Proxmox VE node and a FreeBSD based NAS server.. The drives in the NAS are a ZFS RAID 10 setup and exported over NFS..

Now if I run a simple dd write test from the Proxmox VE node to the mount point I can get a write performance of 80-88 MB/s.. The same test from within a VM is about half the speed at a max of about 40MB/s..

I have atime disabled, I have tried the deadline IO scheduler, I have tried tuning the NAS.. Nothing has made any significant difference..

So based on the fact I was getting 50% throughput on one VM I tried to run the test on two separate VM's at the same time.. This reached a combined
throughput of 47MB/s (~58%)..

I am using Virtio and raw virtual disk format..

So can anyone make any suggestions as to why the VM performance is so poor?

Thanks..

 

[sumsum]

I assume you are using FreeNAS. serveral community memebers are facing the same problem. search this forum for the topcis.
The problem is on the FreeNAS site. check as well their forum for possible solutions.

 

[wipeout_dude]

I assume you are using FreeNAS. serveral community memebers are facing the same problem. search this forum for the topcis.
The problem is on the FreeNAS site. check as well their forum for possible solutions.

No, not using Freenas.. Actually started with Nas4Free which is FreeBSD 9.0 based so has ZFS v28.. I have then tuned it manually for my system to improve the performance of the NAS itself..

I'll lookup the FreeNAS threads and see if there are any pointers there..

 

[wipeout_dude]

Have had a look around the forums but can't see anything new to try and improve my VM performance so any suggestions appreciated..

 

[macday]

don´t use zfs...I also tried zfs on openindiana and nexenta...it is simply damn slow for virtualization...it is good for backups and fileservers but not as storage for kvm...

 

[wipeout_dude]

don´t use zfs...I also tried zfs on openindiana and nexenta...it is simply damn slow for virtualization...it is good for backups and fileservers but not as storage for kvm...

Hmmm.. That's unfortunate because the features of ZFS are brilliant for creating a storage system for a cluster..

Wonder if an SSD ZIL will significantly improve things..

Obviously if I can't solve it I am back to the drawing board..

 

[macday]

i also tried ssd zil...no luck...sorry

 

[wipeout_dude]

Damn!!

So out of interest what did you end up using?

 

[macday]

2 things....local raid10 sas storage with drbd on 2 hosts (10gb intel nic) ....and sas san storage fibrechannel shared...thats the only way to get happy

 

[wipeout_dude]

Was hoping to avoid the complexity and cost of a full blown SAN and have shared on a NAS.. Its odd that the performance is fine natively on the PVE node but rubbish in the VM..

Looks like I am going to have to give it some thought..

 

[macday]

drbd is the way go...if you have the posibility to make a real life test...

 

[wipeout_dude]

I don't think ZFS is the problem..

I have just deleted the ZFS volume and created a 4 disk RAID0 stripe formatted with UFS.. Native throughput from Proxmox VE is *slower* than ZFS was at 72MB/s and throughput from inside the VM is roughly the same at ~42MB/s.. These were the best results..

This suggests its something else limiting the throughput.. Don't think finding it will be that easy either..

*edit*
As a note, write throughput on the NAS box itself to the stripe is over 200MB/s..

 

[macday]

hmmm....try again with a debian 6 "nas" with xfs as storage (but also a hardware raid, no fake raid)...you will see the difference

 

[Erk]

don´t use zfs...I also tried zfs on openindiana and nexenta...it is simply damn slow for virtualization...it is good for backups and fileservers but not as storage for kvm...

That's not true, you just haven't tuned ZFS properly. Google for ZFS tuning, there are lots of parameters to adjust depending on what you are trying to achieve. Storage systems are always a trade of between data integrity and speed. ZFS defaults to data integrity.

 

[wipeout_dude]

I'll give is a try.. Don't have a hardware RAID card in the test box but should still be able to get a relative throughput comparison..

You never know, maybe Linux and FreeBSD NFS implementations don't like each other too much..

 

[Erk]

I'll give is a try.. Don't have a hardware RAID card in the test box but should still be able to get a relative throughput comparison..

You never know, maybe Linux and FreeBSD NFS implementations don't like each other too much..

When you do relative tests from a VM vs the host, you have to remember with the VM that you are passing though multiple i/o subsystems with many variables. What you need to work our it exactly how much throughput you need for the VM, and it's nature, is it manly reads or writes, small or large files etc. and tune accordingly.

If I do a test with dd to local storage /var/lib/vz on the proxmox server, verses a test from a VM that uses a disk image located on /var/lib/vz as, I get 88MB/s from the host and on 48MB/s from the guest. No ZFS or other shared file system involved in that test. So I see a significant reduction in write speed from the guest which was Scientific Linux 6.2 32bit. The test server is not powerful, no raid etc. so the absolute figures you can ignore, but look at the relative difference.

 

[e100]

drbd is the way go...if you have the posibility to make a real life test...

I agree with macday, here is a benchmark test from within a Windows VM on top of DRBD: http://forum.proxmox.com/attachment.php?attachmentid=777
We have Areca 1880ix-12 4GB Cache RAM, 12 7200RPM SATA disks, 10GB Infiniband for DRBD replication network. (used Infiniband gear is inexpensive)

DRBD has very little overhead, especially if you have a RAID card with battery backed write cache.
Typically the limitation on write speed on DRBD is the replication network, not the underlying storage.

 

[wipeout_dude]

If I do a test with dd to local storage /var/lib/vz on the proxmox server, verses a test from a VM that uses a disk image located on /var/lib/vz as, I get 88MB/s from the host and on 48MB/s from the guest. No ZFS or other shared file system involved in that test. So I see a significant reduction in write speed from the guest which was Scientific Linux 6.2 32bit. The test server is not powerful, no raid etc. so the absolute figures you can ignore, but look at the relative difference.

Thanks for the relative numbers.. You have illustrated the problem I am trying to solve.. I was under the impression that KVM ran at about ~90% of bare metal but in terms of IO it appears its closer to ~50%..

This may be the way it is and not much can be done about it but hopefully its can be significantly improved..

 

[wipeout_dude]

Yes, I know DRBD is used a lot and works well.. My issue with it is that its limited to two nodes.. Ultimately I am looking at a ProxmoxVE cluster of more than 2 nodes and need live migration so a central shared storage AFAIK is the only way to achieve that..

If there is a way to achieve live migration without central storage I am all ears..

 

[Erk]

Yes, I know DRBD is used a lot and works well.. My issue with it is that its limited to two nodes.. Ultimately I am looking at a ProxmoxVE cluster of more than 2 nodes and need live migration so a central shared storage AFAIK is the only way to achieve that..

If there is a way to achieve live migration without central storage I am all ears..

Sheepdog looks like it might do that. Still not a solution for a DR site joined via a WAN, but good for the LAN in a single datacenter. Sheepdog is in the pvetest kernel, not quite ready yet, but you can play.

Here is some info https://github.com/collie/sheepdog/wiki/Getting-Started
http://www.osrg.net/sheepdog/

I don't see DRBD or Sheepdog as an answer to the I/O bottleneck issue you raised.