Cache: Performance

[mir]

Hi,

The wiki states the following:

Disk Cache

Note: The information below is based on using raw volumes, other volume formats may behave differently.

cache=none seems to be the best performance and is the default in Proxmox 2.X.
cache=directsync seems similar in performance to cache=none
cache=writethrough should help speed up reads since it can use the host disk cache for reads but in practice it performs worse than cache=none
cache=writeback should help with reads and writes at the risk of loosing data, in practice it performs worse than cache=none.
caching adds additional data copies and bus traffic causing it to perform worse.

When the host uses software RAID the answer is completely different:
cache=none seems to be the worst performance wise
cache=directsync seems to increase read and write performance in average with 40%
cache=writethrough seems to increase read and write performance in average with 50%
cache=writeback seems to increase read performance in average with 50% and write performance in average with 100%

For this reason I would recommend writethrough for anybody using RAID.

Using iozone3 with these options: iozone -s64K -s128K -s256K -s512K -s1024K -s 2048K -s4096K -i 0 -i 1 -i 2

Anyone else with the same outcome?

PS. if anyone is interested I have the detailed reports in Excel format.

 

[mir]

I forgot this important information: The KVM guest is placed on a SAN using iSCSI.

Another interesting article even suggest using writeback if the kernel is latest Redhat 6.2 which is used on Proxmox. Can anybody confer to this?
http://www.ilsistemista.net/index.p...s-on-red-hat-enterprise-linux-62.html?start=2

 

[kbrault]

I would be interested in the excel file. Thanks.

You recommend writethrough but if I am reading your post correctly writeback provides the best increase. Is this correct? Why not recommend writeback?

 

[mir]

I would be interested in the excel file. Thanks.

You recommend writethrough but if I am reading your post correctly writeback provides the best increase. Is this correct? Why not recommend writeback?

I will try attaching a zipped tarball of the reports to this message and hope it will not be to big. Attachments of gizziped tarballs are not allowed so you will have to give me your email address.

The reason for recommending writethrough instead of writeback is caused be the fact that using writeback could lead to loss of data but if the article I linked to in the second message is correct in its assumption that the reason kernel for RHEL 6.3 has added an algorithm that garanties against data loss using any of the cache settings then of course you should used writeback but since this is against any recommendations from the proxmox team I hope the team will comment on the article I have linked to.

So to cut it down:
1) writethrough seems to perform 50% better than the nocache option for KVM over iSCSI (presumeably the same goes with fibre channel).
2) writeback seems to increase write performance with extra 50%.
3) writethrough garanties against data loss
4) according to the proxmox team writeback is not safe since you could loose data.

So making a safety trade-off suggest using writethrough. YMMV of course, but I prefer being safe than sorry

 

[charnov]

Of course there is also the question of where the cache is or stacking cache... do you set it in Proxmox or do you set it on the SAN or both?

 

[mir]

Of course there is also the question of where the cache is or stacking cache... do you set it in Proxmox or do you set it on the SAN or both?

I set this option in proxmox. For each disk assigned to a kvm you can choose cache option.