Caching when using ZFS storage?

[danb35]

Running Proxmox VE 4.0-57 on ZFS, and I've created local ZFS storage for my VMs, so they're using zvols for their virtual disks. How should cache be set for those? The default is no cache, but other options are write back, write through, and direct sync. I don't see a page on the wiki that addresses this, but it's possible I missed one. Any thoughts, or good resources?

 

[mir]

For ZFS over iSCSI tests shows nocache is given best performance in my setup but this depends on pool config, iscsi/local, guest file system, guest OS, disk emulation, and disk controller emulation. So to be absolute sure you need to run your own tests. I can recommend fio.

 

[danb35]

Thanks for the info. I was kind of hoping there would be a "80+% of the time, you want X" answer, but it sounds like that isn't to be.

 

[Q-wulf]

Thanks for the info. I was kind of hoping there would be a "80+% of the time, you want X" answer, but it sounds like that isn't to be.

Worst case scenario you can just benchmark it, then publish your results.

There is only a "small" amount of variables

 

[Nemesiz]

Running Proxmox VE 4.0-57 on ZFS, and I've created local ZFS storage for my VMs, so they're using zvols for their virtual disks. How should cache be set for those? The default is no cache, but other options are write back, write through, and direct sync. I don't see a page on the wiki that addresses this, but it's possible I missed one. Any thoughts, or good resources?

If you are using local ZFS then ZFS already to caching in ARC and L2ARC.

 

[danb35]

Thanks. I'm familiar with ARC and L2ARC (the latter of which I don't have on my system, at least at this time), but those are read caches, and the options seem to deal with write cache.

 

[Erk]

The Wiki says to change cache to write back

https://pve.proxmox.com/wiki/Storage:_ZFS

 

[danb35]

True, but the example in the wiki is when a file is being used as a virtual disk. In that case, using no cache will give an error, since ZFS doesn't support the O_DIRECT flag. But when using zvols for storage, this issue doesn't arise.