Using ZFS in VM on ZVOL (Nesting ZFS)

[pret]

Hi all,

I am currently running two PVE-Hosts with consumer SATA SSDs (ZFS Mirror). I am planing to move to larger enterprise (SATA) SSDs on both hosts (my other hardware isn't yet supporting NVME so I stick to SATA).

Since I am running a separate file server and I don't need a lot of storage (about 1 TB is enough) I would like to virtualize my fileserver to save some energy.

I want to use ZFS inside my Fileserver (due to the snapshot capabilities). The question is if it is a bad idea to use ZFS inside a VM on an ZVOL since that means double checksuming etc. Does it have large impact on (especially write) performance and do you have any recommendation for the settings of the outer ZVOL and the inner ZPOOL/Datasets (ashift,blocksize,compression,etc.)?

cheers,
pret

 

[LnxBil]

Technically possible, yet with the problems and benefits you described. It is a way to go, especially with Samba and Snapshots for Windows hosts.
Concerning the settings: I'd go with host zvolblocksize = guest ashift size. Compression will be much better inside of your guest, due to the default recordsize of 128k, 8k volblocks are seldomly compressible to less than 4k. I you want to get the most compressability, go with ashift=9 inside of your guest. This will optimize for size, yet will have read and write amplification. You normally cannot optimize for everything, so you need to choose.

 

[alexskysilk]

I want to use ZFS inside my Fileserver (due to the snapshot capabilities). The question is if it is a bad idea to use ZFS inside a VM on an ZVOL since that means double checksuming etc. Does it have large impact on (especially write) performance and do you have any recommendation for the settings of the outer ZVOL and the inner ZPOOL/Datasets (ashift,blocksize,compression,etc.)?

easiest way to avoid all that is to run your fileserver on the host.

 

[Dunuin]

And yes, ZFS got lots of overhead and overhead of nested filesystems is multiplying and not adding up. So performance will exponentially drop and SSD wear will exponentially increase. So you better avoid running nested ZFS unless you really need its features in the guestOS and you are willing to accept the additional overhead.

I personally prefer to passthrough my disks and only use ZFS on the guest level to prevent nesting. But passthrough comea with its own downsides and limitations.

 

[pret]

And yes, ZFS got lots of overhead and overhead of nested filesystems is multiplying and not adding up. So performance will exponentially drop and SSD wear will exponentially increase. So you better avoid running nested ZFS unless you really need its features in the guestOS and you are willing to accept the additional overhead.

I personally prefer to passthrough my disks and only use ZFS on the guest level to prevent nesting. But passthrough comea with its own downsides and limitations.

I first did not want to to use passthrough since this prevents migrating the VM from one host to another. But after thinking about it a downtime of the fileserver VM would not be an issue to me. I will try it with passthrough. This approach has another advantage - I don't need large SSDs in all of my hosts .

 

[LnxBil]

I first did not want to to use passthrough since this prevents migrating the VM from one host to another. But after thinking about it a downtime of the fileserver VM would not be an issue to me. I will try it with passthrough. This approach has another advantage - I don't need large SSDs in all of my hosts .

With passthrough and multiple machines, you will have a sync problem.

 

[pret]

With passthrough and multiple machines, you will have a sync problem.

Isn't that what I have written?

 

[LnxBil]

Isn't that what I have written?

Sorry, yes. After reading it once more, it is clear now. I must have misunderstood the comment.

You could, however, build resilient services on both machines on which the content of the VMs (the actual data) is synchronized. In the end, with two servers and without a shared storage, it is always a lot of work and only failover with higher probability of data loss.