ZFS: VM disk as file vs VM disk in ZFS pool

[alexc]

I just installed PVE 5.0 on zfs to play with (used to use LVM before this moment), and I'm in doubt now: if I should create VM disks on zfspool pool (this will not be visible as e.g. *.raw file), or I can create VM disks as files under /var/lib/vz/images/<VM-number>/.

Latter way is easier to maintain these VM disk images (backup = copy of these images, duplicate = simple copy image etc), while I'd like to use this provisioning feature on ZFS and I don't know will this works if I will go with file approach.

 

[fabian]

please use the ZFSPoolPlugin with zvols.

 

[alexc]

Thank you Fabian,

but this is exactly the point I can't figure out. Here is the default storage.conf:

dir: local
        path /var/lib/vz
        content iso,vztmpl,backup

zfspool: local-zfs
        pool rpool/data
        sparse
        content images,rootdir

If I add 'images' to the content list of 'local' storage, then I'll be able to create/store VM disk images under /var/lib/vz/<vm-num>/ and as I create a disk image(s) there I'll be able to see it as *.raw file(s). This files can easily be copied by OS tools, no PVE may be involved.

But if I create a disk image on 'local-zfs' storage then I will not be able to manipulate these virtual disks without PVE involved. I will not be able to copy such a disk from OS prompt, so to say.

So now, if I keep my VM disks as a *.raw files on zfs-backed filesystem, what may be the drawback of this? Will this be slower that keep it on e.g. 'local-zfs', or may this be less secured/less robust?

 

[fabian]

Thank you Fabian,

but this is exactly the point I can't figure out. Here is the default storage.conf:

dir: local
        path /var/lib/vz
        content iso,vztmpl,backup

zfspool: local-zfs
        pool rpool/data
        sparse
        content images,rootdir

If I add 'images' to the content list of 'local' storage, then I'll be able to create/store VM disk images under /var/lib/vz/<vm-num>/ and as I create a disk image(s) there I'll be able to see it as *.raw file(s). This files can easily be copied by OS tools, no PVE may be involved.

But if I create a disk image on 'local-zfs' storage then I will not be able to manipulate these virtual disks without PVE involved. I will not be able to copy such a disk from OS prompt, so to say.

if you create a (VM) disk on the local-zfs storage, PVE will create a 'zvol'. this is exported as block device by ZFS, and you can access it just like any other block device.

So now, if I keep my VM disks as a *.raw files on zfs-backed filesystem, what may be the drawback of this? Will this be slower that keep it on e.g. 'local-zfs', or may this be less secured/less robust?

you lose snapshot, linked clone and replication support, and probably lose some performance on the way as well.

 

[alexc]

Thank you, now it is clear for me!

 

[alexc]

Fabian, just a word of caution for me: if I put use exported zvols as my block devices for VM disk, then I suspect I'll see fragmentation not o my VM disk only but on zvols as well, right?

I ask since I know zfs will become pretty slow over time for fragmentation rise. I use HDDs (now SSDs) as underling storage so this is serious for me.

Please tell, should I worry about this, or this is not a problem for PVE and PVE hosted VMs?