Best way to store VM's on zfs - on the pool or in datasets?

[bytesonend]

Hi

On my PVE host, I have a 2-disk SSD mirrored zfs pool dedicated to VM's, CT's, ISO's, etc. called pvestorage. Now I'm trying too figure out whether to store my VM's directly on the zpool or in subsequent datasets like pvestorage/vms/vm1 ,/vm2, etc.

So I've tried to read up on the topic on PVE Storage, on the PVE forum and elsewhere, and recommendations seams to differ. So far I've only managed to get myself more confused the more I've read...

So if someone could be so kind to shed a little light, touching on the points below, it would be most appricited:
1) What are the pros and cons of storing my VM's either directly on the pool vs. in a dataset, and is it correct that only storage directly on the pool support snapshots?
2) How will PVE store the VM images under the hood in the two solutions and is it correct that when stored VM's on datasets, PVE will create a zvols?
3) Are there any significant I/O performance difference between the two solutions?
4) Can pools and/or datasets be added to PVE in different ways (or rather, as different storage types) and will this make a difference? If this a nonsense question just overlook it

As to my needs, I don't need blazing fast I/O performance but being able to utilize zfs incremental snapshots functionality would definitely be nice...

Thanks in advance.

 

[Dunuin]

1) What are the pros and cons of storing my VM's either directly on the pool vs. in a dataset, and is it correct that only storage directly on the pool support snapshots?

Parent datasets inherit configurations to child datasets/zvols. And you can do stuff recursive so it is a good idea to group stuff in datasets so it is easier to manage stuff. You can snapshot every zvol/dataset whereever it resides.
Lets say your pool is MyPool and you create two datasets "important" and "unimportant" on that pool. Now you create 3 VMs in both datasets so it looks like this:

MyPool
MyPool/important/vm-100-disk-0
MyPool/important/vm-101-disk-0
MyPool/important/vm-102-disk-0
MyPool/unimportant/vm-200-disk-0
MyPool/unimportant/vm-201-disk-0
MyPool/unimportant/vm-202-disk-0

Lets say you want to encrypt your important VMs. In that case you could just encrypt "MyPool/important" and all newly created VMs inside that dataset will be encrypted too while all VMs in "MyPool/unimportant" stay unencrypted.
Or you want to create a snapshot of all unimportant VMs. In that case you could run zfs snapshot -r MyPool/unimportant@test to do a recursive snapshot. This will create snapshots of the "unimportant" dataset and all of its child zvols:
MyPool/unimportant@test
MyPool/unimportant/vm-200-disk-0@test
MyPool/unimportant/vm-201-disk-0@test
MyPool/unimportant/vm-202-disk-0@test
(but its not that important for snapshots because you should use the pve commands instead of native zfs commands for that and I don't think the pve commands can create recursive snapshots).

2) How will PVE store the VM images under the hood in the two solutions and is it correct that when stored VM's on datasets, PVE will create a zvols?

Yes, only LXCs will use datasets as storage. VMs will always use zvols because they need blockdevices because of how virtualization works.
Its no problem to use several datasets on the same pool as VM storages but you need to add every dataset as its own storage.

3) Are there any significant I/O performance difference between the two solutions?

There shouldn't be any difference in general. But using datasets has the potencial to be faster because it is easier to optimize stuff like sync, encryption, compression, volblocksize, recordsize and so on.

4) Can pools and/or datasets be added to PVE in different ways (or rather, as different storage types) and will this make a difference? If this a nonsense question just overlook it

On a ZFS pool you can only store VMs and LXCs. If you want to store stuff like ISOs, Backups and so on on that pool you could create a dataset and use its mountpoint as a storage of type "Directory". That way it is possible to store ISOs and so on on that dataset.

 

[bytesonend]

Hi Dunuin

Thanks a lot for your reply...