ZFS, cluster and datasets

[Springtime]

Hello,
can you please help me understand...
When I installed the PVE server, I created a ZFS pool via GUI.
Then I installed the 2nd PVE server and created the same named ZFS pool via GUI.
Then I clustered the both nodes.
This might have been my first error, not first to cluster and then create the ZFS pools?
Anyway, they do seem to work, as I have a ZFS-something storage called "VMs" in Datacenter/Storage and I can successfully migrate the virtual machines. Replication for the VM does work too.
I am aware that ZFS is in this case a storage-replication, not something like Ceph (HCI).
But what I am missing is twofold:
I have not created any datasets. Which would be cool. Can I do this without any issues, now post-initial-deployment? (create datasets on both nodes equally)
If I migrate the VM with the ISO attached, it will fail. Apparently, ISO Directory is not replicated, although the ZFS storage is.

If I do zpool list -v, I will only get one pool, raid-z2 and my 8 SSDs (per node).
If I do zfs pool list, then I get the mount /zfspool, which is my created pool, and then vm-disks directly in it.

So what can I do, to clean this up a little? Like create some datasets, on both nodes, will they replicate then? Can I move the VM disks into datasets. Have own dataset for ISOs, move ISOs there, also replicate?

Will then a VM that has an attached ISO be able to migrate without having to detach the ISO?

Thank you.

 

[Springtime]

I am one step further. I created separate datasets on the zfspool for isos and vm-disks, moved all appropriately. Seems to work fine.
But if I understand it correctly, a "Directory" can't be replicated. It can only be Shared and available on other nodes.
If I attempt to migrate the VM to a node that doesn't have the ISO, it won't migrate.
So I indeed have to upload the ISOs to all directories on all nodes, if I want to be able to migrate VMs with attached ISOs.
Is that correct?

 

[Kingneutron]

If you don't want to waste storage on duplicated ISOs, you can setup a Samba share and replace the cluster node's local copy with symlinks. You'll have to correct the mount on each VM that has the ISO mounted to point to the ISO in the default dir (/var/lib/vz), but thereafter shouldn't have to worry [TEST THIS]

https://github.com/kneutron/ansitest/blob/master/proxmox/symlink-samba-isos.sh

Rerun the script on each PVE node if you update/delete ISOs on the share // run it nightly in root's crontab

See comments in the script for more info, you will need to edit the script and tell it where things live for your Samba mount(s)

/ lots of other good stuff in that repo ;-)

Personal recommendation, it's probably a good idea to host your ISOs on a ZFS server with at least a mirror, so you get self-healing scrubs - and approx double the Read I/O. And always download your ISOs from torrent whenever possible