Help Understanding Filesystem interactions between Proxmox & VMs

[DJT]

I'm struggling to wrap my head around filesystems and how they're handled between PVE and the various VMs which access them. I'm very new to this and am still trying to wrap my head around things, so I hope you'll excuse my ignorance. Any guidance would be greatly appreciated.

I'm trying to take advantage of ZFS as much as possible. I have PVE installed on its own zpool and have set up additional zpools and ZFS file systems to use for shared storage and for each of my VMs. E.g:

NAME	DESCRIPTION
rpool	PVE installation zpool (default)
rpool/ROOT	PVE root file system (default)
rpool/data	PVE data (default)
share1	zpool for shared storage
share1/backup	storage for backup data
share1/iso-store	ISO storage for VMs
share1/vm1-home	home partition for first virtual machine
vm1	zpool for first virtual machine
vm1/efi	EFI partition for first virtual machine (VM1)
vm1/root	root FS for VM1
vm1/swap	SWAP partition for VM1

Say for example VM1 is an Ubuntu Virtual Machine which I want to run PLEX on. I set up the zpool vm1 on an SSD, with share1 on rotary storage, with the intention of dumping backups to share1/backup and accessing media from share1/vm1-home. I then create the necessary ZFS file systems within those zpools and pass them through as required when configuring the VM in PVE.

When running the Ubuntu installation, it will ask me how I want to format filesystems on the devices it sees passed through from PVE. So, I might end up with Ubuntu writing an ext4 / LVM / SWAP filesystem onto the underlying ZFS filesystem that's passed through from PVE.

Is that a problem or is that just normal behaviour when using a virtualised environment? Are there any pitfalls I should avoid?

My main concerns are:

1. Will ZFS keep doing its job regardless of what data is being written on top of it by the VMs, or is this likely to cause conflict / data corruption?
2. What is the best way to handle backups for the data? Is backing up the individual ZFS filesystems from PVE sufficient, or should I get each VM to write backups for its own files? I presume writing from PVE would be garbled nonsense with filesystem data from the VM's OS tied up in it?
3. Is using a ZFS FS configured in PVE as SWAP for a VM a terrible idea? It seems like it would be; if so, can anyone please outline a better way of implementing it?

Again, please excuse my ignorance. I've worked through a lot of man documentation but it's hard to find guidance on points like these when it's abstracted by having a hypervisor and virtualisation involved as well. Any guidance or resources for where I can read up more about this sort of thing would be much appreciated.

Thanks.

 

[aaron]

ZFS can provide storage in two ways. It can have a regular file system that we use for Proxmox VE itself when installed on ZFS, containers and for whatever else you need.

The other one is a zvol. This will have a block device instead of a file system and still offer the ZFS goodies of snapshots, checksums, etc.
When setting up a VM you will see that the vm-XXX-disk-0 subvol exists when running zfs list but it will not be present in the file system structure.

Within that zvol that is presented as a block device, the VM will use the partition scheme and file systems with which you install it. This means that you cannot look into the VMs file system easily from outside.

Additionally, you cannot just share a local directory with many VMs at the same time. For this you will need something like a network share (NFS, Samba/CIFS) or some other means.

If you run LXC containers on a ZFS storage you will be able to access it's file system directly.

I hope this cleared up a few things.