Best Practise: 'VM's running in ZFS file system or 'Directory'

[gb00s]

HI,

I'm fairly new to Proxmox and ZFS. As you can see in the screenshot below, I have set up 2x pools named 'storage' and 'vm'. In each pool, I have set up files systems with different configs for each filesystem.


In Proxmox my place to store backups is set up just as a directory under the 'storage' pool and backups are placed in the dump folder. If I'm not mistaken as a ZFS newbie, each vm is a single 'block' device/file on it's won. Correct? All vm's are set up as raw devices/files. Now as I realized I may have set up the wrong block size for the zvol's to store the virtual machines (1024K) I'm asking myself what is the best practice to 'store' the virtual machines and backup the same. Is it better to store the vm's and run them from a zvol or do I set up a directory in Proxmox under the pool and store the vm's there? Which way is to better in terms of performance and/or running and maintaining backups?

Additionally, related to backups what would be the best practice to backup each vm in a zvol? Snapshots or replication?

Any share of thoughts is appreciated. If you need more, please let me know. Thank you in advance.

 

[ca_maer]

This is highly depending on your disk configuration but for our case we have a single stripped mirror zpool with all same size disk (which is also the root pool for Proxmox). When creating a CT/VM we select the only storage available: local-zfs. Never had an issue. Keep it simple.

The advantage of having Proxmox create/configure your main pool is that all the pool settings are configured correctly or are the default and most of the time you don't have to change anything.

For the backups, snapshots and replication are not backups. While snapshots could be used as backup I wouldn't rely too much on them. Replication is more for HA scenarios. The best is either use the built-in Proxmox backup feature or use a third party tool like Veam. For CT, it's different since they are mounted to a path on the host (as seen in zfs list) so you could snapshot the running CT, mount the snapshot and rsync it somewhere else.

 

[LnxBil]

If I'm not mistaken as a ZFS newbie, each vm is a single 'block' device/file on it's won. Correct?

Yes, each VM is on a volume (zvol), each LX(C) container is on a filesystem.

Is it better to store the vm's and run them from a zvol or do I set up a directory in Proxmox under the pool and store the vm's there?

The way PVE sets up most things is already the optimal way. You have per VM snapshots (and replication) and a linear snapshot history. If and only if you would use qcow2, you would gain non-linear snapshot history.

Which way is to better in terms of performance and/or running and maintaining backups?

zvol is faster, because there are less storage layers in between. Backups are the same if they are done with the built-in vzdump.

mount the snapshot and rsync it somewhere else.

or just use ZFS send/receive. rsync is only for the non-zfs world and has absolutely no place in the zfs-world.

 

[ca_maer]

or just use ZFS send/receive. rsync is only for the non-zfs world and has absolutely no place in the zfs-world.

I agree that ZFS send/receive is a valid option but saying rsync has no places in the zfs-world is far fetched. What if my backup target is a CIFS share and I can't use ZFS send/receive ? The rsync option is only valid for CT since the filesystem is already mounted on the host. It's then pretty easy to rsync all the files to the CIFS share. You just need to make sure to exclude folders like dev,proc,mnt,sys,... if you are backuping the whole CT and not just some data that's on it. To restore juste create back your CT and rsync back the file in the mounted dir for the filesystem before powering it on.

 

[LnxBil]

I agree that ZFS send/receive is a valid option but saying rsync has no places in the zfs-world is far fetched. What if my backup target is a CIFS share and I can't use ZFS send/receive ? The rsync option is only valid for CT since the filesystem is already mounted on the host. It's then pretty easy to rsync all the files to the CIFS share. You just need to make sure to exclude folders like dev,proc,mnt,sys,... if you are backuping the whole CT and not just some data that's on it. To restore juste create back your CT and rsync back the file in the mounted dir for the filesystem before powering it on.

I understand, but in an ZFS world, you don't have non-ZFS systems as targets and you will not have this problem.

And you can always write the zfs send to a file on cifs and import in an ZFS later. And I don't rsync a whole container to CIFS, depending on the server implementation, you will corrupt everything (permissions, special flags, users, etc.).

 

[ca_maer]

I understand, but in an ZFS world, you don't have non-ZFS systems as targets and you will not have this problem.

If by zfs world you mean an env with 100% ZFS machine sure but it's not always the case.

And you can always write the zfs send to a file on cifs and import in an ZFS later.

Didn't think of that. Thanks !

And I don't rsync a whole container to CIFS, depending on the server implementation, you will corrupt everything (permissions, special flags, users, etc.).

Meant to say an NFS share, my bad, but done it a thousand times (including restores) without issue. Just make sure you exclude the following folders:

dev
run
mnt
sys
proc
tmp
media

My point is that there is no universal solution and that it highly depends on the overall infrastructure in place and what you are trying to backup which is why I said rsync could be a valid option for some.

 

[LnxBil]

My point is that there is no universal solution and that it highly depends on the overall infrastructure in place and what you are trying to backup which is why I said rsync could be a valid option for some.

Yes, that's correct. It gets easier if everything is ZFS and PVE also includes replication and stuff targeted specifically to ZFS.

 

[gb00s]

Hi, just wanted to thank you sharing your thoughts. Much appreciated and learning every day new things in Proxmox Have a nice weekend.