ZFS questions

[tjk]

Hey Folks,

Complete newbie to Proxmox, but coming from a VMware and Xen world, I have some questions on ZFS and setup for a home lab I am building.

My understanding of ZFS with Promox was that it was file based, from reading this.

In my server, I have 2xNVME drives, I created a ZFS mirror on them, but the only image types it supports are raw and from the CLI I can't see any .raw files, so I assume it is storing them in block format even though the wiki link above says file format?

Also, in some basic ioping testing from inside the VM sitting on ZFS, LVM, or NFS I am seeing the same response times of time=726.7 us, I would have expected to see much better latency sitting on local ZFS compared to local LVM/SAS or even NFS network.

Any links or feedback would be appreciated!

 

[Dunuin]

With a ZFS pool you got 2 types of storages...
1.) Datasets that are used for LXCs. These offer file level access.
2.) Zvols that are used for VMs. These are block devices and you can only use RAW.

If you really want file level access for VMs you could create a dataset, add it to PVE as a "Directory" storage and use that storage to save your virtual disks. That way you could for example use qcow2 image files for your virtual disks ontop of a ZFS dataset. But qcow2 are CoW and ZFS is CoW so I would think thats bad for SSD life expectation and performance.
Then it would look like this...
SSDs -> ZFS pool -> dataset -> image file (using qcow2) -> virtio SCSI -> guests filesystem like ext4/NTFS
...instead of this:
SSDs -> ZFS pool -> zvol (using RAW) -> virtio SCSI -> guests filesystem like ext4/NTFS

Also keep in mind that ZFS got a horrible write amplification (I did benchmarks and saw a write amplification between factor 3 and 81 depending on the workload). So you might want to monitor your NVMe SSDs wear using SMART to see if they can survive for long enough.

 

[t.lamprecht]

In my server, I have 2xNVME drives, I created a ZFS mirror on them, but the only image types it supports are raw and from the CLI I can't see any .raw files, so I assume it is storing them in block format even though the wiki link above says file format?

The sentence "ZFS is a combined file system and logical volume manager" from the sysadmin docs may be a better explanation:
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#chapter_zfs

ZFS essential provides both, datasets that can be mounted as file-system and ZVOL's that are essentially block storage.
We may want to actual change the table to "both" for ZFS.

If you configure a ZFS pool as Proxmox VE storage it will be used as FS for Container and ZVOLs for VM disks.

 

[tjk]

With a ZFS pool you got 2 types of storages...
1.) Datasets that are used for LXCs. These offer file level access.
2.) Zvols that are used for VMs. These are block devices and you can only use RAW.

If you really want file level access for VMs you could create a dataset, add it to PVE as a "Directory" storage and use that storage to save your virtual disks. That way you could for example use qcow2 image files for your virtual disks ontop of a ZFS dataset. But qcow2 are CoW and ZFS is CoW so I would think thats bad for SSD life expectation and performance.

Thank you! Yea, I searched the forum and saw you can overlay a dataset on top of ZFS but folks recommended against it, just thought I was missing something else since the Prox wiki page says file level on ZFS but doesn't talk about overlaying a dataset on top of it.

Sounds like the TLDR of it is use ZFS as ZVOL for best performance.

 

[tjk]

Oh, one more question...are their any built in stats in the Prox GUI for storage performance, like IO stats or read/write performance? The only one I see is capacity graphs.

 

[Dunuin]

No, but

Oh, one more question...are their any built in stats in the Prox GUI for storage performance, like IO stats or read/write performance? The only one I see is capacity graphs.

No, but if you want to see how much or how fast your SSDS are reading/writing you could run apt install sysstat on your host. Then iostat will run in the background and you can run iostat to see the average metrics since reboot. And zfs also got buildin stats you can access by zpool iostat