We have a ProxMox 5.1 instance running against OpenMediaVault (OMV) 3.x using ZFS over iSCSI using the iscsitarget plugin on OMV.
Unfortunately, we have found that something in the overall setup doesn't support iSCSI's "DISCARD" ioctl(),
and hence we are unable to run "fstrim" operations against the VM's Disk images on the NAS to keep down the overall disk space usage within the ZFS "pool".
It appears that the OMV iscsitarget plugin is not available for OMV 4.x and may not be reworked completely until OMV 5.x - hence the use of the older OMV 3.x.
We believe that the "discard" TickBox in ProxMox's SCSI Disk Configuration settings for a VM's Disk effectively makes that ioctl() available to the VM,
and passes on the request to the iSCSI target whenever requested by the VM, rather than the ProxMox Host actively performing "discard" operations on the VM's
Disks whenever the Host feels like it. Is that a reasonable summary of its effect?
Hence, if the ZFS/iSCSI NAS properly supports "discard", one still needs to either mount each partition within each VM with it's own "discard" option set so as to
effect a discard operation on every file delete operation within the filesystem, ( generally not recommended ), OR run a Cron Job every so often against each filesystem,
( the preferred option ), so as to keep disk space under control.
This post below indicates that others have found similar problems with a lack of "DISCARD" capability using the iscsitarget service based on IETD on OMV:
forums.servethehome.com/index.php?threads/a-decent-iscsi-target-vm-with-unmap-discard-support.14217/
Whilst answers to that post refer to more modern kernels' use of "LIO",
en.wikipedia.org/wiki/LIO_%28SCSI_target%29
this obviously doesn't appear to correspond to any currently available plugin within OMV.
Along the way, we found the "lsblk -D" command, which when run against the ZFS "partitions" on the NAS indicated that they WERE "DISCARD" capable,
so the "local to the NAS" ZFS/SCSI drivers are obviously "DISCARD" capable.
We've also found that restoring a VM from backup also seems to effectively "Thicken" the disk image as it restores across onto the NAS.
However, it's unclear at present whether that's because ProxMox's restore operation doesn't always takes care to preserve the "Thinness" when
performing Restore and/or Clone operations, or it's simply another side-effect of the missing functionality on the NAS's iSCSI drivers.
Bearing all of the above in mind:
1. Is the apparent lack of "DISCARD" capability within OMV 3.x, ( and/or IETD in general ) a known issue?
2. Is there a plan to rewrite the iscsitarget plugin for OMV to use LIO instead of IETD, which presumably would fix the DISCARD issue?
3. What recommendations do the ProxMox community have for ZFS/iSCSI NAS Hardware/Software which does support DISCARD?
4. If the ZFS/iSCSI NAS DID support the DISCARD option, do all of ProxMox's Restore/Clone operations properly preserve the "Thinness" of ZFS/iSCSI
based VMs.
Thank you in advance for any help.
G.
Unfortunately, we have found that something in the overall setup doesn't support iSCSI's "DISCARD" ioctl(),
and hence we are unable to run "fstrim" operations against the VM's Disk images on the NAS to keep down the overall disk space usage within the ZFS "pool".
It appears that the OMV iscsitarget plugin is not available for OMV 4.x and may not be reworked completely until OMV 5.x - hence the use of the older OMV 3.x.
We believe that the "discard" TickBox in ProxMox's SCSI Disk Configuration settings for a VM's Disk effectively makes that ioctl() available to the VM,
and passes on the request to the iSCSI target whenever requested by the VM, rather than the ProxMox Host actively performing "discard" operations on the VM's
Disks whenever the Host feels like it. Is that a reasonable summary of its effect?
Hence, if the ZFS/iSCSI NAS properly supports "discard", one still needs to either mount each partition within each VM with it's own "discard" option set so as to
effect a discard operation on every file delete operation within the filesystem, ( generally not recommended ), OR run a Cron Job every so often against each filesystem,
( the preferred option ), so as to keep disk space under control.
This post below indicates that others have found similar problems with a lack of "DISCARD" capability using the iscsitarget service based on IETD on OMV:
forums.servethehome.com/index.php?threads/a-decent-iscsi-target-vm-with-unmap-discard-support.14217/
Whilst answers to that post refer to more modern kernels' use of "LIO",
en.wikipedia.org/wiki/LIO_%28SCSI_target%29
this obviously doesn't appear to correspond to any currently available plugin within OMV.
Along the way, we found the "lsblk -D" command, which when run against the ZFS "partitions" on the NAS indicated that they WERE "DISCARD" capable,
so the "local to the NAS" ZFS/SCSI drivers are obviously "DISCARD" capable.
We've also found that restoring a VM from backup also seems to effectively "Thicken" the disk image as it restores across onto the NAS.
However, it's unclear at present whether that's because ProxMox's restore operation doesn't always takes care to preserve the "Thinness" when
performing Restore and/or Clone operations, or it's simply another side-effect of the missing functionality on the NAS's iSCSI drivers.
Bearing all of the above in mind:
1. Is the apparent lack of "DISCARD" capability within OMV 3.x, ( and/or IETD in general ) a known issue?
2. Is there a plan to rewrite the iscsitarget plugin for OMV to use LIO instead of IETD, which presumably would fix the DISCARD issue?
3. What recommendations do the ProxMox community have for ZFS/iSCSI NAS Hardware/Software which does support DISCARD?
4. If the ZFS/iSCSI NAS DID support the DISCARD option, do all of ProxMox's Restore/Clone operations properly preserve the "Thinness" of ZFS/iSCSI
based VMs.
Thank you in advance for any help.
G.