Scratch Space use ZFS or LVM-Thin

[jena]

Hi everyone,

I was wondering if I could get some suggestion for setting up temporary scratch space for non-critical data.
Best balanced I/O performance (typical file size ranges 50k ~ 1MB each) is the priority.

VMs (volblocksize = 16k) are all on ZFS-mirrored enterprise U.2 SSD.
But the enterprise SSD are very expensive and I don't want users to stuff their VM disks unnecessarily big or having habits storing everything on VM disks.

Occasionally, some user wants to store some non-critical data (intermediate results, can always be re-run).
So I added a pair of 2TB Samsung 980 Pro (PCIE 4.0).

I am testing this configuration:
ZFS - stripped (RAID0)
volblocksize = 16K

zpool create -f -o ashift=12 m2pool [nvme1] [nvme2] zfs create -o primarycache=metadata -o compression=lz4 -o sync=disabled -o encryption=aes-128-gcm -o XXXX m2pool/vol16k

Encryption has to be ON to satisfy some rules.
My previous testing (with a bit different configuration) shows no significant speed differences with encrypted zfs dataset.

fio --name=random-write --ioengine=sync --iodepth=4 \
                              --rw=randwrite --bs=4k --direct=0 --size=256m --numjobs=16 \
                              --end_fsync=1

2 x Kioxia CD6-R 7.68T, mirror ZFS
    volblocksize 32k no encryption
    Run status group 0 (all jobs):
    WRITE: bw=668MiB/s (701MB/s), 41.8MiB/s-43.7MiB/s (43.8MB/s-45.8MB/s), io=4096MiB (4295MB), run=5861-6131msec

    volblocksize 32k encryption
    Run status group 0 (all jobs):
    WRITE: bw=705MiB/s (739MB/s), 44.0MiB/s-44.2MiB/s (46.2MB/s-46.4MB/s), io=4096MiB (4295MB), run=5787-5812msec

I think ZFS in RAID 0 can still detect data corruption and just cannot recover from the corruption. Is this understanding correct?
I know there is no redundancy protection, but the user can always re-run the data again.

Any comments or suggestion?

 

[leesteken]

I think ZFS in RAID 0 can still detect data corruption and just cannot recover from the corruption. Is this understanding correct?

Yes, it can detect corrupted data from disk but cannot correct it. Except for important meta-data that is stored multiple times, if the corruption is small and localized.

 

[Dunuin]

You might also want to try LUKS encrypted LVM-Thin striped across several disks. Doesn't got bit rot detection or blocklevel compression but SSD wear should be lower and performance better (because ZFS got massive overhead).