The first zfs, so sanity check about performance

[czechsys]

Testing a new server with our the first zfs setup. Just asking for sanity check for default setup performance of the nvme zfs because we are thinking about adding hw raid card. Main purpose of the server will be GPU AI computing.

CPU: AMD EPYC 9354 32-Core Processor
RAM: 512 GB
Disks: all connected to MB directly, 2x Kinsgton DC600M SATA, 2x Samsung PM9A3 960GB NVMe PCIe4x4
ZFS pools:
ARC=16GB
mirror (raid1)
ashift=9
compression=off
sync=yes
dedup=off

Test used with our standard fio parameters from root mount of the pools directly on the PVE.

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randread
Kingston: read limited by 1 core 100% utilization
  read: IOPS=72.9k, BW=285MiB/s (299MB/s)(4096MiB/14376msec)
   bw (  KiB/s): min=277128, max=357160, per=98.13%, avg=286301.04, stdev=16450.78, samples=28
   iops        : min=69282, max=89290, avg=71575.25, stdev=4112.68, samples=28

Samsung: read limited by 1 core 100% utilization
  read: IOPS=72.9k, BW=285MiB/s (299MB/s)(4096MiB/14386msec)
   bw (  KiB/s): min=277744, max=357440, per=98.10%, avg=286001.71, stdev=15898.45, samples=28
   iops        : min=69436, max=89360, avg=71500.43, stdev=3974.61, samples=28


fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randwrite
Kingston:
  write: IOPS=6263, BW=24.5MiB/s (25.7MB/s)(4096MiB/167410msec); 0 zone resets
   bw (  KiB/s): min=18000, max=170544, per=97.73%, avg=24485.87, stdev=13028.94, samples=334
   iops        : min= 4500, max=42636, avg=6121.47, stdev=3257.24, samples=334

Samsung:
  write: IOPS=19.8k, BW=77.5MiB/s (81.2MB/s)(4096MiB/52872msec); 0 zone resets
   bw (  KiB/s): min=51768, max=314496, per=98.57%, avg=78197.41, stdev=33727.23, samples=105
   iops        : min=12942, max=78624, avg=19549.35, stdev=8431.81, samples=105


fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
Kingston: read limited by 1 core 100% utilization
  read: IOPS=18.6k, BW=72.6MiB/s (76.1MB/s)(3070MiB/42309msec)
   bw (  KiB/s): min=49592, max=205224, per=100.00%, avg=74304.10, stdev=29527.13, samples=84
   iops        : min=12398, max=51306, avg=18576.02, stdev=7381.78, samples=84
  write: IOPS=6208, BW=24.2MiB/s (25.4MB/s)(1026MiB/42309msec); 0 zone resets
   bw (  KiB/s): min=17456, max=68800, per=100.00%, avg=24831.43, stdev=9797.99, samples=84
   iops        : min= 4364, max=17200, avg=6207.86, stdev=2449.50, samples=84

Samsung: read limited by 1 core 100% utilization
  read: IOPS=53.5k, BW=209MiB/s (219MB/s)(3070MiB/14700msec)
   bw (  KiB/s): min=167040, max=356488, per=98.60%, avg=210856.00, stdev=38356.78, samples=29
   iops        : min=41760, max=89122, avg=52714.00, stdev=9589.20, samples=29
  write: IOPS=17.9k, BW=69.8MiB/s (73.2MB/s)(1026MiB/14700msec); 0 zone resets
   bw (  KiB/s): min=55512, max=117584, per=98.62%, avg=70488.55, stdev=12589.05, samples=29
   iops        : min=13878, max=29396, avg=17622.14, stdev=3147.26, samples=29