Problems following proxmox documentation about ZFS benchmarking

[H25E]

Hello everybody,

I want to benchmark my ZFS pools in a server that it's already running with data (mostly testing, but I don't want to lose it). For that purpose I have found this, a ZFS benchmark done by proxmox team. So it looks like a great fit.

They are using fio command like that: fio --ioengine=psync --filename=/dev/zvol/HDD/fio --size=1M --time_based --name=fio --group_reporting --runtime=600 --direct=1 --sync=1 --iodepth=1 --rw=read --threads --bs=4K --numjobs=1 --debug=io

But i get:

fio: set debug option io
fio: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process
fio: Laying out IO file (1 file / 1MiB)
io       21508 declare unneeded cache /dev/zvol/HDD/fio: 0/1048576
fio: looks like your file system does not support direct=1/buffered=0
fio: destination does not support O_DIRECT
io       21545 io_u 0x5571a02d4f40, setting file failed
io       21545 get_io_u failed
io       21545 io_u_queued_complete: min=0
io       21545 getevents: 0
fio: No I/O performed by psync, perhaps try --debug=io option for details?
fio: pid=21545, err=22/file:filesetup.c:701, func=open(/dev/zvol/HDD/fio), error=Invalid argument
io       21545 close ioengine psync
io       21545 free ioengine psync


Run status group 0 (all jobs):
io       21508 ioengine sg unregistered
io       21508 ioengine mtd unregistered
io       21508 ioengine gfapi_async unregistered
io       21508 ioengine gfapi unregistered
io       21508 ioengine rbd unregistered
io       21508 ioengine rados unregistered
io       21508 ioengine splice unregistered
io       21508 ioengine e4defrag unregistered
io       21508 ioengine falloc unregistered
io       21508 ioengine posixaio unregistered
io       21508 ioengine rdma unregistered
io       21508 ioengine libaio unregistered
io       21508 ioengine filecreate unregistered
io       21508 ioengine ftruncate unregistered
io       21508 ioengine net unregistered
io       21508 ioengine netsplice unregistered
io       21508 ioengine null unregistered
io       21508 ioengine sync unregistered
io       21508 ioengine psync unregistered
io       21508 ioengine vsync unregistered
io       21508 ioengine pvsync unregistered
io       21508 ioengine pvsync2 unregistered
io       21508 ioengine mmap unregistered
io       21508 ioengine cpuio unregistered

Why I am getting this error instead of the benchmark?

Also, (I know it's a little bit late to ask this but well...) it's this command going to lead to data loss? At some point they say Note: This command will destroy any data on your disk., but it's for a former fio command that works directly on the disk instead of on the zpool. They don't say anything for the last ones, so I prefer to ask.

 

[H25E]

Found it.

afaics, the tests were all done on a zvol in the zpool not on a file

You can delete the thread if you think it's useless.

 

[Dunuin]

Did you also followed the paper and disabled the cache? Otherwise you are just benchmarking your RAM, not the disks, when doing read tests. zfs set primarycache=none /dev/zvol/HDD/fio

 

[H25E]

They use metadata in the paper

 

[H25E]

I have used primarycache=metadata as in the paper but I'm hitting rates over 1GB/S with a mirror of hard drives, so I am hitting some kind of cache but I don't know how. Rate is over 1GB/S all the time for the whole 10 min with read and randread modes.

After setting primary and secondary cache to none problem has been solved.