Proxmox VE ZFS Benchmark with NVMe

guletz

Famous Member
Apr 19, 2017
1,430
214
83
Brasov, Romania
Hi again,


It will be very intersting for me, if you can do only these tests:


1. (zfs 16 k)fio --filename=/nvme01/test2 --rw=randwrite --ioengine=libaio --bs=4k --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test2

2. (ext4) fio --filename=/mnt/nvme01/test2 --rw=randwrite --ioengine=libaio --bs=4k --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test2

During this tests:

- enable writes info with : echo 1 > /proc/sys/vm/block_dump
- then you can see how many blocks are affected, like this:

Code:
Feb 16 16:23:47 pv2 kernel: [16430.235270] z_wr_int(1576): WRITE block 1694519248 on sdc1 (24 sectors)
Feb 16 16:23:47 pv2 kernel: [16430.235490] z_wr_iss(1556): WRITE block 1577084816 on sdc1 (8 sectors)
Feb 16 16:23:47 pv2 kernel: [16430.235500] z_wr_iss(1566): WRITE block 1510030224 on sdc1 (8 sectors)
Feb 16 16:23:47 pv2 kernel: [16430.235579] z_wr_iss(1570): WRITE block 1560817504 on sdc1 (8 sectors)
Feb 16 16:23:47 pv2 kernel: [16430.235588] z_wr_int(1574): WRITE block 1510030232 on sdc1 (24 sectors)
Feb 16 16:23:47 pv2 kernel: [16430.235589] z_wr_int(1576): WRITE block 1560817512 on sdc1 (8 sectors)
Feb 16 16:23:47 pv2 kernel: [16430.235592] z_wr_iss(1568): WRITE block 1526807432 on sdc1 (32 sectors)
Feb 16 16:23:47 pv2 kernel: [16430.235679] z_wr_int(1577): WRITE block 1526807464 on sdc1 (16 sectors)
Feb 16 16:23:47 pv2 kernel: [16430.235718] z_wr_int(1577): WRITE block 1560817496 on sdc1 (8 sectors)
Feb 16 16:23:47 pv2 kernel: [16430.235741] z_wr_int(1574): WRITE block 1560817520 on sdc1 (8 sectors)
Feb 16 16:23:47 pv2 kernel: [16430.235759] z_wr_int(1578): WRITE block 1577084808 on sdc1 (8 sectors)
Feb 16 16:23:47 pv2 kernel: [16430.235783] z_wr_int(1574): WRITE block 1577084824 on sdc1 (24 sectors)
Feb 16 16:23:47 pv2 kernel: [16430.235810] z_wr_int(1577): WRITE block 1593862080 on sdc1 (40 sectors)
Feb 16 16:23:47 pv2 kernel: [16430.235850] z_wr_int(1579): WRITE block 1680651808 on sdc1 (32 sectors)
Feb 16 16:23:47 pv2 kernel: [16430.235924] z_wr_int(1577): WRITE block 1694519248 on sdc1 (136 sectors)

It would be intersting to see 1 sample log after each test(1 and 2). And see if are some WTITE blocks IO in the log after fio will finish the test!




Thx a lot!

I guess you have zfs set atime=off on your zfs test dataset


Good luck / Bafta!
 
Last edited:
  • Like
Reactions: Alwin Antreich

starlight

New Member
Feb 12, 2021
12
0
1
38
@aaron i am not using the DC1000B for testing. It is advertised as an boot disk for servers. I had one sample and just used it to make sure that it is not the consumer drive which is bad. The read/write test is even lower with the DC1000. For testing I am using a cheap SN750. I just want to test zfs with a nvme drive.


Here is the full benchmark:

Code:
DIRECT TO MOUNTPOINT ((( ZFS recordsize=default )))

root@pve01:~# fio --filename=/nvme01/test1 --rw=randread --ioengine=libaio --bs=4k --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test1
  read: IOPS=48.5k, BW=189MiB/s (199MB/s)(11.1GiB/60002msec)


root@pve01:~# fio --filename=/nvme01/test1 --rw=randread --ioengine=libaio --bs=1M --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test1
  read: IOPS=5169, BW=5169MiB/s (5420MB/s)(60.0GiB/11886msec)


root@pve01:~# fio --filename=/nvme01/test1 --rw=randread --ioengine=libaio --bs=4M --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test1
  read: IOPS=1164, BW=4658MiB/s (4884MB/s)(60.0GiB/13191msec)


root@pve01:~# fio --filename=/nvme01/test2 --rw=randwrite --ioengine=libaio --bs=4k --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test2
test2: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
  write: IOPS=9624, BW=37.6MiB/s (39.4MB/s)(2256MiB/60001msec); 0 zone resets


root@pve01:~# fio --filename=/nvme01/test2 --rw=randwrite --ioengine=libaio --bs=1M --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test2
  write: IOPS=1212, BW=1212MiB/s (1271MB/s)(60.0GiB/50691msec); 0 zone resets


root@pve01:~# fio --filename=/nvme01/test2 --rw=randwrite --ioengine=libaio --bs=4M --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test2
  write: IOPS=302, BW=1209MiB/s (1268MB/s)(60.0GiB/50809msec); 0 zone resets



DIRECT TO MOUNTPOINT ((( ZFS recordsize=16K )))

root@pve01:~# fio --filename=/nvme01/test1 --rw=randread --ioengine=libaio --bs=4k --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test1
  read: IOPS=248k, BW=967MiB/s (1014MB/s)(56.7GiB/60002msec)


root@pve01:~# fio --filename=/nvme01/test1 --rw=randread --ioengine=libaio --bs=1M --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test1
  read: IOPS=3829, BW=3830MiB/s (4016MB/s)(60.0GiB/16042msec)


root@pve01:~# fio --filename=/nvme01/test1 --rw=randread --ioengine=libaio --bs=4M --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test1
  read: IOPS=785, BW=3143MiB/s (3296MB/s)(60.0GiB/19549msec)


root@pve01:~# fio --filename=/nvme01/test2 --rw=randwrite --ioengine=libaio --bs=4k --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test2
  write: IOPS=49.9k, BW=195MiB/s (204MB/s)(11.4GiB/60001msec); 0 zone resets


root@pve01:~# fio --filename=/nvme01/test2 --rw=randwrite --ioengine=libaio --bs=1M --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test2
  write: IOPS=1074, BW=1074MiB/s (1126MB/s)(60.0GiB/57194msec); 0 zone resets


root@pve01:~# fio --filename=/nvme01/test2 --rw=randwrite --ioengine=libaio --bs=4M --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test2
  write: IOPS=270, BW=1084MiB/s (1136MB/s)(60.0GiB/56694msec); 0 zone resets



DIRECT TO MOUNTPOINT ((( ZFS recordsize=32K  )))

root@pve01:~# fio --filename=/nvme01/test1 --rw=randread --ioengine=libaio --bs=4k --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test1
  read: IOPS=153k, BW=598MiB/s (627MB/s)(35.0GiB/60002msec)


root@pve01:~# fio --filename=/nvme01/test1 --rw=randread --ioengine=libaio --bs=1M --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test1
  read: IOPS=4131, BW=4132MiB/s (4332MB/s)(60.0GiB/14871msec)


root@pve01:~# fio --filename=/nvme01/test1 --rw=randread --ioengine=libaio --bs=4M --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test1
  read: IOPS=837, BW=3352MiB/s (3515MB/s)(60.0GiB/18330msec)


root@pve01:~# fio --filename=/nvme01/test2 --rw=randwrite --ioengine=libaio --bs=4k --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test2
  write: IOPS=34.4k, BW=134MiB/s (141MB/s)(8057MiB/60001msec); 0 zone resets


root@pve01:~# fio --filename=/nvme01/test2 --rw=randwrite --ioengine=libaio --bs=1M --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test2
  write: IOPS=1156, BW=1157MiB/s (1213MB/s)(60.0GiB/53120msec); 0 zone resets


root@pve01:~# fio --filename=/nvme01/test2 --rw=randwrite --ioengine=libaio --bs=4M --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test2
  write: IOPS=261, BW=1044MiB/s (1095MB/s)(60.0GiB/58837msec); 0 zone resets



DIRECT TO MOUNTPOINT ((( ZFS recordsize=64K )))

root@pve01:~# fio --filename=/nvme01/test1 --rw=randread --ioengine=libaio --bs=4k --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test1
  read: IOPS=99.8k, BW=390MiB/s (409MB/s)(22.8GiB/60001msec)


root@pve01:~# fio --filename=/nvme01/test1 --rw=randread --ioengine=libaio --bs=1M --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test1
  read: IOPS=4646, BW=4646MiB/s (4872MB/s)(60.0GiB/13223msec)


root@pve01:~# fio --filename=/nvme01/test1 --rw=randread --ioengine=libaio --bs=4M --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test1
  read: IOPS=1129, BW=4519MiB/s (4738MB/s)(60.0GiB/13597msec)


root@pve01:~# fio --filename=/nvme01/test2 --rw=randwrite --ioengine=libaio --bs=4k --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test2
  write: IOPS=16.5k, BW=64.4MiB/s (67.5MB/s)(3862MiB/60002msec); 0 zone resets


root@pve01:~# fio --filename=/nvme01/test2 --rw=randwrite --ioengine=libaio --bs=1M --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test2
  write: IOPS=1002, BW=1003MiB/s (1052MB/s)(58.8GiB/60004msec); 0 zone resets


root@pve01:~# fio --filename=/nvme01/test2 --rw=randwrite --ioengine=libaio --bs=4M --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test2
  write: IOPS=248, BW=995MiB/s (1043MB/s)(58.3GiB/60018msec); 0 zone resets



DIRECT TO MOUNTPOINT ((( EXT4  )))


root@pve01:~# fio --filename=/mnt/nvme01/test1 --rw=randread --ioengine=libaio --bs=4k --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test1
  read: IOPS=286k, BW=1118MiB/s (1172MB/s)(60.0GiB/54949msec)


root@pve01:~# fio --filename=/mnt/nvme01/test1 --rw=randread --ioengine=libaio --bs=1M --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test1
  read: IOPS=7421, BW=7421MiB/s (7782MB/s)(60.0GiB/8279msec)


root@pve01:~# fio --filename=/mnt/nvme01/test1 --rw=randread --ioengine=libaio --bs=4M --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test1
  read: IOPS=1746, BW=6985MiB/s (7324MB/s)(60.0GiB/8796msec)


root@pve01:~# fio --filename=/mnt/nvme01/test2 --rw=randwrite --ioengine=libaio --bs=4k --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test2
  write: IOPS=260k, BW=1014MiB/s (1063MB/s)(59.4GiB/60001msec); 0 zone resets


root@pve01:~# fio --filename=/mnt/nvme01/test2 --rw=randwrite --ioengine=libaio --bs=1M --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test2


root@pve01:~# fio --filename=/mnt/nvme01/test2 --rw=randwrite --ioengine=libaio --bs=4M --iodepth=1 --numjobs=6 --size=10G --runtime=60 --group_reporting --name test2
  write: IOPS=408, BW=1634MiB/s (1714MB/s)(60.0GiB/37598msec); 0 zone resets
 
Last edited:

starlight

New Member
Feb 12, 2021
12
0
1
38
@guletz ext4 and zfs have write blocks after fio is finished.


Code:
ZFS


Feb 16 20:43:38 pve01 kernel: [84848.002858] z_wr_int(9835): WRITE block 33602760 on nvme0n1p1 (48 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.002911] z_wr_iss(49339): WRITE block 50340888 on nvme0n1p1 (32 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.002923] z_wr_iss(49325): WRITE block 58729496 on nvme0n1p1 (32 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.002933] z_wr_iss(49322): WRITE block 41991176 on nvme0n1p1 (240 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.002935] z_wr_iss(49332): WRITE block 318880728 on nvme0n1p1 (72 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.002944] z_wr_iss(49346): WRITE block 327165224 on nvme0n1p1 (168 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.002950] z_wr_int(9831): WRITE block 10569656 on nvme0n1p1 (72 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.003329] z_wr_iss(49331): WRITE block 16799792 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.003346] z_wr_int(9873): WRITE block 25188392 on nvme0n1p1 (16 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.003351] z_wr_int(49349): WRITE block 10569728 on nvme0n1p1 (112 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.003486] z_wr_int(9856): WRITE block 50353944 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.003489] z_wr_int(9854): WRITE block 33602808 on nvme0n1p1 (120 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.003502] z_wr_iss(49331): WRITE block 41991416 on nvme0n1p1 (120 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.003549] z_wr_int(9853): WRITE block 58742552 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.003564] z_wr_int(9863): WRITE block 318880224 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.003628] z_wr_int(9849): WRITE block 327163912 on nvme0n1p1 (16 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.003631] z_wr_int(9872): WRITE block 318880800 on nvme0n1p1 (112 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.003725] z_wr_int(9873): WRITE block 10569152 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.003761] z_wr_int(9835): WRITE block 327165400 on nvme0n1p1 (256 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.004799] z_wr_iss(49338): WRITE block 33602888 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.004811] z_wr_iss(49338): WRITE block 41991496 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.004834] z_wr_int(9863): WRITE block 327163976 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.004860] z_wr_int(9884): WRITE block 327165576 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.004878] z_wr_int(9856): WRITE block 327165656 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.004926] z_wr_int(9849): WRITE block 10569840 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.004951] z_wr_int(9894): WRITE block 16799520 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.005023] z_wr_iss(49330): WRITE block 25188128 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.005068] z_wr_iss(49315): WRITE block 33602320 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.005071] z_wr_int(9847): WRITE block 25188136 on nvme0n1p1 (32 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.005088] z_wr_int(9869): WRITE block 16799528 on nvme0n1p1 (32 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.005146] z_wr_int(9871): WRITE block 16799560 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.005174] z_wr_int(9875): WRITE block 16799776 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.005196] z_wr_int(49353): WRITE block 25188168 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.005218] z_wr_int(9882): WRITE block 25188384 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.005227] z_wr_int(9854): WRITE block 33602360 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.005259] z_wr_int(49354): WRITE block 33602424 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.005264] z_wr_int(9880): WRITE block 33602552 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.005293] z_wr_int(9855): WRITE block 33602624 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.005300] z_wr_int(9845): WRITE block 33602896 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.005327] z_wr_int(9888): WRITE block 41990928 on nvme0n1p1 (8 sectors)
Feb 16 20:43:38 pve01 kernel: [84848.005338] z_wr_int(9846): WRITE block 41990968 on nvme0n1p1 (8 sectors)




EXT4


Feb 16 20:28:59 pve01 kernel: [83968.237384] kworker/u114:2(54634): WRITE block 3785256 on nvme0n1p1 (16 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237389] kworker/u114:2(54634): WRITE block 3785280 on nvme0n1p1 (24 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237392] kworker/u114:2(54634): WRITE block 3785312 on nvme0n1p1 (8 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237394] kworker/u114:2(54634): WRITE block 3785408 on nvme0n1p1 (8 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237397] kworker/u114:2(54634): WRITE block 3785456 on nvme0n1p1 (8 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237400] kworker/u114:2(54634): WRITE block 3785832 on nvme0n1p1 (8 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237403] kworker/u114:2(54634): WRITE block 3785848 on nvme0n1p1 (16 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237405] kworker/u114:2(54634): WRITE block 3785896 on nvme0n1p1 (8 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237408] kworker/u114:2(54634): WRITE block 3786688 on nvme0n1p1 (8 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237411] kworker/u114:2(54634): WRITE block 3787032 on nvme0n1p1 (8 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237414] kworker/u114:2(54634): WRITE block 3787072 on nvme0n1p1 (16 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237434] kworker/u114:2(54634): WRITE block 3787104 on nvme0n1p1 (16 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237436] kworker/u114:2(54634): WRITE block 3787128 on nvme0n1p1 (8 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237439] kworker/u114:2(54634): WRITE block 3787328 on nvme0n1p1 (8 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237444] kworker/u114:2(54634): WRITE block 3787352 on nvme0n1p1 (32 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237447] kworker/u114:2(54634): WRITE block 3787432 on nvme0n1p1 (8 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237449] kworker/u114:2(54634): WRITE block 3787456 on nvme0n1p1 (8 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237452] kworker/u114:2(54634): WRITE block 3787488 on nvme0n1p1 (8 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237456] kworker/u114:2(54634): WRITE block 3787544 on nvme0n1p1 (32 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237462] kworker/u114:2(54634): WRITE block 3787592 on nvme0n1p1 (40 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237473] kworker/u114:2(54634): WRITE block 3787712 on nvme0n1p1 (8 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237477] kworker/u114:2(54634): WRITE block 3787824 on nvme0n1p1 (40 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237481] kworker/u114:2(54634): WRITE block 3787872 on nvme0n1p1 (24 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237486] kworker/u114:2(54634): WRITE block 3787904 on nvme0n1p1 (24 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237491] kworker/u114:2(54634): WRITE block 3787984 on nvme0n1p1 (16 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237496] kworker/u114:2(54634): WRITE block 3788024 on nvme0n1p1 (8 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237500] kworker/u114:2(54634): WRITE block 3788104 on nvme0n1p1 (24 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237511] kworker/u114:2(54634): WRITE block 3788144 on nvme0n1p1 (16 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237513] kworker/u114:2(54634): WRITE block 3788176 on nvme0n1p1 (8 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237516] kworker/u114:2(54634): WRITE block 3788208 on nvme0n1p1 (8 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237519] kworker/u114:2(54634): WRITE block 3788320 on nvme0n1p1 (24 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237525] kworker/u114:2(54634): WRITE block 3788360 on nvme0n1p1 (24 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237528] kworker/u114:2(54634): WRITE block 3788464 on nvme0n1p1 (16 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237533] kworker/u114:2(54634): WRITE block 3788488 on nvme0n1p1 (8 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237537] kworker/u114:2(54634): WRITE block 3788984 on nvme0n1p1 (8 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237540] kworker/u114:2(54634): WRITE block 3789000 on nvme0n1p1 (8 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237544] kworker/u114:2(54634): WRITE block 3789032 on nvme0n1p1 (8 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237549] kworker/u114:2(54634): WRITE block 3789048 on nvme0n1p1 (40 sectors)
Feb 16 20:28:59 pve01 kernel: [83968.237555] kworker/u114:2(54634): WRITE block 3789304 on nvme0n1p1 (16 sectors)
 

aaron

Proxmox Staff Member
Staff member
Jun 3, 2019
1,945
250
83
I am using a cheap SN750
Okay, well, the read speeds (bandwidth) with 1M or 4M block sizes do look good to me. Too good to be honest with about 5000MB/s. Looks like the ZFS cache might be involved. You can try to set the primarycache for the volume to only metadata with
Code:
zfs set primarycache=metadata POOL/DATASET

The writes with the larger block sizes also look okay for a cheaper SSD with a bit over 1000MB/s.

Again just to be clear, the 4k tests will run into the IOPS limit and the 1M/4M tests will run into the bandwidth limits. The reality will be somewhere in between, depending on the use case, overall load, caching etc etc.
 

starlight

New Member
Feb 12, 2021
12
0
1
38
First, thanks for all the help.

@aaron sure, this is cached. i dont think that one pcie gen3 x4 nvme can ever reach over 5000mb/s.

for sure the speeds are OK, but this does not clarify why ext4 is so much faster than zfs in random read/write. for now, nvme and zfs seems to be a waste of money. you could reach the same (and better) speeds with cheaper 2,5inch ssds in a raidz. maybe i miss something... ?

@guletz

yes . for longer than 5 secs ;)

root@pve01:~# nvme id-ns /dev/nvme0n1 -n 1 -H | tail -n 2
LBA Format 0 : Metadata Size: 0 bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good
LBA Format 1 : Metadata Size: 0 bytes - Data Size: 4096 bytes - Relative Performance: 0x1 Better (in use)
 
Last edited:

starlight

New Member
Feb 12, 2021
12
0
1
38
I checked speeds in TRUENAS newest version which should use openzfs 2.0 , results are the same.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!