speed up consumer nvme with write through mode?

[sigmarb]

Even though I'm well aware that consumer SSD/NVME should not be used, it is reasonable to try to get the most out of "cheap" disks when budget is limited. I made the following observation and would like to discuss the pro/cons of this tunable.

Model Number: SAMSUNG MZVLB512HAJQ-00007
Serial Number: S3W9NX0MC01245
Price: 100€ /500GB.

# cat /sys/block/nvme0n1/queue/write_cache
write back

# fio --ioengine=libaio --filename=/dev/nvme0n1 --direct=1 --sync=1 --rw=write --numjobs=1 --iodepth=1 --runtime=60 --time_based --name=fio
fio: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=2058KiB/s][w=514 IOPS][eta 00m:00s]
fio: (groupid=0, jobs=1): err= 0: pid=20017: Tue Jul 13 07:18:58 2021
  write: IOPS=515, BW=2064KiB/s (2113kB/s)(121MiB/60002msec); 0 zone resets
    slat (nsec): min=1122, max=45875, avg=3924.15, stdev=2819.61
    clat (usec): min=1646, max=9101, avg=1933.45, stdev=216.29
     lat (usec): min=1649, max=9137, avg=1937.47, stdev=216.38
    clat percentiles (usec):
     |  1.00th=[ 1745],  5.00th=[ 1811], 10.00th=[ 1844], 20.00th=[ 1876],
     | 30.00th=[ 1893], 40.00th=[ 1909], 50.00th=[ 1926], 60.00th=[ 1942],
     | 70.00th=[ 1958], 80.00th=[ 1975], 90.00th=[ 2008], 95.00th=[ 2024],
     | 99.00th=[ 2073], 99.50th=[ 2343], 99.90th=[ 4948], 99.95th=[ 5145],
     | 99.99th=[ 5211]
   bw (  KiB/s): min= 1888, max= 2120, per=100.00%, avg=2063.80, stdev=44.10, samples=120
   iops        : min=  472, max=  530, avg=515.95, stdev=11.03, samples=120
  lat (msec)   : 2=89.77%, 4=9.76%, 10=0.47%
  cpu          : usr=0.11%, sys=0.19%, ctx=55200, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,30958,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=2064KiB/s (2113kB/s), 2064KiB/s-2064KiB/s (2113kB/s-2113kB/s), io=121MiB (127MB), run=60002-60002msec

Disk stats (read/write):
nvme0n1: ios=0/30905, merge=0/0, ticks=0/59771, in_queue=584, util=99.87%

Numbers are pretty low. However setting mode to write through, numbers are awesome:

# echo "write through" > /sys/block/nvme0n1/queue/write_cache

# fio --ioengine=libaio --filename=/dev/nvme0n1 --direct=1 --sync=1 --rw=write --numjobs=1 --iodepth=1 --runtime=60 --time_based --name=fio
fio: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=235MiB/s][w=60.1k IOPS][eta 00m:00s]
fio: (groupid=0, jobs=1): err= 0: pid=24756: Tue Jul 13 07:21:02 2021
  write: IOPS=59.0k, BW=234MiB/s (246MB/s)(13.7GiB/60001msec); 0 zone resets
    slat (nsec): min=1092, max=80190, avg=1250.79, stdev=244.84
    clat (nsec): min=240, max=7174.0k, avg=15136.54, stdev=4289.01
     lat (usec): min=13, max=7201, avg=16.42, stdev= 4.32
    clat percentiles (nsec):
     |  1.00th=[12736],  5.00th=[14400], 10.00th=[14528], 20.00th=[14656],
     | 30.00th=[14656], 40.00th=[14784], 50.00th=[14784], 60.00th=[14912],
     | 70.00th=[15040], 80.00th=[15296], 90.00th=[16512], 95.00th=[18048],
     | 99.00th=[18816], 99.50th=[19840], 99.90th=[23168], 99.95th=[24704],
     | 99.99th=[29824]
   bw (  KiB/s): min=230720, max=245592, per=99.99%, avg=239900.05, stdev=2731.92, samples=119
   iops        : min=57680, max=61398, avg=59975.03, stdev=682.97, samples=119
  lat (nsec)   : 250=0.01%, 500=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=99.54%, 50=0.46%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%
  cpu          : usr=6.78%, sys=10.47%, ctx=3598763, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,3598755,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=234MiB/s (246MB/s), 234MiB/s-234MiB/s (246MB/s-246MB/s), io=13.7GiB (14.7GB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=0/3592691, merge=0/0, ticks=0/53432, in_queue=4, util=99.87%