proxmox 8.2.7 very slow NVME

[toom18]

Hi, I noticed when backing up the machine that it takes an awfully long time so I did a disk benchmark.
I have the system on the nvme I am doing the test.
I don't think it's normal that this drive is so slow?

root@pve:~# fio --ioengine=libaio --direct=1 --sync=1 --rw=read --bs=4K --numjobs=1 --iodepth=1 --runtime=60 --time_based --name seq_read --filename=/dev/nvme1n1
seq_read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=93.4MiB/s][r=23.9k IOPS][eta 00m:00s]
seq_read: (groupid=0, jobs=1): err= 0: pid=3681: Fri Sep 27 23:12:06 2024
  read: IOPS=26.8k, BW=105MiB/s (110MB/s)(6274MiB/60001msec)
    slat (nsec): min=4278, max=112238, avg=4786.38, stdev=878.55
    clat (nsec): min=1666, max=1434.5k, avg=31392.35, stdev=10631.59
     lat (usec): min=17, max=1439, avg=36.18, stdev=10.76
    clat percentiles (usec):
     |  1.00th=[   20],  5.00th=[   20], 10.00th=[   20], 20.00th=[   32],
     | 30.00th=[   32], 40.00th=[   32], 50.00th=[   33], 60.00th=[   33],
     | 70.00th=[   33], 80.00th=[   33], 90.00th=[   34], 95.00th=[   37],
     | 99.00th=[  101], 99.50th=[  104], 99.90th=[  120], 99.95th=[  121],
     | 99.99th=[  351]
   bw (  KiB/s): min=91280, max=157984, per=100.00%, avg=107283.03, stdev=17202.21, samples=119
   iops        : min=22820, max=39496, avg=26820.77, stdev=4300.54, samples=119
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=14.13%, 50=84.59%
  lat (usec)   : 100=0.16%, 250=1.10%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=11.96%, sys=28.07%, ctx=1606146, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1606228,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=105MiB/s (110MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s), io=6274MiB (6579MB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=1603524/1136, merge=51/28873, ticks=38986/1207, in_queue=40196, util=58.62%

 

[_gabriel]

you've just tested iops as bs=4k, in worst case, 1 thread, 1 queue.
Check nvme model specs if this wrong or not.