proxmox 8.2.7 very slow NVME

toom18 · Sep 27, 2024

Hi, I noticed when backing up the machine that it takes an awfully long time so I did a disk benchmark.
I have the system on the nvme I am doing the test.
I don't think it's normal that this drive is so slow?

Code:

root@pve:~# fio --ioengine=libaio --direct=1 --sync=1 --rw=read --bs=4K --numjobs=1 --iodepth=1 --runtime=60 --time_based --name seq_read --filename=/dev/nvme1n1
seq_read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=93.4MiB/s][r=23.9k IOPS][eta 00m:00s]
seq_read: (groupid=0, jobs=1): err= 0: pid=3681: Fri Sep 27 23:12:06 2024
  read: IOPS=26.8k, BW=105MiB/s (110MB/s)(6274MiB/60001msec)
    slat (nsec): min=4278, max=112238, avg=4786.38, stdev=878.55
    clat (nsec): min=1666, max=1434.5k, avg=31392.35, stdev=10631.59
     lat (usec): min=17, max=1439, avg=36.18, stdev=10.76
    clat percentiles (usec):
     |  1.00th=[   20],  5.00th=[   20], 10.00th=[   20], 20.00th=[   32],
     | 30.00th=[   32], 40.00th=[   32], 50.00th=[   33], 60.00th=[   33],
     | 70.00th=[   33], 80.00th=[   33], 90.00th=[   34], 95.00th=[   37],
     | 99.00th=[  101], 99.50th=[  104], 99.90th=[  120], 99.95th=[  121],
     | 99.99th=[  351]
   bw (  KiB/s): min=91280, max=157984, per=100.00%, avg=107283.03, stdev=17202.21, samples=119
   iops        : min=22820, max=39496, avg=26820.77, stdev=4300.54, samples=119
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=14.13%, 50=84.59%
  lat (usec)   : 100=0.16%, 250=1.10%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=11.96%, sys=28.07%, ctx=1606146, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1606228,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=105MiB/s (110MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s), io=6274MiB (6579MB), run=60001-60001msec

Disk stats (read/write):
  nvme1n1: ios=1603524/1136, merge=51/28873, ticks=38986/1207, in_queue=40196, util=58.62%

_gabriel · Sep 27, 2024

you've just tested iops as bs=4k, in worst case, 1 thread, 1 queue.
Check nvme model specs if this wrong or not.

UdoB · Sep 28, 2024

What kind of NVMe is it?

For comparison, but on top of ZFS:

Code:

~# smartctl  -i /dev/nvme0 | grep Model
Model Number:                       WDC WDS512G1X0C-00ENX0

Code:

~# zfs create -o primarycache=none -o secondarycache=none -o compression=off  rpool/fio
~# cd /rpool/fio/

Read, then write:

Code:

# fio --ioengine=libaio --direct=1 --sync=1 --rw=read --bs=4K --numjobs=1 --iodepth=1 --runtime=60 --time_based --name seq_read --filename fio.dat --size=4G                                                                                                                                               

  read: IOPS=42.9k, BW=167MiB/s (176MB/s)(9.81GiB/60001msec)

# fio --ioengine=libaio --direct=1 --sync=1 --rw=write --bs=4K --numjobs=1 --iodepth=1 --runtime=60 --time_based --name seq_read --filename fio.dat --size=4G

  write: IOPS=1137, BW=4550KiB/s (4659kB/s)(267MiB/60001msec); 0 zone resets

toom18 · Oct 1, 2024

Hi, sorry that after so many days.

Code:

root@pve:~# smartctl  -i /dev/nvme0 | grep Model
Model Number:                       WDC PC SN720 SDAQNTW-512G-1001

Code:

root@pve:~# zfs create -o primarycache=none -o secondarycache=none -o compression=off  rpool/fio
cannot create 'rpool/fio': no such pool 'rpool

Code:

root@pve:~# fio --ioengine=libaio --direct=1 --sync=1 --rw=read --bs=4K --numjobs=1 --iodepth=1 --runtime=60 --time_based --name seq_read --filename fio.dat --size=4G                              
seq_read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
seq_read: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=0): [f(1)][100.0%][r=95.0MiB/s][r=24.3k IOPS][eta 00m:00s]
seq_read: (groupid=0, jobs=1): err= 0: pid=823001: Tue Oct  1 12:17:53 2024
  read: IOPS=23.1k, BW=90.1MiB/s (94.5MB/s)(5407MiB/60001msec)
    slat (nsec): min=1191, max=77999, avg=6244.55, stdev=1939.68
    clat (nsec): min=1450, max=5739.4k, avg=36060.16, stdev=15490.48
     lat (usec): min=24, max=5817, avg=42.30, stdev=15.80
    clat percentiles (usec):
     |  1.00th=[   24],  5.00th=[   29], 10.00th=[   30], 20.00th=[   31],
     | 30.00th=[   32], 40.00th=[   33], 50.00th=[   33], 60.00th=[   33],
     | 70.00th=[   33], 80.00th=[   34], 90.00th=[   42], 95.00th=[   76],
     | 99.00th=[   79], 99.50th=[   83], 99.90th=[   96], 99.95th=[  133],
     | 99.99th=[  192]
   bw (  KiB/s): min=43808, max=134184, per=100.00%, avg=92323.98, stdev=20458.89, samples=119
   iops        : min=10952, max=33546, avg=23080.84, stdev=5114.70, samples=119
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=90.45%
  lat (usec)   : 100=9.44%, 250=0.08%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 10=0.01%
  cpu          : usr=8.73%, sys=27.09%, ctx=1384111, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1384136,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=90.1MiB/s (94.5MB/s), 90.1MiB/s-90.1MiB/s (94.5MB/s-94.5MB/s), io=5407MiB (5669MB), run=60001-60001msec

Disk stats (read/write):
    dm-1: ios=1378347/219, merge=0/0, ticks=42086/3, in_queue=42089, util=70.37%, aggrios=1384168/3069, aggrmerge=0/62888, aggrticks=40733/1960, aggrin_queue=42701, aggrutil=70.26%
  nvme1n1: ios=1384168/3069, merge=0/62888, ticks=40733/1960, in_queue=42701, util=70.26%

Code:

root@pve:~# fio --ioengine=libaio --direct=1 --sync=1 --rw=write --bs=4K --numjobs=1 --iodepth=1 --runtime=60 --time_based --name seq_read --filename fio.dat --size=4G
seq_read: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=6592KiB/s][w=1648 IOPS][eta 00m:00s]
seq_read: (groupid=0, jobs=1): err= 0: pid=823504: Tue Oct  1 12:20:56 2024
  write: IOPS=2227, BW=8911KiB/s (9125kB/s)(522MiB/60001msec); 0 zone resets
    slat (nsec): min=1463, max=433609, avg=8525.93, stdev=10985.66
    clat (usec): min=3, max=14731, avg=438.95, stdev=197.59
     lat (usec): min=280, max=14739, avg=447.48, stdev=201.81
    clat percentiles (usec):
     |  1.00th=[  281],  5.00th=[  281], 10.00th=[  285], 20.00th=[  285],
     | 30.00th=[  289], 40.00th=[  306], 50.00th=[  338], 60.00th=[  553],
     | 70.00th=[  562], 80.00th=[  586], 90.00th=[  635], 95.00th=[  685],
     | 99.00th=[  758], 99.50th=[  816], 99.90th=[ 1123], 99.95th=[ 3490],
     | 99.99th=[ 5800]
   bw (  KiB/s): min= 6344, max= 9936, per=100.00%, avg=8933.58, stdev=975.52, samples=119
   iops        : min= 1586, max= 2484, avg=2233.39, stdev=243.88, samples=119
  lat (usec)   : 4=0.01%, 10=0.01%, 500=55.29%, 750=43.61%, 1000=0.97%
  lat (msec)   : 2=0.06%, 4=0.02%, 10=0.04%, 20=0.01%
  cpu          : usr=0.91%, sys=2.21%, ctx=133672, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,133671,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=8911KiB/s (9125kB/s), 8911KiB/s-8911KiB/s (9125kB/s-9125kB/s), io=522MiB (548MB), run=60001-60001msec

Disk stats (read/write):
    dm-1: ios=0/445694, merge=0/0, ticks=0/54604, in_queue=54604, util=91.05%, aggrios=33/388779, aggrmerge=0/113605, aggrticks=7/57517, aggrin_queue=94782, aggrutil=90.63%
  nvme1n1: ios=33/388779, merge=0/113605, ticks=7/57517, in_queue=94782, util=90.63%

toom18 · Oct 2, 2024

The drive was not new, maybe it's time to buy one? What drives do you recommend under Proxmox?

Kingneutron · Oct 2, 2024

You can try a "factory reset" on the ssd, but I've had pretty good results so far with a Lexar NM790 1TB. So far it's at 1% wear running nearly 24/7 since Feb.

https://wiki.archlinux.org/title/Solid_state_drive/Memory_cell_clearing

Other than that you want to look at used/new Enterprise-level flash on ebay or the like and get something with a high TBW rating.

Search

Search

proxmox 8.2.7 very slow NVME

toom18

New Member

_gabriel

Renowned Member

UdoB

Distinguished Member

toom18

New Member

toom18

New Member

Kingneutron

Renowned Member

We value your privacy