Slow disk performance

[rodgonemox]

Hey everyone, just installed proxmox and I'm experiencing a pretty slow performance on my disk, to the point of taking 1:30 to install Ubuntu on a VM. Not sure if that's expected with the disk I have (Crucial P2 CT500P2SSD8 SSD).

root@pve:~# pveperf
CPU BOGOMIPS:      8908.80
REGEX/SECOND:      2561482
HD SIZE:           93.93 GB (/dev/mapper/pve-root)
BUFFERED READS:    122.53 MB/sec
AVERAGE SEEK TIME: 0.09 ms
FSYNCS/SECOND:     43.26
DNS EXT:           65.81 ms
DNS INT:           0.87 ms (lan)

A simple fio benchmark:

root@pve:~# fio --ioengine=libaio --direct=1 --sync=1 --rw=read --bs=4K --numjobs=1 --iodepth=1 --runtime=60 --time_based --name seq_read --filename=/dev/nvme0n1p3
seq_read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Jobs: 1 (f=0): [f(1)][100.0%][r=4KiB/s][r=1 IOPS][eta 00m:00s]      
seq_read: (groupid=0, jobs=1): err= 0: pid=37683: Sun Jul 17 23:53:20 2022
  read: IOPS=2866, BW=11.2MiB/s (11.7MB/s)(760MiB/67886msec)
    slat (usec): min=2, max=261, avg=21.78, stdev=32.09
    clat (nsec): min=445, max=12588M, avg=320445.83, stdev=31074362.26
     lat (usec): min=16, max=12588k, avg=342.56, stdev=31074.24
    clat percentiles (nsec):
     |  1.00th=[      540],  5.00th=[    15424], 10.00th=[    15808],
     | 20.00th=[    16064], 30.00th=[    16320], 40.00th=[    16512],
     | 50.00th=[    17024], 60.00th=[    17536], 70.00th=[    55552],
     | 80.00th=[    93696], 90.00th=[   109056], 95.00th=[   536576],
     | 99.00th=[   602112], 99.50th=[   675840], 99.90th=[  5341184],
     | 99.95th=[  8716288], 99.99th=[742391808]
   bw (  KiB/s): min=    8, max=68248, per=100.00%, avg=17894.25, stdev=20356.58, samples=87
   iops        : min=    2, max=17062, avg=4473.56, stdev=5089.18, samples=87
  lat (nsec)   : 500=0.12%, 750=1.55%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.20%, 10=0.01%, 20=64.63%, 50=2.71%
  lat (usec)   : 100=16.26%, 250=5.67%, 500=1.39%, 750=7.04%, 1000=0.04%
  lat (msec)   : 2=0.03%, 4=0.11%, 10=0.20%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2000=0.01%, >=2000=0.01%
  cpu          : usr=3.82%, sys=8.97%, ctx=188422, majf=0, minf=18
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=194601,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=11.2MiB/s (11.7MB/s), 11.2MiB/s-11.2MiB/s (11.7MB/s-11.7MB/s), io=760MiB (797MB), run=67886-67886msec

Disk stats (read/write):
  nvme0n1: ios=194620/3313, merge=0/659, ticks=49078/1527957, in_queue=1613395, util=99.47%

Trying to max out the disk, using a command I found on Google Cloud Platform:

root@pve:~# fio --time_based --name=benchmark --size=100G --runtime=30 --filename=/dev/nvme0n1p3 --ioengine=libaio --randrepeat=0 --iodepth=128 --direct=1 --invalidate=
1 --verify=0 --verify_fatal=0 --numjobs=4 --rw=randread --blocksize=4k --group_repor
ting
benchmark: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
...
fio-3.25
Starting 4 processes
Jobs: 4 (f=4): [r(4)][100.0%][r=277MiB/s][r=70.0k IOPS][eta 00m:00s]
benchmark: (groupid=0, jobs=4): err= 0: pid=2166: Mon Jul 18 00:05:21 2022
  read: IOPS=71.3k, BW=278MiB/s (292MB/s)(8358MiB/30028msec)
    slat (nsec): min=1993, max=730558, avg=3920.53, stdev=6542.33
    clat (usec): min=63, max=75268, avg=7179.77, stdev=7142.57
     lat (usec): min=66, max=75271, avg=7183.78, stdev=7142.61
    clat percentiles (usec):
     |  1.00th=[ 2212],  5.00th=[ 3163], 10.00th=[ 3490], 20.00th=[ 4113],
     | 30.00th=[ 4490], 40.00th=[ 4883], 50.00th=[ 5145], 60.00th=[ 5473],
     | 70.00th=[ 5866], 80.00th=[ 6587], 90.00th=[12911], 95.00th=[22938],
     | 99.00th=[41681], 99.50th=[49021], 99.90th=[60031], 99.95th=[62653],
     | 99.99th=[66323]
   bw (  KiB/s): min=128280, max=314040, per=100.00%, avg=285210.40, stdev=7946.70, samples=240
   iops        : min=32070, max=78510, avg=71302.67, stdev=1986.68, samples=240
  lat (usec)   : 100=0.01%, 250=0.03%, 500=0.05%, 750=0.09%, 1000=0.09%
  lat (msec)   : 2=0.55%, 4=17.05%, 10=69.73%, 20=6.34%, 50=5.64%
  lat (msec)   : 100=0.43%
  cpu          : usr=4.21%, sys=13.55%, ctx=1885478, majf=0, minf=570
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=2139588,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
   READ: bw=278MiB/s (292MB/s), 278MiB/s-278MiB/s (292MB/s-292MB/s), io=8358MiB (8764MB), run=30028-30028msec

Disk stats (read/write):
  nvme0n1: ios=2130347/333, merge=0/204, ticks=15279714/6156, in_queue=15288276, util=99.73%

It seems that I have a pretty low value for the FSYNCS/SECOND field, as it should be > 200. Is the performance expected for this particular disk? If so, what disk would you guys recommend?

 

[t.lamprecht]

Not sure if that's expected with the disk I have (Crucial P2 CT500P2SSD8 SSD).

I mean that is a consumer disk and also there rather in the middle to end of that field, I wouldn't expect wonders there.

w.r.t benchmarks, you only ever test 4k blocks, those may not bring out the best performance... possibly check 64k, 256k or even 1m blocks too. The first test is definitively a bit limited by using just a small block size and that without any parallelism, it's more testing latency than throuput or IOPS.

For thorughput I'd use something like the following (destructive!) commands:

writes:

fio --rw=write --name=IOPS-write --bs=1024k --direct=1 --filename=/dev/TESTDEVICE --numjobs=4 --ioengine=libaio --iodepth=32 --refill_buffers --group_reporting --runtime=60 --time_based

reads:

fio --rw=read --name=IOPS-read --bs=1024k --direct=1 --filename=/dev/TESTDEVICE --numjobs=4 --ioengine=libaio --iodepth=32 --refill_buffers --group_reporting --runtime=60 --time_based

IOW., using 1M blocks and a reasonable parallelism, as to high isn't ideal either, as then we're starting to test the Linux kernel IO scheduler more than the NVMe itself.

It seems that I have a pretty low value for the FSYNCS/SECOND field, as it should be > 200.

That was testing the root device, not the NVMe (or partition?).

What's your actual disk layout anyway?
lsblk --ascii -o +PHY-SEC,FSTYPE,MODEL,TRAN

 

[Dunuin]

And CT500P2SSD8 often uses QLC NAND which is terribly slow + undurable and shouldn't be bought in general and especially not when running server workloads. Atleast get a decent consumer TLC SSD or even better an enterprise SSD. The powerloss protection of enterprise SSDs will help alot with sync wirte performance, life expectation with sync writes and will ensure that you don't loose your data on an power outage.

 

[rodgonemox]

Thank you, guys!

I should have mentioned I'm running proxmox on a pretty low-end machine, as the purpose is learning and self-hosting some light services. It's an MSI Cubi, with an Intel N6000 and 16GB of RAM, but I didn't experience such a dramatically lower performance when running Linux directly. I was not expecting virtualization to take such a massive hit on disk-related operations.

The performance seems to be better now too, for some reason. This happened yesterday as well, after rebooting pve, disk performance improved for a while. I have re-run the first test to compare performance when the system is super slow vs when it's just somewhat sluggish.
It seems to be two times faster? Edit: I installed a new Debian VM in about 15min to test the performance too (it was taking over 1h at its lowest performance).

root@pve:~# seq_read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1root@pve:~# fio --ioengine=libaio --direct=1 --sync=1 --rw=read --bs=4K --numjobs=1 --iodepth=1 --runtime=60 --time_based --name seq_read --filename=/dev/nvme0n1p3
seq_read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.25
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=46.5MiB/s][r=11.9k IOPS][eta 00m:00s]
seq_read: (groupid=0, jobs=1): err= 0: pid=102526: Mon Jul 18 10:55:21 2022
  read: IOPS=6294, BW=24.6MiB/s (25.8MB/s)(1475MiB/60001msec)
    slat (usec): min=2, max=277, avg=30.08, stdev=35.72
    clat (nsec): min=445, max=9364.7k, avg=119587.16, stdev=226045.23
     lat (usec): min=16, max=9389, avg=150.16, stdev=219.86
    clat percentiles (nsec):
     |  1.00th=[    506],  5.00th=[  14656], 10.00th=[  15424],
     | 20.00th=[  15808], 30.00th=[  16192], 40.00th=[  16512],
     | 50.00th=[  16768], 60.00th=[  17536], 70.00th=[  66048],
     | 80.00th=[ 100864], 90.00th=[ 536576], 95.00th=[ 626688],
     | 99.00th=[ 675840], 99.50th=[ 700416], 99.90th=[1236992],
     | 99.95th=[3391488], 99.99th=[5079040]
   bw (  KiB/s): min= 5072, max=86536, per=99.12%, avg=24957.71, stdev=21605.92, samples=119
   iops        : min= 1268, max=21634, avg=6239.43, stdev=5401.48, samples=119
  lat (nsec)   : 500=0.73%, 750=1.22%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.42%, 10=0.01%, 20=59.26%, 50=1.94%
  lat (usec)   : 100=15.76%, 250=5.11%, 500=2.73%, 750=12.61%, 1000=0.11%
  lat (msec)   : 2=0.03%, 4=0.03%, 10=0.04%
  cpu          : usr=11.29%, sys=24.36%, ctx=362707, majf=0, minf=16
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=377699,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=24.6MiB/s (25.8MB/s), 24.6MiB/s-24.6MiB/s (25.8MB/s-25.8MB/s), io=1475MiB (1547MB), run=60001-60001msec

Disk stats (read/write):
  nvme0n1: ios=375485/1310, merge=0/440, ticks=42640/5581, in_queue=48994, util=100.00%

The read test:

root@pve:~# fio --rw=read --name=IOPS-read --bs=1024k --direct=1 --filename=/dev/nvme0n1p3 --numjobs=4 --ioengine=libaio --iodepth=32 --refill_buffers --group_reporting --
runtime=60 --time_based
IOPS-read: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
...
fio-3.25
Starting 4 processes
Jobs: 3 (f=3): [_(1),R(3)][50.8%][r=52.1MiB/s][r=52 IOPS][eta 00m:59s]
IOPS-read: (groupid=0, jobs=4): err= 0: pid=103135: Mon Jul 18 10:59:17 2022
  read: IOPS=222, BW=222MiB/s (233MB/s)(13.3GiB/61272msec)
    slat (usec): min=29, max=620, avg=91.07, stdev=49.09
    clat (msec): min=9, max=3854, avg=573.57, stdev=578.62
     lat (msec): min=9, max=3854, avg=573.67, stdev=578.62
    clat percentiles (msec):
     |  1.00th=[   19],  5.00th=[   55], 10.00th=[  100], 20.00th=[  186],
     | 30.00th=[  224], 40.00th=[  284], 50.00th=[  359], 60.00th=[  456],
     | 70.00th=[  609], 80.00th=[  894], 90.00th=[ 1469], 95.00th=[ 1821],
     | 99.00th=[ 2433], 99.50th=[ 2970], 99.90th=[ 3809], 99.95th=[ 3842],
     | 99.99th=[ 3842]
   bw (  KiB/s): min=14336, max=903168, per=100.00%, avg=231884.08, stdev=49951.67, samples=477
   iops        : min=   14, max=  882, avg=226.45, stdev=48.78, samples=477
  lat (msec)   : 10=0.02%, 20=1.12%, 50=3.14%, 100=5.93%, 250=24.56%
  lat (msec)   : 500=28.73%, 750=12.48%, 1000=6.70%, 2000=13.72%, >=2000=3.59%
  cpu          : usr=0.05%, sys=0.70%, ctx=13612, majf=0, minf=32833
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.5%, 32=99.1%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=13622,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=222MiB/s (233MB/s), 222MiB/s-222MiB/s (233MB/s-233MB/s), io=13.3GiB (14.3GB), run=61272-61272msec

Disk stats (read/write):
  nvme0n1: ios=54439/338, merge=0/167, ticks=30669469/145553, in_queue=30826865, util=99.95%

Didn't run the write test as I was not sure what you mean by destructive? How can I make sure it won't write over important files? Reading fio's doc I have found: allow_mounted_write=bool, which is False by default, so it shouldn't mess up anything unless I turn that on?

This is the disk layout:

root@pve:~# lsblk --ascii -o +PHY-SEC,FSTYPE,MODEL,TRAN
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT PHY-SEC FSTYPE      MODEL       TRAN
nvme0n1                      259:0    0 465.8G  0 disk                512             CT500P2SSD8 nvme
|-nvme0n1p1                  259:1    0  1007K  0 part                512                         nvme
|-nvme0n1p2                  259:2    0   512M  0 part /boot/efi      512 vfat                    nvme
`-nvme0n1p3                  259:3    0 465.3G  0 part                512 LVM2_member             nvme
  |-pve-swap                 253:0    0     8G  0 lvm  [SWAP]         512 swap                 
  |-pve-root                 253:1    0    96G  0 lvm  /              512 ext4                 
  |-pve-data_tmeta           253:2    0   3.5G  0 lvm                 512                       
  | `-pve-data-tpool         253:4    0 338.4G  0 lvm                 512                       
  |   |-pve-data             253:5    0 338.4G  1 lvm                 512                       
  |   |-pve-vm--101--disk--0 253:6    0    32G  0 lvm                 512                       
  |   `-pve-vm--100--disk--0 253:7    0   128G  0 lvm                 512                       
  `-pve-data_tdata           253:3    0 338.4G  0 lvm                 512                       
    `-pve-data-tpool         253:4    0 338.4G  0 lvm                 512                       
      |-pve-data             253:5    0 338.4G  1 lvm                 512                       
      |-pve-vm--101--disk--0 253:6    0    32G  0 lvm                 512                       
      `-pve-vm--100--disk--0 253:7    0   128G  0 lvm                 512

Thanks the suggestions @Dunuin . As as I said, this is pretty non-serious stuff, just me playing around with cheap hardware to learn. I will try to get a decent disk.

 

[Dunuin]

Problem with low grade NAND flash like QLC is that it is basically as slow as a HDD. It only looks fast first because the SSDs got a SLC cache and sometimes also a RAM cache. So first its very fast as it only writes to RAM until RAM gets full, then it gets slower but is still usable until the SLC cache also gets full (and the more you fill up your SSD the smaller your SLC cache will get). As soon as all caches are full it will write directly to the QLC NAND and thats the point where you are down to HDD performance. So its not that unusual that you see better performance after boot with degrading performance over time.
Also keep an eye on the SSD wear. PVE can kill cheap consumer SSDs within months as each write will damage the SSD and you got the lowest end of SSDs concering write durability.

 

[rodgonemox]

Problem with low grade NAND flash like QLC is that it is basically as slow as a HDD. It only looks fast first because the SSDs got a SLC cache and sometimes also a RAM cache. So first its very fast as it only writes to RAM until RAM gets full, then it gets slower but is still usable until the SLC cache also gets full (and the more you fill up your SSD the smaller your SLC cache will get). As soon as all caches are full it will write directly to the QLC NAND and thats the point where you are down to HDD performance. So its not that unusual that you see better performance after boot with degrading performance over time.
Also keep an eye on the SSD wear. PVE can kill cheap consumer SSDs within months as each write will damage the SSD and you got the lowest end of SSDs concering write durability.

Thank you, that makes sense. I'm reading about it, found this cool guide but it feels like a rabbit-hole. Would a Samsung 970 EVO plus be a good fit? Any recommendations for a 512GB under 150€?

 

[Dunuin]

Would a Samsung 970 EVO plus be a good fit? Any recommendations for a 512GB under 150€?

Would be better as its TLC NAND with RAM cache. But its still a consumer SSD, so async writes will be fine but sync writes bad too.
For 150€ you also get a low end enterprise M.2 SSD like a "Micron 7400 PRO 480GB" which I would prefer. With that you would get fine async and sync write performance and it will last longer (rated for 800TB TBW random writes or 3800 TB TBW sequential writes...Evo Plus only got 300TB TBW).

 

[rodgonemox]

Would be better as its TLC NAND with RAM cache. But its still a consumer SSD, so async writes will be fine but sync writes bad too.
For 150€ you also get a low end enterprise M.2 SSD like a "Micron 7400 PRO 480GB" which I would prefer. With that you would get fine async and sync write performance and it will last longer (rated for 800TB TBW random writes or 3800 TB TBW sequential writes...Evo Plus only got 300TB TBW).

Thanks for the suggestion, but I cannot find it in stock anywhere in my region (Portugal, EU). I'll look into other options. Can I ask why is there such a performance difference between running Linux directly on the disk and running it in a virtualized instance? I never noticed any disk bottleneck when I had a bunch of docker containers running on a Ubuntu server barebone, nor did it take 1h to install. What's happening underneath?

 

[t.lamprecht]

Can I ask why is there such a performance difference between running Linux directly on the disk and running it in a virtualized instance?

If you run the fio tests from inside a VM ensure that the VM's disk is using SCSI and the SCSI-controller is set to VirtIO SCSI.

Additionally, you may want to try enabling IO-Threads and then switch the controller to VirtIO-SCSI Single to make that actually bring any benefit.

A 970 EVO plus should be quite an improvement compared to the current Crucial P2, FWICT the MSI Cubi with Intel N6000 CPU only got a PCIe 3.x interface for the m.2 NVMe slot, so no point in investing into a PCIe 4.x NVMe SSD (albeit if you get a cheap deal on such one, it won't hurt either).

 

[rodgonemox]

If you run the fio tests from inside a VM ensure that the VM's disk is using SCSI and the SCSI-controller is set to VirtIO SCSI.

Additionally, you may want to try enabling IO-Threads and then switch the controller to VirtIO-SCSI Single to make that actually bring any benefit.

A 970 EVO plus should be quite an improvement compared to the current Crucial P2, FWICT the MSI Cubi with Intel N6000 CPU only got a PCIe 3.x interface for the m.2 NVMe slot, so no point in investing into a PCIe 4.x NVMe SSD (albeit if you get a cheap deal on such one, it won't hurt either).

All those tests were performed in the pve host itself, not in a VM, so it should be below the virtualization layer, right? I'm not sure If understand how there can be such a difference between proxmox and vanilla barebone Linux.

Would be better as its TLC NAND with RAM cache. But its still a consumer SSD, so async writes will be fine but sync writes bad too.
For 150€ you also get a low end enterprise M.2 SSD like a "Micron 7400 PRO 480GB" which I would prefer. With that you would get fine async and sync write performance and it will last longer (rated for 800TB TBW random writes or 3800 TB TBW sequential writes...Evo Plus only got 300TB TBW).

To be honest, I wouldn't mind if it wasn't the fastest disk, as long as it doesn't take 1h to install Debian... I don't mind somewhat sluggish, but it's just unusable when it gets really slow, which didn't happen with barebone Linux.

 

[t.lamprecht]

All those tests were performed in the pve host itself, not in a VM, so it should be below the virtualization layer, right? I'm not sure If understand how there can be such a difference between proxmox and vanilla barebone Linux.

I was just confused by

difference between running Linux directly on the disk and running it in a virtualized instance

As that implied to me that you indeed run the tests inside a VM. Proxmox VE itself is on the host just like any other bare metal Linux after all.

as long as it doesn't take 1h to install Debian

Again a bit confused, do the 1h come from installing Debian inside a VM hosted on Proxmox VE or what do you mean here?

 

[rodgonemox]

I was just confused by

As that implied to me that you indeed run the tests inside a VM. Proxmox VE itself is on the host just like any other bare metal Linux after all.


Again a bit confused, do the 1h come from installing Debian inside a VM hosted on Proxmox VE or what do you mean here?

Sorry for the confusion. Those tests were ran in pve. The 1h installation was when attempting to create a new VM in proxmox, not installing proxmox itself.