[SOLVED] High IO Delay and Server load on NVMe compared to SD card

[atomek]

Hello

I've noticed very strange behaviour and I can't find out what could be a solution for this. This is plain installation of Proxmox 8.0.3, with no VMs running.

When I boot the proxmox from NVMe drive (Samsung 980Pro 2TB), I got constant usage of IO Delay ~ 3%, and roughly each hour I see a spike of load reaching 0.5 - with no daemon showing that load.




When I boot from SD card - this is even the same ROOT (it is installed on NVMe card - it is quite weird to me how it works, but apparently SD installation somehow detects NVMe instance and switch to it?). The IO Delay drops to 0.5% and I don't have load spikes anymore. Here you can see at 14:12 I've rebooted the box but started from SD card - and the load is much less. It shows even the same kernel.




What could be reason for this? I'd like to boot from NVMe card to not use SD card.

 

[sb-jw]

I suspect that the NVMe is occupying the PCIe lanes, which means that the CPU could have more work to do and already has an IO delay of around 3% for managing the NVMe. The SD card does not run via PCIe lanes. The CPU doesn't necessarily shine with much performance.

At least I can't explain it any other way because it doesn't really make any sense.

Do you still have a SATA SSD or other NVMe that you can test this with?

 

[atomek]

I don't have SSD or other NVMe to test. The really strange that even if I boot from SD card, I'm still getting / mounted on NVMe. How come i'm getting more IO waiting from SD than (theoreticaly) faster NVM?

Here is lsblk when booted from SD card (good IO)
sda 8:0 1 29.7G 0 disk
|-sda1 8:1 1 1007K 0 part
|-sda2 8:2 1 512M 0 part
`-sda3 8:3 1 29.2G 0 part
nvme0n1 259:0 0 1.8T 0 disk
|-nvme0n1p1 259:1 0 1007K 0 part
|-nvme0n1p2 259:2 0 1G 0 part /boot/efi
`-nvme0n1p3 259:3 0 1.8T 0 part
|-pve-swap 253:0 0 8G 0 lvm [SWAP]
|-pve-root 253:1 0 30G 0 lvm /
|-pve-data_tmeta 253:2 0 15.9G 0 lvm
| `-pve-data 253:4 0 1.7T 0 lvm
`-pve-data_tdata 253:3 0 1.7T 0 lvm
`-pve-data 253:4 0 1.7T 0 lvm


Here is lsblk when booted from NVMe (bad IO)
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 1 0B 0 disk
nvme0n1 259:0 0 1.8T 0 disk
|-nvme0n1p1 259:1 0 1007K 0 part
|-nvme0n1p2 259:2 0 1G 0 part /boot/efi
`-nvme0n1p3 259:3 0 1.8T 0 part
|-pve-swap 253:0 0 8G 0 lvm [SWAP]
|-pve-root 253:1 0 30G 0 lvm /
|-pve-data_tmeta 253:2 0 15.9G 0 lvm
| `-pve-data 253:4 0 1.7T 0 lvm
`-pve-data_tdata 253:3 0 1.7T 0 lvm
`-pve-data 253:4 0 1.7T 0 lvm

Howver iostat (on SD boot) shows some activity on /dev/sda (SD)
avg-cpu: %user %nice %system %iowait %steal %idle
0.50 0.00 0.60 0.30 0.00 98.60

Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
dm-0 0.00 0.00 0.00 0.00 0 0 0
dm-1 12.60 4.80 44.80 1.60 24 224 8
dm-2 0.00 0.00 0.00 0.00 0 0 0
nvme0n1 5.00 209.60 44.80 1.60 1048 224 8
sda 3.20 204.80 0.00 0.00 1024 0 0

 

[_gabriel]

lsblk show nvme is used with both cases.
it's only the boot partition was swapped.
Perhaps, io delay is from 0B sda drive.

 

[atomek]

lsblk show nvme is used with both cases.
it's only the boot partition was swapped.
Perhaps, io delay is from 0B sda drive.

I couldn't believe that at first point but that was it! First I booted from NVMe with SD card inserted - IO wait was fine. Then I disabled SD card completely in BIOS (it was on USB controller) and it worked like a charm! IO Wait at idle went from 3% to 0.1%.

Thank you very much!

 

[atomek]

Here are idle stats after some optimisations CPU usage ~0.2%, IO WAIT ~ 0.01%, memory: 611 MB

 

[erneiball17]

Here are idle stats after some optimisations CPU usage ~0.2%, IO WAIT ~ 0.01%, memory: 611 MB


View attachment 58622

Hi, I'm facing the same issues but with a 20% of IO Delay. Which was the optimisations that you applied? Cheers