2-3% IO delay floor with Alder Lake-N CPU family

[zodiaq]

This may be a common-sense troubleshooting topic for the more seasoned members of this forum, but I can't seem to figure out the cause of the problem, so I dare to ask a question.

When I use Proxmox 7/8 on Alder Lake-N series CPUs, I always have about 3% IO Delay.

I thought there was some kind of performance problem because I feel laggy response when switching "Disks" screens, but the diskspd results are not that bad.
However, when comparing the diskspd read results with the Ryzen system in the same cluster, it has an IO delay of 0%, while the N95 system has an IO delay of about 4.5%.

CPU	SSD	Site
N95	CSSD-M2B5GPG3VND	https://www.cfd.co.jp/biz/product/detail/cssd-m2b5gpg3vnd.html
Ryzen	TS500GMTE240S	https://us.transcend-info.com/Products/No-1175

I was wondering if this was just happening on my system, but I saw similar results on the 8-Aug episode of ServeTheHome, a well-known YouTuber.
I don't make sure what SSD they use.

If anyone knows the cause of this problem, I would appreciate a pointer to the information.
Also, if there is anything I should consider as a general problem, that would be helpful as well.

 

[iworx]

This may be a common-sense troubleshooting topic for the more seasoned members of this forum, but I can't seem to figure out the cause of the problem, so I dare to ask a question.

When I use Proxmox 7/8 on Alder Lake-N series CPUs, I always have about 3% IO Delay.
View attachment 55131
I thought there was some kind of performance problem because I feel laggy response when switching "Disks" screens, but the diskspd results are not that bad.
However, when comparing the diskspd read results with the Ryzen system in the same cluster, it has an IO delay of 0%, while the N95 system has an IO delay of about 4.5%.

CPU	SSD	Site
N95	CSSD-M2B5GPG3VND	https://www.cfd.co.jp/biz/product/detail/cssd-m2b5gpg3vnd.html
Ryzen	TS500GMTE240S	https://us.transcend-info.com/Products/No-1175

I was wondering if this was just happening on my system, but I saw similar results on the 8-Aug episode of ServeTheHome, a well-known YouTuber.
I don't make sure what SSD they use.
View attachment 55132
If anyone knows the cause of this problem, I would appreciate a pointer to the information.
Also, if there is anything I should consider as a general problem, that would be helpful as well.

I'm experiencing the same with a Jasper Lake Intel Celeron. Already tried updating the bios / microcode / ... still the same issue. Did you successfully get this fixed?

 

[zodiaq]

I'm experiencing the same with a Jasper Lake Intel Celeron. Already tried updating the bios / microcode / ... still the same issue. Did you successfully get this fixed?

Unfortunately, I have not been able to fix it yet.
I thought it might be a PCIe power management issue, and I have looked into BIOS settings and the kernel command line, but I have not found an effective way to help.

 

[beisser]

i had a jasper lake unit (n5105) and it always had 3% base io-delay using nvme ssd's and never changed, even after switching to different ssd models.

my current alderlake u unit (pentium 8505) has about 1% base io delay using enterprise sata ssds.

havent noticed any negative effects on either units.

 

[zodiaq]

OMG. The answer might be right in front of me long ago and I hadn't found it.
_gabriel said in the thread (https://forum.proxmox.com/threads/high-io-delay-and-server-load-on-nvme-compared-to-sd-card.136886/)

lsblk show nvme is used with both cases.
it's only the boot partition was swapped.
Perhaps, io delay is from 0B sda drive.

I checked my node and found it...

I cannot stop the node right now, so will try disable SD card reader capability later.

 

[zodiaq]

My issue seems to be resolved, finally.
In order of what I did,

1. Add sdhci_pci to /etc/modprobe.d/blacklist.conf
   Result: failed; IO delay did not go down by simply not loading the SDCard driver.
2. Turn off EMMC function in BIOS
   Result: incomplete; the PCI device was no longer there, however IO Delay went down a little, still in the 1-2% range.
3. Delete device file (/dev/sda)
   echo 1 > /sys/block/sda/device/delete
   Result: Success! IO Delay is almost zero.

I believe #1 is unnecessary.
If other people also can resolve the issue with the same steps, I'll mark this thread resolved.

 

[beisser]

just tried it on my alder lake u unit (pentium 8505). deleting the device for the sd-card reader dropped io delay from ~1% to 0.0x%

any idea how to make this permanent? or is this command permanent?

 

[zodiaq]

just tried it on my alder lake u unit (pentium 8505). deleting the device for the sd-card reader dropped io delay from ~1% to 0.0x%

any idea how to make this permanent? or is this command permanent?

I have not actually confirmed, but I believe that if you disable the device in the BIOS to prevent device scanning, then delete the device file, it will be permanent. I hope so.

 

[iworx]

I can confirm it works. It's exactly the same on my AliExpress device. Thanks! I've been looking for this solution for months.