Different IO Delay on two identical machine

[enozzac]

Hello,
last year i switched from xcp-ng to proxmox and i'm really happy with it, there is only one thing that bothers me, io delay
My situation is very simple, two identical machine in different location for the same purpose, opnsense, wireguard, pihole and some other container
Hw Mc1: Intel 5105, 4 2,5 gb ethernet port, 16 gb of ram, 256 gb patriot m2 p300 nvme (TLC NAND)
Hw Mc2: Intel 5105, 4 2,5 gb ethernet port, 16 gb of ram, 512 gb patriot m2 p300 nvme (TLC NAND)
My problem is with Mc2, i'm experience io delay on average at 3% in every situation; Mc1, despite having a smaller and slightly slower disk, has 0 io delay even with vm and container running
I've tried to turn off every vm and container on Mc2, same result, 3% io delay; i've tried to backup vm and ct, reinstall proxmox and i've 3% io delay right after the installation

Screenshot are from normal operation, with :
Mc1 : Vm Opnsense, 2 debian ct 1 ubuntu ct
Mc2 : Vm Opnsense and Vm Home Assistant, 2 debian ct 1 ubuntu ct

Some advice?

Regards

 

[Chris]

Hi,

Mc2 has a higher number of tps, which according to man iostat:

tps    Indicate the number of transfers per second that were issued to the device. A transfer is an I/O request to the de‐
       vice. Multiple logical requests can be combined into a single I/O request to the device. A transfer is of  indeter‐
       minate size.

and

%iowait
       Show  the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O re‐
       quest.

So there are more IO requests on Mc2 than on Mc1?
You could check with iotop which processes are causing io, maybe you can determine the difference from there.

Edit: Also, I would suspect the controllers on the two disk to not be the same, please correct me if I'm wrong.

 

[enozzac]

First of all thanks for your answer
I tried looking at iotop, but everything looks fine to my eyes
Tonight, i'll close every vm and ct and i'll try to look at iostat and iotop again
This io delay at 3% wouldn't be a real problem, even on mc2 everything works fine and without any slowdown, I just don't understand what's different and this bothers me

 

[enozzac]

Ok, this is an iostat with all vm and ct off, and this link is a video of iotop -o https://streamable.com/w7fu6t

Everything seems normal to me