[SOLVED] One Debian Guest VM extremely slow on Proxmox VE 9 - all other seem fine on the same Host

[silverstone]

For reference the same base Load on VM 156 APTCacherNG which is fast

 

[silverstone]

@fabian: spotted the Issue, almost by coincidence, while cleaning up /etc/fstab File.

There was a sync Mount Option set for the / Filesystem in /etc/fstab . Not sure why I did that some Time ago ...

So essentially every single write would be forced to be written immediately to Disk without Buffers/Cache. Thus IOPs drop from say 10k to around 100.

Now that I removed the sync Option and, while at it, also added noatime Mount Option for the / File System in /etc/fstab, it's back to around 10k IOPs .

Just the first Iteration of the Benchmark Script, no need for more :

Sleeping for 120 Seconds before starting Benchmark
FIO RANDOM IO Benchmark for Block Size = 512 and Queue Depth = 1
RAW Size = 1073741824 / RAW Block Size = 512 requires 2097152 Number of small Files
Writing one BIG File for Block Size = 512 and Queue Depth = 1:
==============================================================================================
write_iops: (g=0): rw=randwrite, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=libaio, iodepth=1
fio-3.39
Starting 1 process
write_iops: Laying out IO file (1 file / 1024MiB)
Jobs: 1 (f=1): [w(1)][99.6%][w=5208KiB/s][w=10.4k IOPS][eta 00m:01s]
write_iops: (groupid=0, jobs=1): err= 0: pid=1900: Tue Sep 23 21:02:10 2025
  write: IOPS=8457, BW=4229KiB/s (4330kB/s)(1018MiB/246458msec); 0 zone resets
    slat (usec): min=7, max=294567, avg=31.46, stdev=247.67
    clat (nsec): min=1438, max=39950k, avg=84832.18, stdev=122029.35
     lat (usec): min=46, max=294576, avg=116.29, stdev=271.49
    clat percentiles (usec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[   71],
     | 30.00th=[   76], 40.00th=[   81], 50.00th=[   86], 60.00th=[   90],
     | 70.00th=[   96], 80.00th=[  104], 90.00th=[  117], 95.00th=[  135],
     | 99.00th=[  225], 99.50th=[  285], 99.90th=[  529], 99.95th=[  766],
     | 99.99th=[ 4146]
   bw (  KiB/s): min= 1291, max= 5427, per=100.00%, avg=4231.13, stdev=756.63, samples=492
   iops        : min= 2582, max=10855, avg=8462.40, stdev=1513.28, samples=492
  lat (usec)   : 2=0.74%, 4=11.09%, 10=0.09%, 20=0.05%, 50=0.06%
  lat (usec)   : 100=63.31%, 250=23.91%, 500=0.63%, 750=0.06%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=4.51%, sys=15.53%, ctx=2084715, majf=0, minf=37
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2084387,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=4229KiB/s (4330kB/s), 4229KiB/s-4229KiB/s (4330kB/s-4330kB/s), io=1018MiB (1067MB), run=246458-246458msec

Disk stats (read/write):
  sda: ios=1/2098210, sectors=80/4127840, merge=0/477220, ticks=0/224602, in_queue=239198, util=76.96%
==============================================================================================

EDIT 1: added [Solved] to the Thread Title (Prefix) since the Issue is now solved .

 

[fabian]

so it was an I/O issue after all, just not a usual one glad it cleared up!

 

[silverstone]

so it was an I/O issue after all, just not a usual one glad it cleared up!

Yeah, a very stupid one you could say . Weird that I didn't notice it until now . Yet another Case of the Problem being between the Chair and the Screen. Sorry about that .

I would have guessed that since the default none Cache Options for the Virtual Disk were used (writes ARE cached, NOT on the Host though) that sync wouldn't have done anything. Obviously it did tank Performance completely.

https://pve.proxmox.com/wiki/Performance_Tweaks#Disk_Cache