VM Performance not good

[Datawolk BV]

Good afternoon,

We recently switched from an old Hyper-V cluster to a ProxMox Cluster.
However, since the migration there have been many complaints about performance.
I myself have no experience with ProxMox so I am 100% newbie.
That's why I came to ask the specialists how I can optimize the cluster or where mistakes have been made.
The cluster itself was set up by an external party that sets up clusters more often, but they let me know that the cluster looks good and recommended that I come here for advice.

There are currently 80 VMs running on the cluster.
These are mainly virtual PFsenses, domain controllers, remote desktop servers and some application servers (MSSQL).

Complaints:
There is a delay on typing in outlook, for example.
Opening PDF files is slower than before (on Hyper-V).
The server does not feel as snappy as before.
etc..

Specifications (per node):

1. 3 Node Cluster
2. 1U Supermicro AS-1115SV-WTNRT
3. 1x AMD EPYC™ Siena 8324P 32C/64T - 2.65Ghz - 3.0Ghz boost - 128MB - 180W - SP6
4. 6x Samsung 96GB DDR-5 5600Mhz Registered ECC
5. 4x Samsung DC PM9A3, 3.84TB , NVMe PCIe4.0x4, V4 TLC V, U2/2.5 inch Enterprise SSD, medium endurance
6. OS Boot: 2x Samsung PM9A1 - 512GB NVME M.2 Gen4 - workstation class - 6.9GB/s read - 5GB/s write - short form factor - low endurance

Networking (per node):

1. CoroSync 2x 10Gbps
2. Management Network / Live Migrations 2x 25Gbps
3. Ceph Storage 2x 25Gbps
4. VM Network 2x 25Gbps

Storage:

1. CEPH Storage
2. 12x OSD

If there are things missing that are important let me know!

-

I know we shouldn't do storage tests directly on the VM, but did it anyway. :-D
Attached is a print screen.
Attached also an export FIO tests on the node (recommended by ChatGPT :-D).
Test 1:
fio --name=cephtest --rw=randrw --rwmixread=70 --bs=4k --size=10G --numjobs=4 --iodepth=16 --ioengine=libaio --runtime=60 --group_reporting --filename=/dev/rbd7


Test 2:
fio --name=cephfile --rw=randrw --rwmixread=70 --bs=4k --size=10G --numjobs=4 --iodepth=16 --ioengine=libaio --runtime=60 --group_reporting --filename=/mnt/testfio/testfile

-

Where should I start looking?
Because the performance should be better since the Hyper-Vs were really old machines with ISCSI storage over 2Gbps network.

I have already researched some things I found on this forum, google & ChatGPT.
But nothing I have tried has offered a solution.
Then I think of deactivating ballooning, setting CPU to “HOST”, QEMU agent, Write Back cache, etc....

Thanks in advance!

Kind Regards,

 

[Johannes S]

Are there any Differenzen regarding Performance depending on the guest OS? So do the PFSense or Linux VMs different than the Windows VMs?

 

[Datawolk BV]

Are there any Differenzen regarding Performance depending on the guest OS? So do the PFSense or Linux VMs different than the Windows VMs?

So do you mean disk testing like CrystalDiskMark on Linux based OS?
Because with Linux servers it is not so easy to notice if the performance is not so good since one does not work on it daily like an RDS.
The uplink of pfsense is 1Gbps (WAN) and in a speed test we get the speeds back.
The GUI is also performant as it should be.

 

[Johannes S]

So do you mean disk testing like CrystalDiskMark on Linux based OS?

Exactly. On Linux guests you can use also use fio, the software isn't limited to Debian https://pve.proxmox.com/wiki/Benchmarking_Storage

If there is a signifikant difference we need to look in your Windows VM Setup.
https://pve.proxmox.com/wiki/Windows_2025_guest_best_practices has some hinter how to get most out of by using correct drivers and vm settings.

Please note that the CPU type host ( due to security-mitigations for CPU Security bugs) might hurt performance on Windows, here is a recent discussion ( including Option to try out for a better experience):

K

Thread 'The Reasons for poor performance of Windows when the CPU type is host'

Mar 4, 2025

Hey guys, I did some experiments recently and I think I finally found out why Windows performs poorly when the CPU type is host. You can check the complete experiment process and conclusion in my blog (Chinese, use google translate!)

In short, the experiment finally found that the reason was that the two flags md_clear and flush_l1d caused performance problems. They would activate the CPU vulnerability mitigation measures of Windows, which would cause a significant increase in memory read latency, thus causing Windows to freeze.

The two flags md_clear and flush_l1d are not passed...

• Kobayashi_Bairuo
• Replies: 26
• Forum: Proxmox VE: Installation and configuration

• 

On Linux and BSD host should be best for performance and not have issues.
If every kind of OS has the issues though something is odd with your cluster Setup or configuration ( although from the Hardware you should be fine).

 

[Datawolk BV]

Exactly. On Linux guests you can use also use fio, the software isn't limited to Debian https://pve.proxmox.com/wiki/Benchmarking_Storage

If there is a signifikant difference we need to look in your Windows VM Setup.
https://pve.proxmox.com/wiki/Windows_2025_guest_best_practices has some hinter how to get most out of by using correct drivers and vm settings.

Please note that the CPU type host ( due to security-mitigations for CPU Security bugs) might hurt performance on Windows, here is a recent discussion ( including Option to try out for a better experience):

K

Thread 'The Reasons for poor performance of Windows when the CPU type is host'

Mar 4, 2025

Hey guys, I did some experiments recently and I think I finally found out why Windows performs poorly when the CPU type is host. You can check the complete experiment process and conclusion in my blog (Chinese, use google translate!)

In short, the experiment finally found that the reason was that the two flags md_clear and flush_l1d caused performance problems. They would activate the CPU vulnerability mitigation measures of Windows, which would cause a significant increase in memory read latency, thus causing Windows to freeze.

The two flags md_clear and flush_l1d are not passed...

• Kobayashi_Bairuo
• Replies: 26
• Forum: Proxmox VE: Installation and configuration

• 

On Linux and BSD host should be best for performance and not have issues.
If every kind of OS has the issues though something is odd with your cluster Setup or configuration ( although from the Hardware you should be fine).

Ok, clear!
The info about the CPU I have yet to read!
Thanks for the info!

VM Config (Linux)
root@pve2:~# qm config 133
agent: 1
balloon: 0
boot: order=scsi0;ide2;net0
cores: 4
cpu: host
ide2: none,media=cdrom
machine: q35
memory: 4096
meta: creation-qemu=9.2.0,ctime=1747914291
name: 000. LS-JRD-00
net0: virtio=BC:24:11:72:84:50,bridge=VM
numa: 0
ostype: l26
scsi0: VM-pool:vm-133-disk-0,backup=0,iothread=1,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=8d9c1e7e-62cd-41be-bc1b-ff3023a684df
sockets: 1
vmgenid: 1d06def8-3b0f-4ca7-945d-924a32dcea31

I may still need to change cache to write back for this VM.
But these are the results:

root@ls-jrd-00:/opt/tacticalmesh# fio --name=test-file \
--filename=/opt/tacticalmesh/fiofile \
--rw=randrw \
--rwmixread=70 \
--bs=4k \
--size=1G \
--numjobs=2 \
--iodepth=8 \
--runtime=30 \
--ioengine=libaio \
--group_reporting
test-file: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.36
Starting 2 processes
test-file: Laying out IO file (1 file / 1024MiB)

Jobs: 2 (f=2): [m(2)][100.0%][r=24.8MiB/s,w=10.7MiB/s][r=6345,w=2740 IOPS][eta 00m:00s]
test-file: (groupid=0, jobs=2): err= 0: pid=272869: Fri Jul 18 09:02:40 2025
read: IOPS=4940, BW=19.3MiB/s (20.2MB/s)(579MiB/30001msec)
slat (usec): min=2, max=5787, avg=392.32, stdev=221.56
clat (usec): min=5, max=8557, avg=1988.01, stdev=768.12
lat (usec): min=24, max=9119, avg=2380.33, stdev=834.29
clat percentiles (usec):
| 1.00th=[ 461], 5.00th=[ 848], 10.00th=[ 1004], 20.00th=[ 1352],
| 30.00th=[ 1549], 40.00th=[ 1778], 50.00th=[ 1958], 60.00th=[ 2147],
| 70.00th=[ 2376], 80.00th=[ 2606], 90.00th=[ 2966], 95.00th=[ 3261],
| 99.00th=[ 3949], 99.50th=[ 4228], 99.90th=[ 5145], 99.95th=[ 5735],
| 99.99th=[ 7046]
bw ( KiB/s): min=14040, max=26264, per=99.64%, avg=19691.12, stdev=1659.41, samples=118
iops : min= 3510, max= 6566, avg=4922.78, stdev=414.85, samples=118
write: IOPS=2117, BW=8470KiB/s (8673kB/s)(248MiB/30001msec); 0 zone resets
slat (usec): min=2, max=1327, avg=13.60, stdev=10.10
clat (usec): min=2, max=8237, avg=1980.99, stdev=768.47
lat (usec): min=18, max=8259, avg=1994.59, stdev=769.38
clat percentiles (usec):
| 1.00th=[ 457], 5.00th=[ 848], 10.00th=[ 996], 20.00th=[ 1352],
| 30.00th=[ 1532], 40.00th=[ 1778], 50.00th=[ 1942], 60.00th=[ 2147],
| 70.00th=[ 2343], 80.00th=[ 2606], 90.00th=[ 2966], 95.00th=[ 3261],
| 99.00th=[ 3949], 99.50th=[ 4228], 99.90th=[ 5342], 99.95th=[ 5932],
| 99.99th=[ 7308]
bw ( KiB/s): min= 5344, max=11120, per=99.60%, avg=8436.22, stdev=720.18, samples=118
iops : min= 1336, max= 2780, avg=2109.05, stdev=180.05, samples=118
lat (usec) : 4=0.01%, 10=0.01%, 50=0.12%, 100=0.43%, 250=0.03%
lat (usec) : 500=1.15%, 750=2.59%, 1000=5.55%
lat (msec) : 2=42.65%, 4=46.63%, 10=0.86%
cpu : usr=2.09%, sys=10.93%, ctx=118192, majf=0, minf=30
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=148227,63525,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
READ: bw=19.3MiB/s (20.2MB/s), 19.3MiB/s-19.3MiB/s (20.2MB/s-20.2MB/s), io=579MiB (607MB), run=30001-30001msec
WRITE: bw=8470KiB/s (8673kB/s), 8470KiB/s-8470KiB/s (8673kB/s-8673kB/s), io=248MiB (260MB), run=30001-30001msec

Disk stats (read/write):
dm-0: ios=117391/9131, sectors=939128/76304, merge=0/0, ticks=52678/31107, in_queue=83785, util=97.81%, aggrios=118169/9097, aggsectors=945352/76304, aggrmerge=0/34, aggrticks=54088/29296, aggrin_queue=83398, aggrutil=80.30%
sda: ios=118169/9097, sectors=945352/76304, merge=0/34, ticks=54088/29296, in_queue=83398, util=80.30%