VM's very slow

rvdk92

Member
Nov 8, 2020
58
1
13
31
Hello,

We have now for 1,5 year running Proxmox on 9 fyshical machines and works fine.

I have in the past weeks upgrade all machines to version 7. and now from last friday (couple of days) we have problems with some of our virtual machines.

They are very and very slow! we running linux and windows machines.. and some virtual machines on same fyshical-servers are very slow.

I can't find anything in the monitoring in High CPU/ High Memory / IO's or something en doens't know where to search know.
on virtual machine or fyshical machine the cpu/memroy/swap enc. are fine.

Anybody tips?
 

shanreich

Proxmox Staff Member
Staff member
Sep 1, 2022
365
63
28
Which kernel version are you running? 5.18.14 and 5.15.57 include mitigations for the Retbleed vulnerability [1] [2]. This might cause significant performance drops on your machine. Work is underway to improve the performance of those mitigations, but they have yet to be released.

[1] https://en.wikipedia.org/wiki/Retbleed
[2] https://lwn.net/Articles/902316/
 

rvdk92

Member
Nov 8, 2020
58
1
13
31
This is the kernel where we have problems on both servers:

Linux 5.15.53-1-pve #1 SMP PVE 5.15.53-1
 

shanreich

Proxmox Staff Member
Staff member
Sep 1, 2022
365
63
28
Can you post the output of cat /proc/cpuinfo ? There is a section named bugs that should contain the information about which mitigations get applied. If it contains retbleed then those mitigations are active on your machine. You can try booting the kernel with the parameter
retbleed=off and check whether this is the cause of your performance regression. You should only do this for testing purposes, as turning off those mitigations can have severe implications on the security of your system.
 
Last edited:

rvdk92

Member
Nov 8, 2020
58
1
13
31
Thank you in the attachment of the output.

Yes i can't change anything for now because it is a production enveroiment.. only tonight we can making changes.
 

Attachments

  • cpuinfo.txt
    105.9 KB · Views: 10

shanreich

Proxmox Staff Member
Staff member
Sep 1, 2022
365
63
28
As you probably saw as well those mitigations are active. I think the easiest way to confirm this is turning off the mitigations temporarily and then checking the performance of the system. If turning off the mitigations doesn't improve your performance, please report back.

Those performance regressions should get addressed in future versions of the kernel, although there will always be some form of performance impact. It might take awhile for those patches to get merged though.
 

rvdk92

Member
Nov 8, 2020
58
1
13
31
Thank you i will try it this evening and let you know!

What I find special is that I can't see anywhere that the CPU has also increased in performance, can this be explained why it is nowhere to be found? in the attachment screenshot from last month. only had problems for a few days.

I have already removed 5 VMs but unfortunately still performance issues.
 

Attachments

  • server.png
    server.png
    221.8 KB · Views: 13

shanreich

Proxmox Staff Member
Staff member
Sep 1, 2022
365
63
28
I'm actually unsure how this would affect the CPU graphs, so I cannot give you a definite answer. My expectation would be a rise in load average and CPU usage, but I cannot say this for sure. When was the time of the upgrade for the server graphs you posted?
 

rvdk92

Member
Nov 8, 2020
58
1
13
31
The problems with slowly beginning friday the 16-09-2022 and when is upgrade this 2 servers with the most problems 19-09 and 20-09..

so we had the performance issues before the upgrade actually last friday.
 

rvdk92

Member
Nov 8, 2020
58
1
13
31
What I can see on my linux servers and on the host itself via Zabbix getting messages from sdb: Disk read/write request responses are to high
 

Pifouney

Member
Oct 17, 2021
148
12
18
33
Hey :)

not sure to solve your problem.

But, with pve7-xxx, my problems come of older BIOS Firmware. Can you check yours BIOS versions on servers slow ? (hosts).

You can too try to update your virtual-disk configuration by enabling iothread ( with treads ON), and never forget writeback cache.

Hope this help you :)
 

rvdk92

Member
Nov 8, 2020
58
1
13
31
Hi,

Thank you.. yes i have seen there are new bios firmware for the HP Proliant DL360Gen10.. so i will update this tonight! thank you for advise.

By some of virtual disks is IDE (migration from VMWare) so i can't select iothread.. and the best practicse is Cache: Write back? or just off?
 

Pifouney

Member
Oct 17, 2021
148
12
18
33
For my part, best performances = writeback. BUT: you need a good storage managing for use it safely :)

Hum ... Check how convert that, for really good performance boost for you :)

BIOS Update are really important theses times, because UEFI et iommu technology evolve really fastly, and need goods firmware for good perfs
 
Last edited:

rvdk92

Member
Nov 8, 2020
58
1
13
31
Oke thank you!

For tonight to do:

- Update bios firmware of the servers
- Try also disable retbleet
- Search for convert IDE to SCSI on virtuel machine.

Thanks i will let it know.
 

rvdk92

Member
Nov 8, 2020
58
1
13
31
Can you post the output of cat /proc/cpuinfo ? There is a section named bugs that should contain the information about which mitigations get applied. If it contains retbleed then those mitigations are active on your machine. You can try booting the kernel with the parameter
retbleed=off and check whether this is the cause of your performance regression. You should only do this for testing purposes, as turning off those mitigations can have severe implications on the security of your system.

Sorry stupid question, but know anybody how i can boot the kernel with paramater retbleed=off ?
 

rvdk92

Member
Nov 8, 2020
58
1
13
31
Sorry stupid question, but know anybody how i can boot the kernel with paramater retbleed=off ?

I have now edit /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet retbleed=off"

Hope is good so.
 

rvdk92

Member
Nov 8, 2020
58
1
13
31
Can you post the output of cat /proc/cpuinfo ? There is a section named bugs that should contain the information about which mitigations get applied. If it contains retbleed then those mitigations are active on your machine. You can try booting the kernel with the parameter
retbleed=off and check whether this is the cause of your performance regression. You should only do this for testing purposes, as turning off those mitigations can have severe implications on the security of your system.

I have now upgrade the bios firmware to latest version and boot it with parameter retbleed=off this works:

by lscpu i see now:
Vulnerability Retbleed: Vulnerable

I just think it's even slower now! anybody new options?
 

_gabriel

Active Member
Mar 30, 2021
207
41
28
38
i'm not sure if trim is supported over ide controller.
try to move a slow vm to a new datastore on a spare ssd
what ssd model ?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!