VM's very slow

R

rvdk92

Active Member
Nov 8, 2020
67
3
28
32
Hello,

We have now for 1,5 year running Proxmox on 9 fyshical machines and works fine.

I have in the past weeks upgrade all machines to version 7. and now from last friday (couple of days) we have problems with some of our virtual machines.

They are very and very slow! we running linux and windows machines.. and some virtual machines on same fyshical-servers are very slow.

I can't find anything in the monitoring in High CPU/ High Memory / IO's or something en doens't know where to search know.
on virtual machine or fyshical machine the cpu/memroy/swap enc. are fine.

Anybody tips?
 
Which kernel version are you running? 5.18.14 and 5.15.57 include mitigations for the Retbleed vulnerability [1] [2]. This might cause significant performance drops on your machine. Work is underway to improve the performance of those mitigations, but they have yet to be released.

[1] https://en.wikipedia.org/wiki/Retbleed
[2] https://lwn.net/Articles/902316/
 
This is the kernel where we have problems on both servers:

Linux 5.15.53-1-pve #1 SMP PVE 5.15.53-1
 
Can you post the output of cat /proc/cpuinfo ? There is a section named bugs that should contain the information about which mitigations get applied. If it contains retbleed then those mitigations are active on your machine. You can try booting the kernel with the parameter
retbleed=off and check whether this is the cause of your performance regression. You should only do this for testing purposes, as turning off those mitigations can have severe implications on the security of your system.
 
Last edited:
Thank you in the attachment of the output.

Yes i can't change anything for now because it is a production enveroiment.. only tonight we can making changes.
 

Attachments

As you probably saw as well those mitigations are active. I think the easiest way to confirm this is turning off the mitigations temporarily and then checking the performance of the system. If turning off the mitigations doesn't improve your performance, please report back.

Those performance regressions should get addressed in future versions of the kernel, although there will always be some form of performance impact. It might take awhile for those patches to get merged though.
 
Thank you i will try it this evening and let you know!

What I find special is that I can't see anywhere that the CPU has also increased in performance, can this be explained why it is nowhere to be found? in the attachment screenshot from last month. only had problems for a few days.

I have already removed 5 VMs but unfortunately still performance issues.
 

Attachments

  • server.png
    server.png
    221.8 KB · Views: 43
I'm actually unsure how this would affect the CPU graphs, so I cannot give you a definite answer. My expectation would be a rise in load average and CPU usage, but I cannot say this for sure. When was the time of the upgrade for the server graphs you posted?
 
The problems with slowly beginning friday the 16-09-2022 and when is upgrade this 2 servers with the most problems 19-09 and 20-09..

so we had the performance issues before the upgrade actually last friday.
 
What I can see on my linux servers and on the host itself via Zabbix getting messages from sdb: Disk read/write request responses are to high
 
Hey :)

not sure to solve your problem.

But, with pve7-xxx, my problems come of older BIOS Firmware. Can you check yours BIOS versions on servers slow ? (hosts).

You can too try to update your virtual-disk configuration by enabling iothread ( with treads ON), and never forget writeback cache.

Hope this help you :)
 
Hi,

Thank you.. yes i have seen there are new bios firmware for the HP Proliant DL360Gen10.. so i will update this tonight! thank you for advise.

By some of virtual disks is IDE (migration from VMWare) so i can't select iothread.. and the best practicse is Cache: Write back? or just off?
 
For my part, best performances = writeback. BUT: you need a good storage managing for use it safely :)

Hum ... Check how convert that, for really good performance boost for you :)

BIOS Update are really important theses times, because UEFI et iommu technology evolve really fastly, and need goods firmware for good perfs
 
Last edited:
Oke thank you!

For tonight to do:

- Update bios firmware of the servers
- Try also disable retbleet
- Search for convert IDE to SCSI on virtuel machine.

Thanks i will let it know.
 
shanreich said:
Can you post the output of cat /proc/cpuinfo ? There is a section named bugs that should contain the information about which mitigations get applied. If it contains retbleed then those mitigations are active on your machine. You can try booting the kernel with the parameter
retbleed=off and check whether this is the cause of your performance regression. You should only do this for testing purposes, as turning off those mitigations can have severe implications on the security of your system.
Click to expand...

Sorry stupid question, but know anybody how i can boot the kernel with paramater retbleed=off ?
 
shanreich said:
Can you post the output of cat /proc/cpuinfo ? There is a section named bugs that should contain the information about which mitigations get applied. If it contains retbleed then those mitigations are active on your machine. You can try booting the kernel with the parameter
retbleed=off and check whether this is the cause of your performance regression. You should only do this for testing purposes, as turning off those mitigations can have severe implications on the security of your system.
Click to expand...

I have now upgrade the bios firmware to latest version and boot it with parameter retbleed=off this works:

by lscpu i see now:
Vulnerability Retbleed: Vulnerable

I just think it's even slower now! anybody new options?
 
i'm not sure if trim is supported over ide controller.
try to move a slow vm to a new datastore on a spare ssd
what ssd model ?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom