poor performance Proxmox 6.1 on ProLiant DL385 Gen 10

[oliver.fink@ak-vorarlberg]

We have a problem on one of our Proxmox Machines
-ProLiant DL385 Gen 10
-2 AMD EPYC 7451 24CORES
-128GB RAM
We updated all of the firmware on this machine and Proxmox to the newest version (6.1)

Under Proxmox 5.4 (and old firmware on Server) the hosts ran perfectly. Now all machine run very very slow.

If I Migrate the VirtualMachine to another Proxmox Machine (also 6.1) on my cluster it runs perfectly (same NAS)

Does anybody know this Problem. Maybe a small setting in BIOS???????


Thanks for your help
Greetings
Fink Oliver

 

[aaron]

Now all machine run very very slow.

Can you explain how the slowness shows?

Slow when accessing anything on the disk?

Do you have a detailed performance monitoring set up on the PVE node?
Anything suspicious in the Summary view of the node like a high CPU usage or IO Delay?

 

[oliver.fink@ak-vorarlberg]

Hi Aarron,
It boot s very very slow (windows Server 2019) - it reacts very slow on keypress ....

No - we didn't monitor the machine.
Do you know any possibilities in proxmox to minitor the machine?

Attached is the system-report of the proxmox-node...

Boot on "faulty" proxmox: 90 seconds
Boot on "good" proxmox: 20 seconds (same NAS - just migrated the VM-HOST)

Greeting and thanks
Olli

 

[aaron]

Can you migrate the VMs disk to a storage that is local on the node to eliminate any NFS or network related problems?

 

[oliver.fink@ak-vorarlberg]

We already did that - same bad performance ...
Any other idea - maybe BIOS- or Processor-related?

Here the output of pveperf:

root@proxmox03-01:/var/log# pveperf
CPU BOGOMIPS: 439994.88
REGEX/SECOND: 2110289
HD SIZE: 54.63 GB (/dev/mapper/pve-root)
BUFFERED READS: 608.39 MB/sec
AVERAGE SEEK TIME: 0.24 ms
FSYNCS/SECOND: 807.23
DNS EXT: 24.25 ms
DNS INT: 4.57 ms (akv.lan)

Thanks a lot Aaron

 

[aaron]

Hmm, I would not know of any BIOS settings that would show this behavior.

You can try to install the latest Kernel ( see https://forum.proxmox.com/threads/linux-kernel-5-4-for-proxmox-ve.66854/ ) and test if the situation is better if you boot the node with it.

If you do not have a monitoring solution running that can give detailed performance stats, the node's summary view can give you some hints.
How is the IO Delay and CPU usage graph?

How is the Disk IO Graph for the VM itself? (summary panel of the VM)

 

[oliver.fink@ak-vorarlberg]

Tried installing kernel 5.4 - still the same - even a bit slower.

The CPU is constantly high - even if there happens nothing on the VM (Server 2019) - 4 processors - ca. 50% usage ...
When booting between 50% and 100% ...

 

[oliver.fink@ak-vorarlberg]

Hi Aaron,

Today we tried to install PROXMOX 5.4 onto this Machine...

Works like a charm - really fast ...

What could be the reason for this?

Greetings and thanks for your answer
Oliver

 

[aaron]

No idea what could be the reason.
Is it the same hardware as on the other nodes?

Firmware and BIOS are up to date?

When testing with PVE 6.x, is there anything reported in dmesg or the syslog that could hint at any problems?

Maybe someone else has an idea where to look or what to test?

You could test it with a freshly installed guest, maybe even differentiate between a Windows and a Linux based guest. See if you can reproduce the issue in order to narrow in where to investigate further.