Proxmox 5.4 Issues with VM performance

[avoleoo]

First of all thank you so much for a great, great product! I have switched from Hyper-V to Proxmox about a year ago and have been using 5.1, 5.2, 5.3 proxmox versions. When 5.4 came out, I decided to upgrade and did clean install of the proxmox and noticed a couple of performance issues.

1. When proxmox is doing backups ( USB HDD mounted and added as directory and also NFS share ) it seriously slows down the machines and even proxmox web interface in both cases ( USB and NFS )

2. Machine performance and relation between each other has changed. I have 2 machines running ( both windows ) if CPU of the machine 1 is loaded then I will feel performance issues on the machine 2 and vice versa. In versions 5.2 and 5.3 I could not tell the difference. Performance tests for the each machine individually is about the same as in previous versions, but somehow they are more sensitive to the CPU load. its like you live in an apartment and you can hear the noise of the neighbors.

Load on the machine 2 when machine 1 is very busy shows idle, but performance feels like loaded.
CPU performance test on machine 2 while machine 1 is loaded shows no chance
HDD performance test on machine 2 while machine 1 is loaded shows no hesitation ( Read 518 Mb/s Write 459 Mb/s )

Yet is still feels sluggish ( not very sluggish but noticable difference from previous version )

Im running two machines, one is windows 7, one windows server 2016, both on Virtio.
is there anything I can try to fix this issue?

Thank you !

 

[tim]

How do the problems show up on the second node? Anything in the logs of your nodes (/var/log/syslog) ?

 

[avoleoo]

Nothing in logs. The problem shows up as lagging. Slow to open things. Not very slow but much slower than 5.2 version. Also machines very slow when doing backups from proxmox.

 

[vshaulsk]

What hardware are you running? Processors, memory, disk, etc.

I have not noticed anything on my setup, but will try to recreate by loading up other VM's while using windows VM
- My setup for test: R620 Dell with 128gb memory; dual E5-2680V2 or dual E5-2663V2 (depends on node) and either all SSD raid 5 or raid 10 (depends on node)

All of the nodes are running 5.4 .......

 

[vshaulsk]

So I ran a couple of tests on one of my proxmox node, which is running latest 5.4 (fully updated)
Dell R620 - dual E5-2643V2 (24 cores - 3.5ghz) 128gb of ram; 8x400gb Intel S3610 SSD in raid 10 (Perc H710p 1gb controller)- EXT4 file system from proxmox installer

I have 5 VM's running on this node

I stressed 3 VM's with phoronix test suite, plex trans-coding and crystal disk mark (windows VM).
On the node I was seeing loads of about 30 and CPU usage over 90% in some parts of the test on node.
- My libreNMS was sending me warnings like crazy.....hahahaha

I am writing this from the windows VM on the node; currently running crystal disk mark.
No sluggishness or performance impact visible under regular use (browsing net, excel, powerpoint, etc..)
- I would say based on this test Proxmox 5.4 performance is just like 5.2 & 5.3

Your hardware and software configuration maybe different (ZFS vs hardware raid, etc.), but I don't think it is an issue with the version of Proxmox directly. Something within the hardware of configuration.....

 

[avoleoo]

vshaulsk thank you so much for the test. I will look into my config.

 

[vshaulsk]

I was thinking about it some more last night:

I have the following modifications to the base proxmox install:
vm.swappiness = 1 (I found that the base value of proxmox caused me some sluggishness over time)

The two below .... I am not sure I have seen much difference with them implemented (maybe I should disable them)
vm.dirty_background_ratio = 1
vm.dirty_ratio = 3