~30% performance loss on vm compared to underlying bm.

[dkerov]

Hi,

I'm very new to Proxmox... Our company is the process of looking for VMWare substitute for k8s clusters.
We tried out Proxmox and it looks very promising and solid.
However, while doing performance testing on our proprietary application we observed very noticeable performance degradation.

For the test we are using two absolutely identical HPe DL360 G8, dual 2690 v1, 192G RAM, Storage - 1 Intel 3.8T SSDs. One is bare metal, the other one is pve-manager/6.3-2/22f57405 (running kernel: 5.4.73-1-pve).

On a bare metal 6 thread test takes 3 minutes.
On a VM with "kvm64" CPU type test takes 7 minutes.
On a VM with "host" CPU type test takes 4 minutes

1 thread test takes 7m40s on a BM and 10m40s on a VM (host cpu type).

For a clarity sake we can leave kvm64 cpu type aside, but 30% performance loss on virtualization looks abnormal for a host cpu type.
RAM should not be an issue - free shows plenty of RAM free.
vm config attached. (lvm-main there is LVM Thin pool on that intel SSD.)
Let me know if I can provide more info.

Any piece of advice or best practices for tuning?

DK

agent: 1
boot: c
bootdisk: scsi0
cores: 6
cpu: host
hotplug: disk,network,usb
ide2: lvm-main:vm-104203-cloudinit,media=cdrom,size=4M
ipconfig0: gw=10.32.104.1,ip=10.32.104.203/24
memory: 32768
name: vmtest01
net0: virtio=52:57:EA:06:C1:CC,bridge=vmbr1104,firewall=0
numa: 1
onboot: 1
ostype: l26
protection: 0
scsi0: lvm-main:vm-104203-disk-0,size=40G
scsihw: virtio-scsi-pci
searchdomain: stratus.northernlight.com
serial0: socket
smbios1: uuid=9781786c-653a-46df-8506-269e4963d438
sockets: 2
sshkeys:
vga: serial0
vmgenid: 54e7e432-f91e-4383-a368-7a5f56ec7196

 

[entilza]

Could this be due to linux kernel mitigations. I had a similar issue. Try turning mitigations off as a test and see if that improves your performance. At least you'll know why.

 

[spirit]

is the cpu the limiting factor ? or does your application is also doing disk or network io when your benchmark is running ?

 

[dkerov]

Could this be due to linux kernel mitigations. I had a similar issue. Try turning mitigations off as a test and see if that improves your performance. At least you'll know why.

Interesting idea... I thought about a little... phoronix test running on BM CentOS claims that all mitigations in place. Same test on VM says that there are open vulnerabilities.
Anyway - do you mean to turn mitigations of on the pve hypervisor or in the VM?

is the cpu the limiting factor ? or does your application is also doing disk or network io when your benchmark is running ?

network is not involved. disk is involved and might be heavily involved.
We did few synthetic benchmark tests.
sysbench did not show any significant difference between CPU performance on BM and on pve VM. But this sysbench looks very rudimentary.
fio shows ~15% performance penalty on VM compared to BM. Storage used is Thin LVM. We tried "thick"-LVM but that did not give any performance gain.

Can you recommend any trust worthy PVE specific benchmark tests I can set up to isolate performance losses?

 

[entilza]

Interesting idea... I thought about a little... phoronix test running on BM CentOS claims that all mitigations in place. Same test on VM says that there are open vulnerabilities.
Anyway - do you mean to turn mitigations of on the pve hypervisor or in the VM?

Try on both. Not sure how to remove on centos but on debian

To try: Modify /etc/default/grub
Example: GRUB_CMDLINE_LINUX_DEFAULT="quiet mitigations=off"
Then update-grub

 

[dkerov]

Try on both. Not sure how to remove on centos but on debian

To try: Modify /etc/default/grub
Example: GRUB_CMDLINE_LINUX_DEFAULT="quiet mitigations=off"
Then update-grub

First I need to find a good CPU benchmark. Just to compare pure CPU metrics.