VM with HAProxy uses 3-4 times more CPU than running in ESXi

Jan 21, 2025
7
0
1
Hi.

We have noticed another issue with our migration from VMWare ESXi, to Proxmox.

One of our VMs has a HAProxy configured in TCP mode which passes connections to SQL server on other machine. Nothing fancy, there is almost no logic behind it, just take a TCP connection and pass it over to one of the SQL servers statically set in config.

That machine handles likes 400-500 sessions per second, which is not much, but eats up 70-80% of CPU (8 cores) when doing so. We started same machine on one of our remaining ESXi nodes (same hardware as Promox) and it uses like 15-20% of CPU... Well, that is a huge difference.

We have tried to change CPU types (default was x86-64-v2 AES) to host but it didn't change much. We have also installed irqbalance on the host (ksoftirqd in haproxy VM uses 50-60% of single CPU core which does not happen on ESXi).

Any ideas what we can do?

pveversion -v
Code:
proxmox-ve: 8.3.0 (running kernel: 6.8.12-8-pve)
pve-manager: 8.3.3 (running version: 8.3.3/f157a38b211595d6)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-8
proxmox-kernel-6.8.12-8-pve-signed: 6.8.12-8
ceph-fuse: 16.2.15+ds-0+deb12u1
corosync: 3.1.7-pve3
criu: 3.17.1-2+deb12u1
glusterfs-client: 10.3-5
ifupdown: residual config
ifupdown2: 3.2.0-1+pmx11
intel-microcode: 3.20241112.1~deb12u1
libjs-extjs: 7.0.0-5
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.5.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.2.0
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.10
libpve-cluster-perl: 8.0.10
libpve-common-perl: 8.2.9
libpve-guest-common-perl: 5.1.6
libpve-http-server-perl: 5.2.0
libpve-network-perl: 0.10.0
libpve-rs-perl: 0.9.1
libpve-storage-perl: 8.3.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.5.0-1
proxmox-backup-client: 3.3.2-1
proxmox-backup-file-restore: 3.3.2-2
proxmox-firewall: 0.6.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.3.1
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.4
pve-cluster: 8.0.10
pve-container: 5.2.3
pve-docs: 8.3.1
pve-edk2-firmware: not correctly installed
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.1.0
pve-firmware: 3.14-3
pve-ha-manager: 4.0.6
pve-i18n: 3.3.3
pve-qemu-kvm: 9.0.2-5
pve-xtermjs: 5.3.0-3
qemu-server: 8.3.7
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.7-pve1

vm config:
Code:
agent: 1                                                                                                                                                                                                                                                                    
balloon: 0                                                                                                                                                                                                                                                                  
bios: seabios                                                                                                                                                                                                                                                               
boot: order=sata0;scsi0                                                                                                                                                                                                                                                     
cores: 8                                                                                                                                                                                                                                                                    
cpu: x86-64-v2-AES                                                                                                                                                                                                                                                          
memory: 4096                                                                                                                                                                                                                                                                
meta: creation-qemu=9.0.2,ctime=1739074661                                                                                                                                                                                                                                  
name: percona-lb-test                                                                                                                                                                                                                                                       
net0: vmxnet3=BC:24:11:22:9D:60,bridge=vmbr0                                                                                                                                                                                                                                
numa: 0                                                                                                                                                                                                                                                                     
onboot: 1                                                                                                                                                                                                                                                                   
ostype: l26                                                                                                                                                                                                                                                                 
sata0: none,media=cdrom                                                                                                                                                                                                                                                     
scsi0: ssd-raid1:vm-199-disk-0,iothread=1,size=16G                                                                                                                                                                                                                          
scsihw: virtio-scsi-single                                                                                                                                                                                                                                                  
smbios1: uuid=59c707c5-83f8-4e8b-b5ee-7203acd706f4                                                                                                                                                                                                                          
sockets: 1                                                                                                                                                                                                                                                                  
tags: tmp                                                                                                                                                                                                                                                                   
vmgenid: f488b404-c3fe-4209-b074-1d1409d705cc    
[/CODE}
 
Last edited:
Hi.

We have noticed another issue with our migration from VMWare ESXi, to Proxmox.

One of our VMs has a HAProxy configured in TCP mode which passes connections to SQL server on other machine. Nothing fancy, there is almost no logic behind it, just take a TCP connection and pass it over to one of the SQL servers statically set in config.

That machine handles likes 400-500 sessions per second, which is not much, but eats up 70-80% of CPU (8 cores) when doing so. We started same machine on one of our remaining ESXi nodes (same hardware as Promox) and it uses like 15-20% of CPU... Well, that is a huge difference.

We have tried to change CPU types (default was x86-64-v2 AES) to host but it didn't change much. We have also installed irqbalance on the host (ksoftirqd in haproxy VM uses 50-60% of single CPU core which does not happen on ESXi).

Any ideas what we can do?

pveversion -v
Code:
proxmox-ve: 8.3.0 (running kernel: 6.8.12-8-pve)
pve-manager: 8.3.3 (running version: 8.3.3/f157a38b211595d6)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-8
proxmox-kernel-6.8.12-8-pve-signed: 6.8.12-8
ceph-fuse: 16.2.15+ds-0+deb12u1
corosync: 3.1.7-pve3
criu: 3.17.1-2+deb12u1
glusterfs-client: 10.3-5
ifupdown: residual config
ifupdown2: 3.2.0-1+pmx11
intel-microcode: 3.20241112.1~deb12u1
libjs-extjs: 7.0.0-5
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.5.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.2.0
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.10
libpve-cluster-perl: 8.0.10
libpve-common-perl: 8.2.9
libpve-guest-common-perl: 5.1.6
libpve-http-server-perl: 5.2.0
libpve-network-perl: 0.10.0
libpve-rs-perl: 0.9.1
libpve-storage-perl: 8.3.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.5.0-1
proxmox-backup-client: 3.3.2-1
proxmox-backup-file-restore: 3.3.2-2
proxmox-firewall: 0.6.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.3.1
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.4
pve-cluster: 8.0.10
pve-container: 5.2.3
pve-docs: 8.3.1
pve-edk2-firmware: not correctly installed
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.1.0
pve-firmware: 3.14-3
pve-ha-manager: 4.0.6
pve-i18n: 3.3.3
pve-qemu-kvm: 9.0.2-5
pve-xtermjs: 5.3.0-3
qemu-server: 8.3.7
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.7-pve1

vm config:
Code:
agent: 1                                                                                                                                                                                                                                                                  
balloon: 0                                                                                                                                                                                                                                                                
bios: seabios                                                                                                                                                                                                                                                             
boot: order=sata0;scsi0                                                                                                                                                                                                                                                   
cores: 8                                                                                                                                                                                                                                                                  
cpu: x86-64-v2-AES                                                                                                                                                                                                                                                        
memory: 4096                                                                                                                                                                                                                                                              
meta: creation-qemu=9.0.2,ctime=1739074661                                                                                                                                                                                                                                
name: percona-lb-test                                                                                                                                                                                                                                                     
net0: vmxnet3=BC:24:11:22:9D:60,bridge=vmbr0                                                                                                                                                                                                                              
numa: 0                                                                                                                                                                                                                                                                   
onboot: 1                                                                                                                                                                                                                                                                 
ostype: l26                                                                                                                                                                                                                                                               
sata0: none,media=cdrom                                                                                                                                                                                                                                                   
scsi0: ssd-raid1:vm-199-disk-0,iothread=1,size=16G                                                                                                                                                                                                                        
scsihw: virtio-scsi-single                                                                                                                                                                                                                                                
smbios1: uuid=59c707c5-83f8-4e8b-b5ee-7203acd706f4                                                                                                                                                                                                                        
sockets: 1                                                                                                                                                                                                                                                                
tags: tmp                                                                                                                                                                                                                                                                 
vmgenid: f488b404-c3fe-4209-b074-1d1409d705cc  
[/CODE}
first things first, change NIC to VirtIO, if the network traffic is heavy, enable multi queue.

have you done any kind of profiling?, processes of interest?, what's the split in CPU time look like?, any relevant OS counters you could share?
 
  • Like
Reactions: bluethunder
Changing the network card type to Virtio fixed the problem. Thanks.

Looks like migration from ESXi to Proxmox some VM settings are left in "sub-optimal" state for Proxmox itself. We need to pay more attention when importing VMs.