Poor Performance, arround 100CTs per Clusternode

[linux team]

Hi,

we were running 200 CTs (LAMP Stacks) on a Proxmox Cluster of two nodes with 24 cores & 64GB RAM each.

as we deployed around 80 or 90 CTs on each node, the load of the Cluster nodes starts to increase unproportional and the performance of the CTs was getting very poor.

with 100 CTs on each node, the CTs were responding so slow that we had to add a third cluster node.
Apache processes were waiting for MySQL responses until running into PHP-timeouts (more then 30 seconds).

before migrating our OpenVZ environment to Proxmox, we could run all of our 200 CTs on just one hardware node without noticing any slowdown.

as storage back-end we are using an external CEPH cluster. we could notice no performance issues related to the CEPH cluster.

is there a way to configure a Proxmox Cluster which scales for 100's of CTs & produces less overhead and does not slow down the CTs?

Best wishes,
Stephan.


pveversion -v

proxmox-ve: 5.1-42 (running kernel: 4.13.13-4-pve)
pve-manager: 5.1-49 (running version: 5.1-49/1e427a54)
pve-kernel-4.13: 5.1-44
pve-kernel-4.13.16-2-pve: 4.13.16-47
pve-kernel-4.13.16-1-pve: 4.13.16-46
pve-kernel-4.13.13-6-pve: 4.13.13-42
pve-kernel-4.13.13-5-pve: 4.13.13-38
pve-kernel-4.13.13-4-pve: 4.13.13-35
pve-kernel-4.13.13-3-pve: 4.13.13-34
pve-kernel-4.13.13-2-pve: 4.13.13-33
pve-kernel-4.13.13-1-pve: 4.13.13-31
pve-kernel-4.13.8-3-pve: 4.13.8-30
pve-kernel-4.13.8-2-pve: 4.13.8-28
pve-kernel-4.13.4-1-pve: 4.13.4-26
pve-kernel-4.10.17-4-pve: 4.10.17-24
pve-kernel-4.10.17-3-pve: 4.10.17-23
pve-kernel-4.10.17-2-pve: 4.10.17-20
corosync: 2.4.2-pve3
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-4
libpve-common-perl: 5.0-30
libpve-guest-common-perl: 2.0-14
libpve-http-server-perl: 2.0-8
libpve-storage-perl: 5.0-18
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.0-2
lxcfs: 3.0.0-1
novnc-pve: 0.6-4
proxmox-widget-toolkit: 1.0-14
pve-cluster: 5.0-23
pve-container: 2.0-21
pve-docs: 5.1-17
pve-firewall: 3.0-7
pve-firmware: 2.0-4
pve-ha-manager: 2.0-5
pve-i18n: 1.0-4
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.1-5
pve-xtermjs: 1.0-2
qemu-server: 5.0-24
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.7-pve1~bpo9​

 

[dietmar]

before migrating our OpenVZ environment to Proxmox, we could run all of our 200 CTs on just one hardware node without noticing any slowdown.

And you run OpenVZ on Ceph? I guess not, so you compare totally different things.

 

[linux team]

you are right, we were running the OpenVZ VEs on a FC-LUN.

right now I'm migrating the CTs from the CEPH-storage to the old FC-LUN.
but it seems the performance is getting worse...

 

[tschanness]

Whats the CPU and RAM utilisation? How many cores do you give each CT? Whats the network utilisation? Storage I/O? Connection to the CEPH cluster? Do they all talk to the same DB?
So many questions, so little answers.

 

[linux team]

the CTs are LAMP stacks, every CT has its own Apache & MySQL server, every CT can use the CPU-cores which are available (No CPU restrictions).

utilization per node (currently 70 CTs are running per node):
CPU utilization: ~1%, Load: ~0.5
Memory utilization: 45%
Netin/Netout: < 2 Mb/s average

when I increase the amount of CTs to ~100/node the load increases to over 40 & the CTs are slowing down and they becoming
no longer accessible.
e.g. a simple command line command like 'php --version' needs up to a minute to terminate.

at the CEPH side we can not see any latencies or unusaual peaks.

 

[tschanness]

What does "top" say when you have the performance issues?

 

[linux team]

just an update: rebooted into the newest kernel and the performance seems to be way better.

 

[linux team]

running over 100 cts on each node! performance stays quite good!