High IO with a lot of LXC Containers

[henning]

Hi community,

First some context: I'm running a KVM VM with CentOS 6 and OpenVZ Kernel on a Proxmox cluster for several years. There are 100 OpenVZ containers inside the VM. Each container is very similar, hosting mainly Apache and Asterisk. Everything is calm. The VM has 4 CPU cores, only uses 2.5 GB RAM are used, load is always below 1 and iostat shows 16 IOPS.

Because CentOS 6 and OpenVZ are outdated I wanted to migrate the system to LXC. I installed Proxmox in a KVM VM (yes, Proxmox inside a VM) and started 100 LXC containers with Apache and Asterisk using one base image 100 linked clones. The VM has 8 CPU cores, uses 12.3 GB RAM and load is 2.5 on idle and sometimes goes up above 10 when the containers are used. Biggest issue and probably the source of the high load are IOPS: iostat shows an average of 140 IOPS.

Looking at iotop I can see the majority of IOs are caused by kmmpd and jbd2 which are ext4’s multiple mount protection and journaling. The problem seems to be that each LXC container has its own disc with ext4 file system which causes IOPS to go up.

Is there any way to reduce IOPS in my setup? Does anyone have experience how to efficiently run 100 or more LXC containers on one system?

Cheers
Henning

 

[gurubert]

Why do you want to run the LXC containers inside a Proxmox VM?

 

[henning]

Because I have ~1000 containers in total (10 of the described VMs with 100 containers) and also ~15 KVM VMs for database and other stuff on a 3 node cluster. I don't want to mix 1000 containers with 15 VMs on 3 hosts.

 

[lifeboy]

I don't see much feedback on your issue, @henning, and I'm wondering if you managed to figure this out?

 

[henning]

No, haven't found a solution for this.

Switching to LXD inside my VMs now and will keep using Proxmox on my hardware nodes.

 

[lifeboy]

No, haven't found a solution for this.

Switching to LXD inside my VMs now and will keep using Proxmox on my hardware nodes.

Just from experience I would think that you should drop one layer of proxmox (I cannot see why you need to run proxmox in a VM inside proxmox when proxmox provides nice features like pools & SDN to keep things separated) and run your you VM with the containers on a proxmox instance that is not virtualised.

Have you considered using a different filesystem? btrfs or zfs will probably deal with your conditions much better.