Limitations of LXC containers for (large) production servers?

[Colin 't Hart]

I'm concerned about the limitations of LXC containers for production servers.

Currently we run our main production servers (web, database, etc) as KVM virtual machines but were considering deploying them in containers in future.

We have quite a few small applications running in containers and they work really well -- with the notable exception of our VPN servers which I had trouble getting working due to the difficulty in letting the container create a TUN device.

Recently we deployed a full-stack container for a customer which seems to sometimes stop responding for no apparent reason. Logging in, we see that the web server, application and database are running fine, but HTTP(S) request to it just hang.
We see warnings from Apache:

ulimit: error setting limit (Operation not permitted)

and many services output

Failed to reset devices.list: Operation not permitted

which could be a thing that systemd is trying to do.

Comparing ulimits in an LXC container to a KVM virtual machine, the soft limits for a container are actually higher, though some hard limits are lower.

We also use keepalived in almost all of our virtual machines and containers.

Could resource usage be causing the problems we are having with this one container? Are we likely to experience problems with eg web or database servers if we run them inside containers in the future?

 

[oguz]

hi,

many services output

these can be usually ignored. do you see anything else in the logs which seem out of place?

Recently we deployed a full-stack container for a customer which seems to sometimes stop responding for no apparent reason. Logging in, we see that the web server, application and database are running fine, but HTTP(S) request to it just hang.

can you provide more details about the container? pct config CTID is a good start

 

[Colin 't Hart]

Output of pct config:

arch: amd64
cores: 4
hostname: fof1
memory: 1024
mp0: local-lvm:vm-114-disk-2,mp=/home,backup=1,size=1T
net0: name=eth0,bridge=vmbr0,gw=192.168.100.1,hwaddr=42:6A:E2:C1:3C:FE,ip=192.168.100.231/24,type=veth
onboot: 1
ostype: ubuntu
protection: 1
rootfs: local-lvm:vm-114-disk-1,size=8G
swap: 1024
unprivileged: 1

Container is running postgres, apache, python app via uwsgi.
Top shows plenty of resources available:

top - 14:52:20 up 6 days,  6:52,  2 users,  load average: 1,05, 0,84, 0,76
Tasks:  53 total,   1 running,  52 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0,1 us,  0,0 sy,  0,0 ni, 99,9 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
KiB Mem :  1048576 total,   345192 free,   152660 used,   550724 buff/cache
KiB Swap:  1048576 total,   936516 free,   112060 used.   895916 avail Mem

 

[oguz]

do the HTTPS requests to this container always hang? what happens if you run curl -v https://container.ip.goes.here for example?

 

[Colin 't Hart]

No, it was the UWSGI app that wasn't responding. Apache was working fine.

But there's no inherent limitation that makes it a bad idea for us to deploy our new webservers as containers rather than VMs?

 

[oguz]

No, it was the UWSGI app that wasn't responding. Apache was working fine.

so perhaps it was an app-level issue?

But there's no inherent limitation that makes it a bad idea for us to deploy our new webservers as containers rather than VMs?

there are advantages and disadvantages to both, this will depend on what you exactly use it for and your workload in general.

one thing to take note of is that containers cannot be live-migrated, however VMs can be live-migrated between cluster nodes.

otherwise performance is usually very comparable between the two, on some hardware containers work better and on some hardware VMs work better. you can try some benchmarks and see what applies in your situation.

hope this helps