[SOLVED] Can't scale LXC CTs past 670, they fail to start.

[andwoo8182]

Hello everyone, my first post here, so please go slow

I have a single Proxmox host/node that is running approximately 670 containers just fine, but it struggles to start any more - and when at this max number of containers running, i slowly notice the odd container losing network connectivity, which is corrected by a reboot.

'pveversion -v'
proxmox-ve: 6.1-2 (running kernel: 5.3.13-1-pve)
pve-manager: 6.1-5 (running version: 6.1-5/9bf06119)
pve-kernel-5.3: 6.1-1
pve-kernel-helper: 6.1-1
pve-kernel-5.0: 6.0-11
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-5.3.10-1-pve: 5.3.10-1
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.21-3-pve: 5.0.21-7
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 1.2.8-1+pve4
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-9
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-3
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-1
pve-cluster: 6.1-2
pve-container: 3.0-15
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-9
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 3.13.2-1
qemu-server: 6.1-4
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2

container config:

arch: amd64
cores: 22
cpulimit: 4
hostname: 065.horizen.li
memory: 1280
net0: name=eth1,bridge=vmbr1,firewall=1,hwaddr=AE:F3:14:46:9D:39,ip=dhcp,ip6=auto,type=veth
ostype: ubuntu
rootfs: xdata16:subvol-810-disk-0,size=35G
swap: 3072
unprivileged: 1

i have run this debug command to get the logs for the container:

lxc-start --logfile /tmp/lxc-start810.log -n 810 --logpriority DEBUG

logs are attached.

does anyone have any advice as to which direction to look? I can't tell if its something related to cgroups/apparmor/seccomp.

any advice is appreciated, thanks!

 

[oguz]

hi,

from your log you can see the following:

lxc-start 810 20200111140321.951 ERROR    cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1287 - File exists - Failed to create directory "/sys/fs/cgroup/unified//lxc/810"
lxc-start 810 20200111140321.954 ERROR    cgfsng - cgroups/cgfsng.c:container_create_path_for_hierarchy:1336 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc/810"
lxc-start 810 20200111140321.955 ERROR    cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1496 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc/810"
lxc-start 810 20200111140321.957 ERROR    cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1287 - File exists - Failed to create directory "/sys/fs/cgroup/unified//lxc/810-1"
lxc-start 810 20200111140321.957 ERROR    cgfsng - cgroups/cgfsng.c:container_create_path_for_hierarchy:1336 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc/810-1"
lxc-start 810 20200111140321.958 ERROR    cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1496 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc/810-1"
lxc-start 810 20200111140321.959 ERROR    cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1287 - File exists - Failed to create directory "/sys/fs/cgroup/unified//lxc/810-2"
lxc-start 810 20200111140321.960 ERROR    cgfsng - cgroups/cgfsng.c:container_create_path_for_hierarchy:1336 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc/810-2"
lxc-start 810 20200111140321.961 ERROR    cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1496 - Failed to create cgroup "/sys/fs/cgroup/unified//lxc/810-2"
lxc-start 810 20200111140321.571 INFO     cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1510 - The container process uses "lxc/810-3" as cgroup

which probably means the namespace is getting confused and creating ID-num (810-3) directories where it should create only ID directories (810)

this happens when a container process dies before lxc can clean it (leftover from the last boot)

check the directories in /sys/fs/cgroup/*/lxc/ID and stop the affected containers. clean these duplicate directories afterwards. you might have to possibly reboot the system as well.

you can clean the directories with the following: find /sys/fs/cgroup/*/lxc/ID* -depth -type d -print -delete . replace 'ID' with the actual ID of the container

 

[andwoo8182]

Hi, thanks very much for getting back to me. I had originally come across the directory issue, but I thought perhaps the sub-directories would be handledl - in any case, when I tried deleting them previously I think I didnt realise all the directories that needed to be deleted. That command worked well & I cleared all the folders when attempting to start more containers - please see attached the logs for another container that is failing to start - but now gets past those mkdir issues.

The only log entries that catch my eye are these ones, but again I am not sure where to start:

lxc-start 845 20200116155627.868 WARN cgfsng - cgroups/cgfsng.c:chowmod:1569 - No such file or directory - Failed to chown(/sys/fs/cgroup/unified//lxc/845/memory.oom.group, 65536, 0)

lxc-start 845 20200116155631.410 WARN cgfsng - cgroups/cgfsng.c:chowmod:1569 - No such file or directory - Failed to chown(/sys/fs/cgroup/unified//lxc/845/ns/memory.oom.group, 65536, 0)

lxc-start 845 20200116155631.627 ERROR conf - conf.c:lxc_setup_boot_id:3527 - Permission denied - Failed to mount /dev/.lxc-boot-id to /proc/sys/kernel/random/boot_id

Thanks again for the guidance!

 

[oguz]

hi again,

this part is probably the main reason the container fails:

lxc-start 845 20200116155631.629 ERROR    seccomp - seccomp.c:lxc_seccomp_load:1252 - Unknown error 524 - Error loading the seccomp policy
lxc-start 845 20200116155631.629 ERROR    sync - sync.c:__sync_wait:62 - An error occurred in another process (expected sequence number 5)

so it looks like there's some problem loading seccomp.

do you have some custom lxc profile or seccomp policy for this container?

you can post your pveversion -v and pct config CTID and if you have any custom lxc profile / seccomp policy post those as well please

 

[andwoo8182]

Hi, apologies for not getting back to you on this thread. I brought a new host online using new architecture, and it presented multiple new issues that looks a long time to work out & resolve, some of which were solved by the new 5.4 kernel. In regards to this issue I was encountering, it has been resolved here:

https://discuss.linuxcontainers.org/t/lxd-cannot-scale-over-597-containers-seccomp-errors/7046

I think this issue should be resolved for me now (busy testing now). This document was useful in ensuring that all kernel parameters were set correctly for a high container production system:

https://linuxcontainers.org/lxd/docs/master/production-setup

The addition that resolves my issue is:

/etc/sysctl.conf

net.core.bpf_jit_limit 3000000000

Thanks!