Container with mount point randomly not starting at boot

[Fathi]

Hi,
I have a poc proxmox server embedded on a train and running some containers and virtual machines.
One of this containers have huge data that I don't want to backup, so I put it on a separate mount point marked to not beeing backed up.
The proxmox server is setup to start when ups gets eletric power. The problem I am facing is that many times when ups gets elevtric power, proxmox starts but this container doesn't while all other containers and VM do.
In proxmox logs, I get the following:
Feb 1 09:27:36 pve-emu-003 kernel: [17406.208327] audit: type=1400 audit(1549009656.019:17): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-100_</var/lib/lxc>" pid=16242 comm="apparmor_parser"
Feb 1 09:27:38 pve-emu-003 kernel: [17408.208980] audit: type=1400 audit(1549009658.019:18): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=16470 comm="mount" flags="rw, remount"
Feb 1 09:27:51 pve-emu-003 kernel: [17421.566511] audit: type=1400 audit(1549009671.379:19): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=17139 comm="(hitch)" flags="rw, rslave"
Feb 1 09:27:51 pve-emu-003 kernel: [17421.571902] audit: type=1400 audit(1549009671.383:20): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=17142 comm="(openvpn)" flags="rw, rslave"
Feb 1 09:27:51 pve-emu-003 kernel: [17421.728114] audit: type=1400 audit(1549009671.539:21): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=17143 comm="(varnishd)" flags="rw, rslave"
Feb 1 10:07:41 pve-emu-003 kernel: [19811.533207] audit: type=1400 audit(1549012061.365:24): apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc-100_</var/lib/lxc>" pid=3630 comm="apparmor_parser"
Feb 1 10:10:19 pve-emu-003 kernel: [ 44.973357] audit: type=1400 audit(1549012219.598:12): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-100_</var/lib/lxc>" pid=2357 comm="apparmor_parser"
Feb 1 10:10:24 pve-emu-003 kernel: [ 49.653137] audit: type=1400 audit(1549012224.278:13): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=2688 comm="mount" flags="rw, remount"
Feb 1 10:10:36 pve-emu-003 kernel: [ 61.743090] audit: type=1400 audit(1549012236.366:14): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=3371 comm="(openvpn)" flags="rw, rslave"
Feb 1 10:10:36 pve-emu-003 kernel: [ 62.057382] audit: type=1400 audit(1549012236.682:15): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=3374 comm="(varnishd)" flags="rw, rslave"
Feb 1 10:10:37 pve-emu-003 kernel: [ 62.790578] audit: type=1400 audit(1549012237.414:16): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=3377 comm="(hitch)" flags="rw, rslave"
Feb 1 18:17:29 pve-emu-003 kernel: [29274.356377] audit: type=1400 audit(1549041449.715:22): apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc-100_</var/lib/lxc>" pid=13566 comm="apparmor_parser"
Feb 1 18:39:22 pve-emu-003 kernel: [ 45.615838] audit: type=1400 audit(1549042762.239:12): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-100_</var/lib/lxc>" pid=2440 comm="apparmor_parser"
Feb 1 18:39:27 pve-emu-003 kernel: [ 50.692219] audit: type=1400 audit(1549042767.319:13): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=2814 comm="mount" flags="rw, remount"
Feb 1 18:39:39 pve-emu-003 kernel: [ 62.818157] audit: type=1400 audit(1549042779.443:14): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=3453 comm="(openvpn)" flags="rw, rslave"
Feb 1 18:39:39 pve-emu-003 kernel: [ 63.229681] audit: type=1400 audit(1549042779.855:15): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=3456 comm="(varnishd)" flags="rw, rslave"
Feb 1 18:39:39 pve-emu-003 kernel: [ 63.315849] audit: type=1400 audit(1549042779.939:16): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=3459 comm="(hitch)" flags="rw, rslave"
Feb 1 22:43:50 pve-emu-003 kernel: [ 134.603629] audit: type=1400 audit(1549057430.227:12): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-100_</var/lib/lxc>" pid=2861 comm="apparmor_parser"
Feb 1 22:43:51 pve-emu-003 kernel: [ 135.592779] audit: type=1400 audit(1549057431.219:13): apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc-100_</var/lib/lxc>" pid=3044 comm="apparmor_parser"
Feb 2 06:35:10 pve-emu-003 kernel: [ 48.402957] audit: type=1400 audit(1549085710.019:12): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-100_</var/lib/lxc>" pid=2462 comm="apparmor_parser"
Feb 2 06:35:14 pve-emu-003 kernel: [ 53.110372] audit: type=1400 audit(1549085714.727:13): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=2741 comm="mount" flags="rw, remount"
Feb 2 06:35:26 pve-emu-003 kernel: [ 65.211671] audit: type=1400 audit(1549085726.827:14): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=3472 comm="(varnishd)" flags="rw, rslave"
Feb 2 06:35:26 pve-emu-003 kernel: [ 65.283222] audit: type=1400 audit(1549085726.899:15): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=3473 comm="(hitch)" flags="rw, rslave"
Feb 2 06:35:26 pve-emu-003 kernel: [ 65.321553] audit: type=1400 audit(1549085726.939:16): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=3477 comm="(openvpn)" flags="rw, rslave"
Feb 5 04:25:44 pve-emu-003 kernel: [ 137.099427] audit: type=1400 audit(1549337144.715:12): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-100_</var/lib/lxc>" pid=2938 comm="apparmor_parser"
Feb 5 04:25:45 pve-emu-003 kernel: [ 138.053255] audit: type=1400 audit(1549337145.671:13): apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc-100_</var/lib/lxc>" pid=3121 comm="apparmor_parser"
Feb 5 12:38:29 pve-emu-003 kernel: [29701.913929] audit: type=1400 audit(1549366709.535:17): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-100_</var/lib/lxc>" pid=29668 comm="apparmor_parser"
Feb 5 12:38:31 pve-emu-003 kernel: [29703.717392] audit: type=1400 audit(1549366711.339:18): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=29879 comm="mount" flags="rw, remount"
Feb 5 12:38:44 pve-emu-003 kernel: [29717.129045] audit: type=1400 audit(1549366724.751:19): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=30530 comm="(varnishd)" flags="rw, rslave"
Feb 5 12:38:44 pve-emu-003 kernel: [29717.129453] audit: type=1400 audit(1549366724.751:20): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=30531 comm="(openvpn)" flags="rw, rslave"
Feb 5 12:38:44 pve-emu-003 kernel: [29717.130787] audit: type=1400 audit(1549366724.755:21): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/" pid=30534 comm="(hitch)" flags="rw, rslave"
Feb 6 04:14:47 pve-emu-003 kernel: [ 136.950746] audit: type=1400 audit(1549422887.570:12): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-100_</var/lib/lxc>" pid=2903 comm="apparmor_parser"
Feb 6 04:14:48 pve-emu-003 kernel: [ 137.936372] audit: type=1400 audit(1549422888.558:13): apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc-100_</var/lib/lxc>" pid=3091 comm="apparmor_parser"

Could someone help me please ?
TIA.

 

[oguz]

Which distribution is running on this container? Can you start the container manually, or does it not start at all?

Could you send the container config please? You can get it by running:

pct config CTID

where CTID is the ID of the container.

My guess is there's some problem with systemd. You might try to work around it by enabling "Nesting" in the Container Features.

In GUI: Container -> Options -> Features -> Nesting

or you can add:

features: nesting=1

to the Container Config File.

 

[Fathi]

Hi @oguz,
I don't remember having trouble starting the container manually. Also, some days the container starts without problems but some other days doesn't start at all. Today, we had to send someone on train to start it manually.
Here after is my config:

arch: amd64
cores: 4
description: Emby on CentOS 7.%0A
hostname: mediaflix
memory: 6144
mp0: local-lvm:vm-100-disk-2,mp=/opt/Emby,size=500G
nameserver: 1.1.1.1 8.8.8.8
net0: name=eth0,bridge=vmbr0,hwaddr=02:D4:00:46:9F:01,ip=10.255.255.254/8,type=veth
net1: name=eth1,bridge=vmbr2,gw=192.168.3.101,hwaddr=8E:BA:F0:43:0A:AC,ip=192.168.3.100/24,type=veth
onboot: 1
ostype: centos
protection: 1
rootfs: local-lvm:vm-100-disk-1,size=8G
searchdomain: sncftwifi.eu.org
startup: order=1
swap: 6144
lxc.cgroup.devices.allow: c 10:200 rwm
lxc.hook.autodev: sh -c "modprobe tun; cd ${LXC_ROOTFS_MOUNT}/dev; mkdir net; mknod net/tun c 10 200; chmod 0666 net/tun"

By the way I have another identical container on another computer with much less horse power but which doesn't need to restart every day as it is set up in a train station, also as a poc.

 

[oguz]

You can try following my suggestion with the Nesting feature and see if that makes any difference.

I'm afraid I can't diagnose your issue thoroughly, other than the fact that some mounts are not being done properly and enabling the nesting feature might solve it.

A (hacky and ugly) solution is running something like "pct start CTID" some time after boot (via cron for example). This is the same as starting the container manually, so if there are no problems with that, it might just work.

EDIT:

One possibility is the lxc.hook.autodev you have. It might be that the modprobe doesn't work so soon after booting, and that could be why it hangs in the beginning. It would also explain why it works when manually started, since the modprobe would then work.

 

[Fathi]

Hi @oguz and thank you,
I prefer to test nesting as last resort. This is a resource hungry container which could hang all the proxmox if given direct access to cpu.
So, as first I will try to delay starting this container as it is set to start first, then if this doesn't work, I will try the nesting feature.
Best regards.

 

[jim.bond.9862]

I am just curious, but why do you have a swap the size of your ram in ct ?
Wouldn't be more prudent to add more ram.

 

[Fathi]

I am just curious, but why do you have a swap the size of your ram in ct ?
Wouldn't be more prudent to add more ram.

Hi @jim.bond.9862 what values do you suggest for swap in general ? From what I have understood from another post, the real swap size is the difference between the values in the Proxmox Gui (memory - swap). So in my case 6144 - 6144 = 0.

 

[jim.bond.9862]

Not sure what you mean here. But on a real pc I usually balance swap to ram ratio based on the real ram. If i have 8gb or more on a Linux machine I put swap at 1gb or 2 gb. Good to have but hope to never use. I am talking about workstation setup.
If I have less than 8gb I make swap eather half the ram or , if I know the pc will be heavily used, swap equal ram.
I try to do the same for .VMs but in my home lab I mostly divide things in different VM so all my home setups have 1 or 2 gb of swap l. Mostly I have no swap at all.
Except on host. Since you can overcomit memory for VMs I feel that host may need swap to keep things running.
Don't know if I am right or wrong but it have worked so far.