Hi,
I'm a bit worried as I looked in dmesg and got this:
282316.641808] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[282316.642871] fwbr102i0: port 2(veth102i0) entered blocking state
[282316.643784] fwbr102i0: port 2(veth102i0) entered forwarding state
[282316.645844] audit: type=1400 audit(1598994079.025:262): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-102_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=31111 comm="(resolved)" srcname="/" flags="rw, rbind"
[282316.724247] audit: type=1400 audit(1598994079.105:263): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=31053 comm="apparmor_parser"
[282316.798525] audit: type=1400 audit(1598994079.181:264): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=30981 comm="apparmor_parser"
[282316.873708] audit: type=1400 audit(1598994079.253:265): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=31479 comm="apparmor_parser"
[282316.884778] audit: type=1400 audit(1598994079.265:266): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=31481 comm="apparmor_parser"
[283132.304116] fwbr102i0: port 2(veth102i0) entered disabled state
[283132.319490] device veth102i0 left promiscuous mode
[283132.320521] fwbr102i0: port 2(veth102i0) entered disabled state
[283132.575581] kauditd_printk_skb: 5 callbacks suppressed
[283132.575582] audit: type=1400 audit(1598994894.929:272): apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc-102_</var/lib/lxc>" pid=32332 comm="apparmor_parser"
[283133.869125] fwbr102i0: port 1(fwln102i0) entered disabled state
[283133.870122] vmbr0: port 4(fwpr102p0) entered disabled state
[283133.885305] device fwln102i0 left promiscuous mode
[283133.886323] fwbr102i0: port 1(fwln102i0) entered disabled state
[283133.922571] device fwpr102p0 left promiscuous mode
[283133.923656] vmbr0: port 4(fwpr102p0) entered disabled state
[283134.522127] EXT4-fs (loop1): Mount option "noacl" will be removed by 3.5
Contact linux-ext4@vger.kernel.org if you think we should keep it.
[283134.573694] EXT4-fs (loop1): mounted filesystem with ordered data mode. Opts: noacl
[283134.884442] audit: type=1400 audit(1598994897.237:273): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-102_</var/lib/lxc>" pid=6622 comm="apparmor_parser"
[283135.224005] fwbr102i0: port 1(fwln102i0) entered blocking state
[283135.225050] fwbr102i0: port 1(fwln102i0) entered disabled state
[283135.227080] device fwln102i0 entered promiscuous mode
[283135.228040] fwbr102i0: port 1(fwln102i0) entered blocking state
[283135.228955] fwbr102i0: port 1(fwln102i0) entered forwarding state
[283135.234529] vmbr0: port 4(fwpr102p0) entered blocking state
[283135.235422] vmbr0: port 4(fwpr102p0) entered disabled state
[283135.236324] device fwpr102p0 entered promiscuous mode
[283135.237144] vmbr0: port 4(fwpr102p0) entered blocking state
[283135.237916] vmbr0: port 4(fwpr102p0) entered forwarding state
[283135.243716] fwbr102i0: port 2(veth102i0) entered blocking state
[283135.244520] fwbr102i0: port 2(veth102i0) entered disabled state
[283135.245379] device veth102i0 entered promiscuous mode
[283135.272735] eth0: renamed from vethqnihU5
[283137.612362] audit: type=1400 audit(1598994899.964:274): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-102_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=11643 comm="(networkd)" srcname="/" flags="rw, rbind"
[283137.792527] audit: type=1400 audit(1598994900.144:275): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=11873 comm="apparmor_parser"
[283137.803365] audit: type=1400 audit(1598994900.156:276): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=11869 comm="apparmor_parser"
[283137.826997] audit: type=1400 audit(1598994900.180:277): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=11905 comm="apparmor_parser"
[283137.828156] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[283137.829145] audit: type=1400 audit(1598994900.180:278): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=11905 comm="apparmor_parser"
[283137.829986] fwbr102i0: port 2(veth102i0) entered blocking state
[283137.832966] fwbr102i0: port 2(veth102i0) entered forwarding state
[283137.836621] audit: type=1400 audit(1598994900.188:279): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-102_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=12043 comm="(resolved)" srcname="/" flags="rw, rbind"
[283137.927333] audit: type=1400 audit(1598994900.280:280): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=11878 comm="apparmor_parser"
[283137.929943] audit: type=1400 audit(1598994900.280:281): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=11992 comm="apparmor_parser"
[283138.045394] audit: type=1400 audit(1598994900.396:282): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=12337 comm="apparmor_parser"
[283138.048922] audit: type=1400 audit(1598994900.396:283): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=12337 comm="apparmor_parser"
[283591.692403] fwbr102i0: port 2(veth102i0) entered disabled state
[283591.707712] device veth102i0 left promiscuous mode
[283591.708928] fwbr102i0: port 2(veth102i0) entered disabled state
[283591.961972] kauditd_printk_skb: 5 callbacks suppressed
[283591.961974] audit: type=1400 audit(1598995354.301:289): apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc-102_</var/lib/lxc>" pid=1090 comm="apparmor_parser"
[283593.242151] fwbr102i0: port 1(fwln102i0) entered disabled state
[283593.243176] vmbr0: port 4(fwpr102p0) entered disabled state
[283593.258401] device fwln102i0 left promiscuous mode
[283593.259247] fwbr102i0: port 1(fwln102i0) entered disabled state
[283593.298111] device fwpr102p0 left promiscuous mode
[283593.298963] vmbr0: port 4(fwpr102p0) entered disabled state
[285906.930583] audit: type=1400 audit(1598997669.191:290): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=15384 comm="(ogrotate)" srcname="/" flags="rw, rbind"
[372249.771516] audit: type=1400 audit(1599084009.194:291): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=24077 comm="(ogrotate)" srcname="/" flags="rw, rbind"
As you can see there's a lot of what I believe are errors. I have looked up the nic interfaces going up and down which seems normal when LXC's are starting up, but the apparmor stuff that are associated with privileged container lxc-100 is worrying. I have to run it privileged as I need full bidirectional RW access between host and LXC in combination with a tricky FUSE-unionfs setup. There were no other way to do this than making it privileged.
My question is: how can I find out the problematic service, and why is this service creating this issue?
I'm running the latest PVE on an intel system with the latest updates.
I'm a bit worried as I looked in dmesg and got this:
282316.641808] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[282316.642871] fwbr102i0: port 2(veth102i0) entered blocking state
[282316.643784] fwbr102i0: port 2(veth102i0) entered forwarding state
[282316.645844] audit: type=1400 audit(1598994079.025:262): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-102_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=31111 comm="(resolved)" srcname="/" flags="rw, rbind"
[282316.724247] audit: type=1400 audit(1598994079.105:263): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=31053 comm="apparmor_parser"
[282316.798525] audit: type=1400 audit(1598994079.181:264): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=30981 comm="apparmor_parser"
[282316.873708] audit: type=1400 audit(1598994079.253:265): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=31479 comm="apparmor_parser"
[282316.884778] audit: type=1400 audit(1598994079.265:266): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=31481 comm="apparmor_parser"
[283132.304116] fwbr102i0: port 2(veth102i0) entered disabled state
[283132.319490] device veth102i0 left promiscuous mode
[283132.320521] fwbr102i0: port 2(veth102i0) entered disabled state
[283132.575581] kauditd_printk_skb: 5 callbacks suppressed
[283132.575582] audit: type=1400 audit(1598994894.929:272): apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc-102_</var/lib/lxc>" pid=32332 comm="apparmor_parser"
[283133.869125] fwbr102i0: port 1(fwln102i0) entered disabled state
[283133.870122] vmbr0: port 4(fwpr102p0) entered disabled state
[283133.885305] device fwln102i0 left promiscuous mode
[283133.886323] fwbr102i0: port 1(fwln102i0) entered disabled state
[283133.922571] device fwpr102p0 left promiscuous mode
[283133.923656] vmbr0: port 4(fwpr102p0) entered disabled state
[283134.522127] EXT4-fs (loop1): Mount option "noacl" will be removed by 3.5
Contact linux-ext4@vger.kernel.org if you think we should keep it.
[283134.573694] EXT4-fs (loop1): mounted filesystem with ordered data mode. Opts: noacl
[283134.884442] audit: type=1400 audit(1598994897.237:273): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-102_</var/lib/lxc>" pid=6622 comm="apparmor_parser"
[283135.224005] fwbr102i0: port 1(fwln102i0) entered blocking state
[283135.225050] fwbr102i0: port 1(fwln102i0) entered disabled state
[283135.227080] device fwln102i0 entered promiscuous mode
[283135.228040] fwbr102i0: port 1(fwln102i0) entered blocking state
[283135.228955] fwbr102i0: port 1(fwln102i0) entered forwarding state
[283135.234529] vmbr0: port 4(fwpr102p0) entered blocking state
[283135.235422] vmbr0: port 4(fwpr102p0) entered disabled state
[283135.236324] device fwpr102p0 entered promiscuous mode
[283135.237144] vmbr0: port 4(fwpr102p0) entered blocking state
[283135.237916] vmbr0: port 4(fwpr102p0) entered forwarding state
[283135.243716] fwbr102i0: port 2(veth102i0) entered blocking state
[283135.244520] fwbr102i0: port 2(veth102i0) entered disabled state
[283135.245379] device veth102i0 entered promiscuous mode
[283135.272735] eth0: renamed from vethqnihU5
[283137.612362] audit: type=1400 audit(1598994899.964:274): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-102_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=11643 comm="(networkd)" srcname="/" flags="rw, rbind"
[283137.792527] audit: type=1400 audit(1598994900.144:275): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=11873 comm="apparmor_parser"
[283137.803365] audit: type=1400 audit(1598994900.156:276): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=11869 comm="apparmor_parser"
[283137.826997] audit: type=1400 audit(1598994900.180:277): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=11905 comm="apparmor_parser"
[283137.828156] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[283137.829145] audit: type=1400 audit(1598994900.180:278): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=11905 comm="apparmor_parser"
[283137.829986] fwbr102i0: port 2(veth102i0) entered blocking state
[283137.832966] fwbr102i0: port 2(veth102i0) entered forwarding state
[283137.836621] audit: type=1400 audit(1598994900.188:279): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-102_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=12043 comm="(resolved)" srcname="/" flags="rw, rbind"
[283137.927333] audit: type=1400 audit(1598994900.280:280): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=11878 comm="apparmor_parser"
[283137.929943] audit: type=1400 audit(1598994900.280:281): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=11992 comm="apparmor_parser"
[283138.045394] audit: type=1400 audit(1598994900.396:282): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=12337 comm="apparmor_parser"
[283138.048922] audit: type=1400 audit(1598994900.396:283): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined" pid=12337 comm="apparmor_parser"
[283591.692403] fwbr102i0: port 2(veth102i0) entered disabled state
[283591.707712] device veth102i0 left promiscuous mode
[283591.708928] fwbr102i0: port 2(veth102i0) entered disabled state
[283591.961972] kauditd_printk_skb: 5 callbacks suppressed
[283591.961974] audit: type=1400 audit(1598995354.301:289): apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc-102_</var/lib/lxc>" pid=1090 comm="apparmor_parser"
[283593.242151] fwbr102i0: port 1(fwln102i0) entered disabled state
[283593.243176] vmbr0: port 4(fwpr102p0) entered disabled state
[283593.258401] device fwln102i0 left promiscuous mode
[283593.259247] fwbr102i0: port 1(fwln102i0) entered disabled state
[283593.298111] device fwpr102p0 left promiscuous mode
[283593.298963] vmbr0: port 4(fwpr102p0) entered disabled state
[285906.930583] audit: type=1400 audit(1598997669.191:290): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=15384 comm="(ogrotate)" srcname="/" flags="rw, rbind"
[372249.771516] audit: type=1400 audit(1599084009.194:291): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-100_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=24077 comm="(ogrotate)" srcname="/" flags="rw, rbind"
As you can see there's a lot of what I believe are errors. I have looked up the nic interfaces going up and down which seems normal when LXC's are starting up, but the apparmor stuff that are associated with privileged container lxc-100 is worrying. I have to run it privileged as I need full bidirectional RW access between host and LXC in combination with a tricky FUSE-unionfs setup. There were no other way to do this than making it privileged.
My question is: how can I find out the problematic service, and why is this service creating this issue?
I'm running the latest PVE on an intel system with the latest updates.