Very frequent AppArmor logs after updating PVE

R

Raphael11

New Member
Jan 14, 2025
2
0
1
Hello everyone,

I have a problem after updating Proxmox. More precisely, I only noticed it after the update, so I’m fairly certain it was caused by it. I updated from 9.0.0 to 9.1.1, and since then I’ve been getting these messages in the system log every few seconds:

Code: 
Nov 23 17:06:51 pve1 kernel: audit: type=1400 audit(1763914011.755:37407): apparmor="DENIED" operation="sendmsg" class="file" namespace="root//lxc-102_<-var-lib-lxc>" profile="rsyslogd" name="/run/systemd/journal/dev-log" pid=387332 comm="systemd-journal" requested_mask="r" denied_mask="r" fsuid=100000 ouid=100000
Nov 23 17:06:51 pve1 kernel: audit: type=1400 audit(1763914011.759:37408): apparmor="DENIED" operation="sendmsg" class="file" namespace="root//lxc-102_<-var-lib-lxc>" profile="rsyslogd" name="/run/systemd/journal/dev-log" pid=387332 comm="systemd-journal" requested_mask="r" denied_mask="r" fsuid=100000 ouid=100000
Nov 23 17:06:51 pve1 kernel: audit: type=1400 audit(1763914011.769:37409): apparmor="DENIED" operation="sendmsg" class="file" namespace="root//lxc-102_<-var-lib-lxc>" profile="rsyslogd" name="/run/systemd/journal/dev-log" pid=387332 comm="systemd-journal" requested_mask="r" denied_mask="r" fsuid=100000 ouid=100000
Nov 23 17:06:51 pve1 kernel: audit: type=1400 audit(1763914011.789:37410): apparmor="DENIED" operation="sendmsg" class="file" namespace="root//lxc-102_<-var-lib-lxc>" profile="rsyslogd" name="/run/systemd/journal/dev-log" pid=387332 comm="systemd-journal" requested_mask="r" denied_mask="r" fsuid=100000 ouid=100000
Nov 23 17:06:51 pve1 kernel: audit: type=1400 audit(1763914011.806:37411): apparmor="DENIED" operation="sendmsg" class="file" namespace="root//lxc-102_<-var-lib-lxc>" profile="rsyslogd" name="/run/systemd/journal/dev-log" pid=387332 comm="systemd-journal" requested_mask="r" denied_mask="r" fsuid=100000 ouid=100000
Nov 23 17:06:52 pve1 kernel: audit: type=1400 audit(1763914012.000:37412): apparmor="DENIED" operation="sendmsg" class="file" namespace="root//lxc-102_<-var-lib-lxc>" profile="rsyslogd" name="/run/systemd/journal/dev-log" pid=387332 comm="systemd-journal" requested_mask="r" denied_mask="r" fsuid=100000 ouid=100000
Nov 23 17:06:52 pve1 kernel: audit: type=1400 audit(1763914012.010:37413): apparmor="DENIED" operation="sendmsg" class="file" namespace="root//lxc-102_<-var-lib-lxc>" profile="rsyslogd" name="/run/systemd/journal/dev-log" pid=387332 comm="systemd-journal" requested_mask="r" denied_mask="r" fsuid=100000 ouid=100000


I’ve already checked several threads, but I didn’t really understand the actual solution:

https://forum.proxmox.com/threads/o...le-on-test-no-subscription.173920/post-809360

This post most likely describes my issue, but I don’t know which file I need to modify to allow access.

The container is running CasaOS, meaning Docker is running inside it, and it seems to write logs very frequently.

Thank you in advance for any help.

Kind regards
 
Hi,
the file should be /etc/apparmor.d/usr.sbin.rsyslogd inside the container. If it does not exist, look for other files containing rsyslogd in the same directory.
 
I don't understand the actual solution.
So it is apt purge apparmor in the container ?
 
Hi,
Temtaime said:
I don't understand the actual solution.
So it is apt purge apparmor in the container ?
Click to expand...
no. The solution is editing the relevant profile file inside the container. See my previous response and the link in the first post.
 
fiona said:
Hi,
the file should be /etc/apparmor.d/usr.sbin.rsyslogd inside the container. If it does not exist, look for other files containing rsyslogd in the same directory.
Click to expand...

Hi, sorry I'm not really into apparmor but apparmor is a kernel feature, and on an LXC ("inside the container") tuning the apparmor config files should have no effect ? Did you mean tuning the hosts' (the pve) config file ?
Or do I simply need more experience to fully understand, in which case could you please elaborate a bit for us non-specialists ?

thanks @fiona

For context this error the OP talks about is actually a rather critical one ; indeed it seems to happen when a debian hypervisor with apparmor ( pve ) hosts a ubuntu 24 instance (possibly other as well, didn't investigate). It breaks part of rsyslogd, which seems harmless because it starts and doesn't complain, but with the side effect to actually break logging of many processes relying on rsyslogd (postfix smtpd, ...). The log files of those daemons/processes are just ... empty. And nothing complains about it due to the intricacies of modern logging systems, don't get me started on systemd.
tldr : this simple error actually "silently kills" logging of many processes inside Ubuntu lxc containers in PVE - something that, I think, is a very frequent setup.

I discovered this issue specifically on a mail relay (post fix smtpd) where the mail.log file refused to log new info/connections, without any error, smptd perfectly working, rsyslogd working, etc. I'm sure this affects MANY other setups, and the fact this is entirely silent (on the container) makes it dangerous.
 
Last edited:
weppa said:
For context this error the OP talks about is actually a rather critical one ; indeed it seems to happen when a debian hypervisor with apparmor ( pve ) hosts a ubuntu 24 instance (possibly other as well, didn't investigate). It breaks part of rsyslogd, which seems harmless because it starts and doesn't complain, but with the side effect to actually break logging of many processes relying on rsyslogd (postfix smtpd, ...). The log files of those daemons/processes are just ... empty. And nothing complains about it due to the intricacies of modern logging systems, don't get me started on systemd.
tldr : this simple error actually "silently kills" logging of many processes inside Ubuntu lxc containers in PVE - something that, I think, is a very frequent setup.

I discovered this issue specifically on a mail relay (post fix smtpd) where the mail.log file refused to log new info/connections, without any error, smptd perfectly working, rsyslogd working, etc. I'm sure this affects MANY other setups, and the fact this is entirely silent (on the container) makes it dangerous.
Click to expand...
Yes, I ran into the same unfortunate situation when I noticed none of my guests' syslog files had anything in them all of a sudden. My temporary workaround was to add lxc.apparmor.profile: unconfined to/etc/pve/lxc/*.conf and cross my fingers and hope the underlying issue eventually gets addressed.
 
Last edited:
  • Like
Reactions: Schattenwelt and spacedragonspirit
It seems to be a bug in the package rsyslogd 's apparmor config, inside ubuntu.
It's been filed and corrected in september 2025 but, to date, the package hasn't been deployed to ubuntu 24

I don't really understand how this turns out to be a "PVE problem" once we run ubuntu in an LXC but as I said i'm not an expert in this matter... I don't understand how a kernel problem can be fixed container side in an lxc config

As far as i'm concerned I fixed the issue (as suggested above) by editing the apparmor config in the containers but I had to make this edit dozens of times + I don't think fixing the package's apparmor config survives package updates ( we should have used a local apparmor policy which fiona didn't suggest)

Interestingly, as you mentionned this can be fixed PVE side as well (as I was thinking), thanks for sharing @leeh
 
Last edited:
  • Like
Reactions: fiona
weppa said:
Interestingly, as you mentionned this can be fixed PVE side as well (as I was thinking), thanks for sharing @leeh
Click to expand...
I would not consider setting unconfined to be a "real" fix, because you lose security guarantees..
 
weppa said:
It seems to be a bug in the package rsyslogd 's apparmor config, inside ubuntu.
It's been filed and corrected in september 2025 but, to date, the package hasn't been deployed to ubuntu 24

I don't really understand how this turns out to be a "PVE problem" once we run ubuntu in an LXC but as I said i'm not an expert in this matter... I don't understand how a kernel problem can be fixed container side in an lxc config

As far as i'm concerned I fixed the issue (as suggested above) by editing the apparmor config in the containers but I had to make this edit dozens of times + I don't think fixing the package's apparmor config survives package updates ( we should have used a local apparmor policy which fiona didn't suggest)

Interestingly, as you mentionned this can be fixed PVE side as well (as I was thinking), thanks for sharing @leeh
Click to expand...
Failing on my google skills :-(. Do you have any ID / referens to the fix that is implemented (in sep 2025) so one could track it and make the containers "confined" again when fixed?
 
Hi,
rosendal said:
Failing on my google skills :-(. Do you have any ID / referens to the fix that is implemented (in sep 2025) so one could track it and make the containers "confined" again when fixed?
Click to expand...
you don't have to make the container unconfined. Just allow read access to the single file in the apparmor profile for rsyslogd as described here:
fiona

Post in thread 'Opt-in Linux 6.17 Kernel for Proxmox VE 9 available on test & no-subscription'

bossanova808 said:
But - I added abi <abi/3.0>, to the top of /etc/apparmor.d/usr.sbin.rsyslogd in the container, then reload-or-restart the apparmor service, then for good measure rebooted the container.

No change in behaviour, dmesg is still completely full of hundreds/thousands of:

[65656.861041] audit: type=1400 audit(1760651059.654:590680): apparmor="DENIED" operation="sendmsg" class="file" namespace="root//lxc-205_<-var-lib-lxc>" profile="rsyslogd" name="/run/systemd/journal/dev-log" pid=2964609 comm="systemd-journal" requested_mask="r" denied_mask="r" fsuid=100000 ouid=100000...
Click to expand...
  • Like
 
Came across the same problem with Ubuntu 24.04 LXC on Proxmox 9.1-1.
The AppArmor profile for rsyslogd is broken and spams denials like .../run/systemd/journal/dev-log.

The fix that worked for me and survives package upgrades:

<code> apparmor_parser -r /etc/apparmor.d/usr.sbin.rsyslogd<br></span><span><span>sudo</span></span><span> systemctl reload rsyslog<br></span></code>

After this, all AppArmor spam stopped and logging inside the container works normally again.

Not sure how to apply properly, did reboot (downtime not an issue for me)
 
You must log in or register to reply here.
Top Bottom