Apparmor problems on Debian 11 Install

T

TGT_9000

New Member
Apr 25, 2023
7
0
1
Hello,

I have followed the instructions HERE to install Proxmox packages on top of a fully updated and upgraded Debian 11 stable image.

I am able to log in to the node via the web interface, join it to my cluster, and adopt it into my Ceph cluster.

I am running into an issue where I can not start or migrate an LXC into this node. I get similar errors to those described HERE but I am seeing this despite the pve-lxc >= 5.0.2-2 "fix" being present.

This may be related to an error message I am seeing onscreen on boot.

20230424_234128_small.jpg


Code: 
root@thinnode:/etc/pve/lxc# ls
100.conf
root@thinnode:/etc/pve/lxc# pct config 100
bash: pct: command not found
root@thinnode:/etc/pve/lxc# cat 100.conf
arch: amd64
cores: 4
features: nesting=1
hostname: ThinTest
memory: 4012
net0: name=eth0,bridge=vmbr0,hwaddr=BE:32:2F:AF:6B:05,ip=dhcp,type=veth
ostype: ubuntu
rootfs: local:100/vm-100-disk-0.raw,size=8G
swap: 0
unprivileged: 1
root@thinnode:/etc/pve/lxc# pveversion -v
proxmox-ve: 7.4-1 (running kernel: 5.10.0-21-amd64)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.15: 7.4-1
pve-kernel-5.15.104-1-pve: 5.15.104-2
ceph: 17.2.5-pve1
ceph-fuse: 17.2.5-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-4
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.5
libpve-storage-perl: 7.4-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.1-1
proxmox-backup-file-restore: 2.4.1-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-widget-toolkit: 3.6.5
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-2
pve-firewall: 4.3-1
pve-firmware: 3.6-4
pve-ha-manager: 3.6.0
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1
root@thinnode:/etc/pve/lxc#
root@thinnode:/etc/pve/lxc# cat /tmp/lxc-100.log
lxc-start 100 20230425032310.992 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2273 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 100 20230425032310.992 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2273 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 100 20230425032310.993 INFO     lxccontainer - ../src/lxc/lxccontainer.c:do_lxcapi_start:998 - Set process title to [lxc monitor] /var/lib/lxc 100
lxc-start 100 20230425032310.993 DEBUG    lxccontainer - ../src/lxc/lxccontainer.c:wait_on_daemonized_start:859 - First child 3673 exited
lxc-start 100 20230425032310.994 INFO     lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
lxc-start 100 20230425032310.994 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "100", config section "lxc"
lxc-start 100 20230425032311.623 INFO     cgfsng - ../src/lxc/cgroups/cgfsng.c:unpriv_systemd_create_scope:1227 - Running privileged, not using a systemd unit
lxc-start 100 20230425032311.623 DEBUG    seccomp - ../src/lxc/seccomp.c:parse_config_v2:656 - Host native arch is [3221225534]
lxc-start 100 20230425032311.623 INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
lxc-start 100 20230425032311.623 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
lxc-start 100 20230425032311.623 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
lxc-start 100 20230425032311.623 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
lxc-start 100 20230425032311.623 INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "[all]"
lxc-start 100 20230425032311.623 INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "kexec_load errno 1"
lxc-start 100 20230425032311.623 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[246:kexec_load] action[327681:errno] arch[0]
lxc-start 100 20230425032311.623 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741827]
lxc-start 100 20230425032311.623 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741886]
lxc-start 100 20230425032311.623 INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "open_by_handle_at errno 1"
lxc-start 100 20230425032311.623 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[304:open_by_handle_at] action[327681:errno] arch[0]
lxc-start 100 20230425032311.623 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741827]
lxc-start 100 20230425032311.623 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741886]
lxc-start 100 20230425032311.623 INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "init_module errno 1"
lxc-start 100 20230425032311.623 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[175:init_module] action[327681:errno] arch[0]
lxc-start 100 20230425032311.623 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741827]
lxc-start 100 20230425032311.623 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741886]
lxc-start 100 20230425032311.623 INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "finit_module errno 1"
lxc-start 100 20230425032311.623 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[313:finit_module] action[327681:errno] arch[0]
lxc-start 100 20230425032311.623 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741827]
lxc-start 100 20230425032311.623 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741886]
lxc-start 100 20230425032311.623 INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "delete_module errno 1"
lxc-start 100 20230425032311.623 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[176:delete_module] action[327681:errno] arch[0]
lxc-start 100 20230425032311.624 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741827]
lxc-start 100 20230425032311.624 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741886]
lxc-start 100 20230425032311.624 INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "ioctl errno 1 [1,0x9400,SCMP_CMP_MASKED_EQ,0xff00]"
lxc-start 100 20230425032311.624 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:547 - arg_cmp[0]: SCMP_CMP(1, 7, 65280, 37888)
lxc-start 100 20230425032311.624 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[16:ioctl] action[327681:errno] arch[0]
lxc-start 100 20230425032311.624 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:547 - arg_cmp[0]: SCMP_CMP(1, 7, 65280, 37888)
lxc-start 100 20230425032311.624 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[16:ioctl] action[327681:errno] arch[1073741827]
lxc-start 100 20230425032311.624 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:547 - arg_cmp[0]: SCMP_CMP(1, 7, 65280, 37888)
lxc-start 100 20230425032311.624 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[16:ioctl] action[327681:errno] arch[1073741886]
lxc-start 100 20230425032311.624 INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "keyctl errno 38"
lxc-start 100 20230425032311.624 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[250:keyctl] action[327718:errno] arch[0]
lxc-start 100 20230425032311.624 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[250:keyctl] action[327718:errno] arch[1073741827]
lxc-start 100 20230425032311.624 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[250:keyctl] action[327718:errno] arch[1073741886]
lxc-start 100 20230425032311.624 INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:1017 - Merging compat seccomp contexts into main context
lxc-start 100 20230425032311.917 ERROR    apparmor - ../src/lxc/lsm/apparmor.c:run_apparmor_parser:916 - Failed to run apparmor_parser on "/var/lib/lxc/100/apparmor/lxc-100_<-var-lib-lxc>": apparmor_parser: Unable to replace "lxc-100_</var/lib/lxc>".  Profile doesn't conform to protocol
lxc-start 100 20230425032311.917 ERROR    apparmor - ../src/lxc/lsm/apparmor.c:apparmor_prepare:1088 - Failed to load generated AppArmor profile
lxc-start 100 20230425032311.917 ERROR    start - ../src/lxc/start.c:lxc_init:876 - Failed to initialize LSM
lxc-start 100 20230425032311.917 ERROR    start - ../src/lxc/start.c:__lxc_start:2027 - Failed to initialize container "100"
lxc-start 100 20230425032311.917 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_payload_destroy:555 - Uninitialized limit cgroup
lxc-start 100 20230425032311.917 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_monitor_destroy:881 - Uninitialized monitor cgroup
lxc-start 100 20230425032311.917 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "100", config section "lxc"
lxc-start 100 20230425032312.484 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "100", config section "lxc"
lxc-start 100 20230425032312.988 ERROR    lxccontainer - ../src/lxc/lxccontainer.c:wait_on_daemonized_start:870 - No such file or directory - Failed to receive the container state
lxc-start 100 20230425032312.988 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:306 - The container failed to start
lxc-start 100 20230425032312.988 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:309 - To get more details, run the container in foreground mode
lxc-start 100 20230425032312.988 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:311 - Additional information can be obtained by setting the --logfile and --logpriority options

I am also having trouble uploading uploading filed to local storage similar to those described HERE. I am observing that no mater the path on my local system the webUI is always loading C:\fakepath\. I have tried this with multiple web browsers and my other cluster node is not having the same problem.

Proxmox Fakepath.JPG

Thanks for any help you can provide.
 
Last edited:
Hi,

first of, the "kvm: disabled by bios" message means the hypervisor-support is disabled in the BIOS/UEFI of the machine. Normally, there is a setting to either enable VT-x (on Intel processors) or AMD-V (on AMD processors). These settings must be enabled for virtualization (i.e. VMs) to work.

Edit: Also, please update your kernel (you are currently running 5.10.0-21-amd64, which is long EOL. Also, when did you last reboot? As the current installed kernel is at least a bit newer (pve-kernel-5.15), but also EOL. Please upgrade to at least pve-kernel-5.19 or even pve-kernel-6.2.

And going by that, what does uname -a provide?

Secondly, AppArmor fails to start - are there any more logs/error message present in the syslog (journalctl -b). To only show messages from AppArmor, you can use journalctl -b -u apparmor. Please provide that log, so we can debug this furher.

Edit: Also, what does cat /var/lib/lxc/108/apparmor/lxc-108_\<-var-lib-lxc\> put out? The AppArmor profile for the above mentioned container doesn't seem to be parsable by AppArmor.

Also, notice the
Code: 
root@thinnode:/etc/pve/lxc# pct config 100
bash: pct: command not found
This should not happen, `pct` should be on the PATH. Did you correctly install and set up everything according to guide?
Can you provide the output of echo $PATH and ls -lah /usr/sbin/pct?

Secondly, when uploading images, it will always show C:\fakepath\ as prefix, which is intentional behavior. This is hardcoded behavior, due to JavaScript sandboxing/security concerns. As long as the actual upload works (meaning, it uploads successfully and afterwards shows up in the list), you don't have to worry about that.
 
Last edited:
Thanks Christoph,

Let me go through this one by one.

  • My computer has Secure Boot enabled with AMD-V disabled in the bios. I have not been able to unlock it so I am aware virtualization will not work. I am able to boot into Debian 11 because it has a signed kernel. I CAN NOT boot into pve kernel. Given that, I have updated to 6.1.0-0.deb11.5-amd64 from backport repo. My main goal for this node is to provide cluster quorum, be a Ceph node, and hopefully enable migration and High Availability for LXCs/CTs.
Code: 
root@thinnode:/home/ericc# pveversion -v
proxmox-ve: 7.4-1 (running kernel: 6.1.0-0.deb11.5-amd64)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-6.2: 7.4-1
pve-kernel-5.15: 7.4-1
pve-kernel-5.19: 7.2-15
pve-kernel-6.2.9-1-pve: 6.2.9-1
pve-kernel-5.19.17-2-pve: 5.19.17-2
pve-kernel-5.15.104-1-pve: 5.15.104-2
ceph: 17.2.5-pve1
ceph-fuse: 17.2.5-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-4
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.5
libpve-storage-perl: 7.4-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.1-1
proxmox-backup-file-restore: 2.4.1-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-widget-toolkit: 3.6.5
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-2
pve-firewall: 4.3-1
pve-firmware: 3.6-4
pve-ha-manager: 3.6.0
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1
  • I provided uname -a because that was requested in the older tickets I referenced.
  • Apparmor is not running and /var/lib/lxc/108/apparmor/lxc-108_\<-var-lib-lxc\> does not exist.
Code: 
-- Journal begins at Mon 2023-04-24 22:34:31 EDT, ends at Tue 2023-04-25 18:35:35 EDT. --
Apr 25 18:26:51 thinnode systemd[1]: Starting Load AppArmor profiles...
Apr 25 18:26:51 thinnode apparmor.systemd[650]: Restarting AppArmor
Apr 25 18:26:51 thinnode apparmor.systemd[650]: Reloading AppArmor profiles
Apr 25 18:26:51 thinnode apparmor.systemd[678]: /sbin/apparmor_parser: Unable to replace "/usr/bin/lxc-start".  Profile doesn't conform to protocol
Apr 25 18:26:51 thinnode apparmor.systemd[680]: /sbin/apparmor_parser: Unable to replace "swtpm".  Profile doesn't conform to protocol
Apr 25 18:26:51 thinnode apparmor.systemd[676]: /sbin/apparmor_parser: Unable to replace "lsb_release".  Profile doesn't conform to protocol
Apr 25 18:26:51 thinnode apparmor.systemd[681]: /sbin/apparmor_parser: Unable to replace "kmod".  Profile doesn't conform to protocol
Apr 25 18:26:51 thinnode apparmor.systemd[681]: /sbin/apparmor_parser: Unable to replace "nvidia_modprobe".  Profile doesn't conform to protocol
Apr 25 18:26:51 thinnode apparmor.systemd[677]: /sbin/apparmor_parser: Unable to replace "/usr/bin/man".  Profile doesn't conform to protocol
Apr 25 18:26:51 thinnode apparmor.systemd[679]: /sbin/apparmor_parser: Unable to replace "lxc-container-default".  Profile doesn't conform to protocol
Apr 25 18:26:51 thinnode apparmor.systemd[728]: /sbin/apparmor_parser: Unable to replace "/usr/bin/lxc-start".  Profile doesn't conform to protocol
Apr 25 18:26:51 thinnode apparmor.systemd[714]: /sbin/apparmor_parser: Unable to replace "lsb_release".  Profile doesn't conform to protocol
Apr 25 18:26:51 thinnode apparmor.systemd[734]: /sbin/apparmor_parser: Unable to replace "swtpm".  Profile doesn't conform to protocol
Apr 25 18:26:51 thinnode apparmor.systemd[722]: /sbin/apparmor_parser: Unable to replace "kmod".  Profile doesn't conform to protocol
Apr 25 18:26:51 thinnode apparmor.systemd[722]: /sbin/apparmor_parser: Unable to replace "nvidia_modprobe".  Profile doesn't conform to protocol
Apr 25 18:26:51 thinnode apparmor.systemd[732]: /sbin/apparmor_parser: Unable to replace "/usr/bin/man".  Profile doesn't conform to protocol
Apr 25 18:26:51 thinnode apparmor.systemd[719]: /sbin/apparmor_parser: Unable to replace "lxc-container-default".  Profile doesn't conform to protocol
Apr 25 18:26:51 thinnode apparmor.systemd[650]: Error: At least one profile failed to load
Apr 25 18:26:51 thinnode systemd[1]: apparmor.service: Main process exited, code=exited, status=1/FAILURE
Apr 25 18:26:51 thinnode systemd[1]: apparmor.service: Failed with result 'exit-code'.
Apr 25 18:26:51 thinnode systemd[1]: Failed to start Load AppArmor profiles.
Apr 25 18:26:51 thinnode systemd[1]: apparmor.service: Consumed 1.192s CPU time.
root@thinnode:/home/ericc# cat /var/lib/lxc/108/apparmor/lxc-108_\<-var-lib-lxc\>
cat: '/var/lib/lxc/108/apparmor/lxc-108_<-var-lib-lxc>': No such file or directory
  • bash: pct: command not found happened because I was using putty to ssh in at the time. When using the web terminal, the command works.
pct.JPG
  • Thanks for the clarification. I have been able to use the web ui to upload .iso files.
 
Problem seems to be applying to all containers, not just LXC. I was unable to trigger the Hello World docker container after going through the instructions HERE.

Code: 
root@thinnode:/home/ericc# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
2db29710123e: Pull complete
Digest: sha256:4e83453afed1b4fa1a3500525091dbfca6ce1e66903fd4c01ff015dbcb1ba33e
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: AppArmor enabled on system but the docker-default profile could not be loaded: running `/usr/sbin/apparmor_parser apparmor_parser -Kr /var/lib/docker/tmp/docker-default1841135079` failed with output: apparmor_parser: Unable to replace "docker-default".  Profile doesn't conform to protocol

error: exit status 185.
ERRO[0002] error waiting for container:
 
TGT_9000 said:
My computer has Secure Boot enabled with AMD-V disabled in the bios. I have not been able to unlock it so I am aware virtualization will not work. I am able to boot into Debian 11 because it has a signed kernel. I CAN NOT boot into pve kernel. Given that, I have updated to 6.1.0-0.deb11.5-amd64 from backport repo.
Click to expand...
That explains that. Less than ideal, but you already know that it seems and are aware of the consequences.
We don't have any patches in the pve-kernel that concern LXC/NS/containers AFAIK, so it should work.

TGT_9000 said:
bash: pct: command not found happened because I was using putty to ssh in at the time. When using the web terminal, the command works.
Click to expand...
Okay, so that's not broken. Just caught my eyes while reading.

TGT_9000 said:
Problem seems to be applying to all containers, not just LXC. I was unable to trigger the Hello World docker container after going through the instructions HERE.
Click to expand...
Just FYI, running Docker either directly on the Proxmox VE or inside LXC containers is strongly recommended against, as it only leads to more problems than it really solves.

And thanks for providing the full system log. The root cause really seems that AppArmor chokes on some configuration.

What is the content of /etc/apparmor.d/lxc-containers and all files under /etc/apparmor.d/lxc/?
 
Loaded Docker to see if I could trigger Apparmor failure with another containerization program.

Code: 
root@thinnode:~# cat /etc/apparmor.d/lxc-containers
# This file exists only to ensure that all per-container policies
# listed under /etc/apparmor.d/lxc get loaded at boot.  Please do
# not edit this file.

#include <tunables/global>

#include <lxc>
root@thinnode:~# cd /etc/apparmor.d/lxc/
root@thinnode:/etc/apparmor.d/lxc# ls
lxc-default  lxc-default-cgns  lxc-default-with-mounting  lxc-default-with-nesting
root@thinnode:/etc/apparmor.d/lxc# cat *
# Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
# will source all profiles under /etc/apparmor.d/lxc

profile lxc-container-default flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/lxc/container-base>

  # the container may never be allowed to mount devpts.  If it does, it
  # will remount the host's devpts.  We could allow it to do it with
  # the newinstance option (but, right now, we don't).
  deny mount fstype=devpts,
}
# Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
# will source all profiles under /etc/apparmor.d/lxc

profile lxc-container-default-cgns flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/lxc/container-base>

  # the container may never be allowed to mount devpts.  If it does, it
  # will remount the host's devpts.  We could allow it to do it with
  # the newinstance option (but, right now, we don't).
  deny mount fstype=devpts,
  mount fstype=cgroup -> /sys/fs/cgroup/**,
  mount fstype=cgroup2 -> /sys/fs/cgroup/**,
  mount fstype=overlay,
}
# Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
# will source all profiles under /etc/apparmor.d/lxc

profile lxc-container-default-with-mounting flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/lxc/container-base>

# allow standard blockdevtypes.
# The concern here is in-kernel superblock parsers bringing down the
# host with bad data.  However, we continue to disallow proc, sys, securityfs,
# etc to nonstandard locations.
  mount fstype=ext*,
  mount fstype=xfs,
  mount fstype=btrfs,
}
# Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
# will source all profiles under /etc/apparmor.d/lxc

profile lxc-container-default-with-nesting flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/lxc/container-base>
  #include <abstractions/lxc/start-container>

  deny /dev/.lxc/proc/** rw,
  deny /dev/.lxc/sys/** rw,
  mount fstype=proc -> /var/cache/lxc/**,
  mount fstype=sysfs -> /var/cache/lxc/**,
  mount options=(rw,bind),
  mount fstype=cgroup -> /sys/fs/cgroup/**,
  mount fstype=cgroup2 -> /sys/fs/cgroup/**,
}

Code: 
root@thinnode:/etc/apparmor.d# cat *
cat: abstractions: Is a directory
cat: disable: Is a directory
cat: force-complain: Is a directory
cat: local: Is a directory
# Note: This profile does not specify an attachment path because it is
# intended to be used only via "Px -> lsb_release" exec transitions from
# other profiles. We want to confine the lsb_release(1) utility when it
# is invoked from other confined applications, but not when it is used
# in regular (unconfined) shell scripts or run directly by the user.

#include <tunables/global>

# Do not attach to /usr/bin/lsb_release by default
profile lsb_release {
  #include <abstractions/base>
  #include <abstractions/python>

  owner @{PROC}/@{pid}/fd/ r,

  /dev/tty rw,

  /usr/bin/lsb_release r,
  /usr/bin/python3.[0-9] mr,

  /etc/debian_version r,
  /etc/default/apport r,
  /etc/dpkg/origins/** r,
  /etc/lsb-release r,
  /etc/lsb-release.d/ r,

  /{usr/,}bin/bash ixr,
  /{usr/,}bin/dash ixr,
  /usr/bin/basename ixr,
  /usr/bin/dpkg-query ixr,
  /usr/bin/getopt ixr,
  /usr/bin/sed ixr,
  /usr/bin/tr ixr,

  # TODO - many more permissions needed for this to work
  deny /usr/bin/apt-cache x,

  /usr/bin/ r,
  /usr/include/python*/pyconfig.h r,
  /usr/share/distro-info/** r,
  /usr/share/dpkg/** r,
  /usr/share/terminfo/** r,
  /var/lib/dpkg/** r,

  # file_inherit
  deny /tmp/gtalkplugin.log w,

  # Site-specific additions and overrides. See local/README for details.
  #include <local/lsb_release>
}
cat: lxc: Is a directory
# This file exists only to ensure that all per-container policies
# listed under /etc/apparmor.d/lxc get loaded at boot.  Please do
# not edit this file.

#include <tunables/global>

#include <lxc>
# vim:syntax=apparmor

#include <tunables/global>

profile nvidia_modprobe {
  #include <abstractions/base>

  # Capabilities

  capability chown,
  capability mknod,
  capability setuid,
  capability sys_admin,

  # Main executable

  /usr/bin/nvidia-modprobe mr,

  # Other executables

  /usr/bin/kmod Cx -> kmod,

  # System files

  /dev/nvidia-modeset w,
  /dev/nvidia-uvm w,
  /dev/nvidia-uvm-tools w,
  @{sys}/bus/pci/devices/ r,
  @{sys}/devices/pci[0-9]*/**/config r,
  @{PROC}/devices r,
  @{PROC}/driver/nvidia/params r,
  @{PROC}/modules r,
  @{PROC}/sys/kernel/modprobe r,

  # Child profiles

  profile kmod {
    #include <abstractions/base>

    # Capabilities

    capability sys_module,

    # Main executable

    /usr/bin/kmod mrix,

    # Other executables

    /{,usr/}bin/{,ba,da}sh ix,

    # System files

    /etc/modprobe.d/{,*.conf} r,
    /etc/nvidia/current/*.conf r,
    @{sys}/module/ipmi_devintf/initstate r,
    @{sys}/module/ipmi_msghandler/initstate r,
    @{sys}/module/nvidia/initstate r,
    @{PROC}/cmdline r,
  }

  # Site-specific additions and overrides. See local/README for details.
  #include <local/nvidia_modprobe>
}

cat: tunables: Is a directory
#include <tunables/global>

/usr/bin/lxc-start flags=(attach_disconnected) {
  #include <abstractions/lxc/start-container>
}
# vim:syntax=apparmor

#include <tunables/global>

/usr/bin/man {
  #include <abstractions/base>

  # Use a special profile when man calls anything groff-related.  We only
  # include the programs that actually parse input data in a non-trivial
  # way, not wrappers such as groff and nroff, since the latter would need a
  # broader profile.
  /usr/bin/eqn rmCx -> &man_groff,
  /usr/bin/grap rmCx -> &man_groff,
  /usr/bin/pic rmCx -> &man_groff,
  /usr/bin/preconv rmCx -> &man_groff,
  /usr/bin/refer rmCx -> &man_groff,
  /usr/bin/tbl rmCx -> &man_groff,
  /usr/bin/troff rmCx -> &man_groff,
  /usr/bin/vgrind rmCx -> &man_groff,

  # Similarly, use a special profile when man calls decompressors and other
  # simple filters.
  /{,usr/}bin/bzip2 rmCx -> &man_filter,
  /{,usr/}bin/gzip rmCx -> &man_filter,
  /usr/bin/col rmCx -> &man_filter,
  /usr/bin/compress rmCx -> &man_filter,
  /usr/bin/iconv rmCx -> &man_filter,
  /usr/bin/lzip.lzip rmCx -> &man_filter,
  /usr/bin/tr rmCx -> &man_filter,
  /usr/bin/xz rmCx -> &man_filter,

  # Allow basically anything in terms of file system access, subject to DAC.
  # The purpose of this profile isn't to confine man itself (that might be
  # nice in the future, but is tricky since it's quite configurable), but to
  # confine the processes it calls that parse untrusted data.
  /** mrixwlk,
  unix,

  capability setuid,
  capability setgid,

  # Ordinary permission checks sometimes involve checking whether the
  # process has this capability, which can produce audit log messages.
  # Silence them.
  deny capability dac_override,
  deny capability dac_read_search,

  signal peer=@{profile_name},
  signal peer=/usr/bin/man//&man_groff,
  signal peer=/usr/bin/man//&man_filter,

  # Site-specific additions and overrides.  See local/README for details.
  #include <local/usr.bin.man>
}

profile man_groff {
  #include <abstractions/base>
  # Recent kernels revalidate open FDs, and there are often some still
  # open on TTYs.  This is temporary until man learns to close irrelevant
  # open FDs before execve.
  #include <abstractions/consoles>
  # man always runs its groff pipeline with the input file open on stdin,
  # so we can skip <abstractions/user-manpages>.

  /usr/bin/eqn rm,
  /usr/bin/grap rm,
  /usr/bin/pic rm,
  /usr/bin/preconv rm,
  /usr/bin/refer rm,
  /usr/bin/tbl rm,
  /usr/bin/troff rm,
  /usr/bin/vgrind rm,

  /etc/groff/** r,
  /etc/papersize r,
  /usr/lib/groff/site-tmac/** r,
  /usr/share/groff/** r,

  /tmp/groff* rw,

  signal peer=/usr/bin/man,
  # @{profile_name} doesn't seem to work here.
  signal peer=/usr/bin/man//&man_groff,
}

profile man_filter {
  #include <abstractions/base>
  # Recent kernels revalidate open FDs, and there are often some still
  # open on TTYs.  This is temporary until man learns to close irrelevant
  # open FDs before execve.
  #include <abstractions/consoles>

  /{,usr/}bin/bzip2 rm,
  /{,usr/}bin/gzip rm,
  /usr/bin/col rm,
  /usr/bin/compress rm,
  /usr/bin/iconv rm,
  /usr/bin/lzip.lzip rm,
  /usr/bin/tr rm,
  /usr/bin/xz rm,

  # Manual pages can be more or less anywhere, especially with "man -l", and
  # there's no harm in allowing wide read access here since the worst it can
  # do is feed data to the invoking man process.
  /** r,

  # Allow writing cat pages.
  /var/cache/man/** w,

  signal peer=/usr/bin/man,
  # @{profile_name} doesn't seem to work here.
  signal peer=/usr/bin/man//&man_filter,
}
# vim:syntax=apparmor
# AppArmor policy for swtpm

#include <tunables/global>

profile swtpm /usr/bin/swtpm {
  #include <abstractions/base>
  #include <abstractions/openssl>

  # Site-specific additions and overrides. See local/README for details.
  #include <local/usr.bin.swtpm>

  capability chown,
  capability dac_override,
  capability dac_read_search,
  capability fowner,
  capability fsetid,
  capability setgid,
  capability setuid,

  network inet stream,
  network inet6 stream,
  unix (send) type=dgram addr=none peer=(addr=none),
  unix (send, receive) type=stream addr=none peer=(label=libvirt-*),

  /usr/bin/swtpm rm,

  # Proxmox VE allow to save states on many possible locations, so allow everything for now.
  /** rwk,
}
 
Can you please additionally provide the output of apt-cache policy apparmor apparmor-profiles apparmor-profiles-extra apparmor-utils?

The profiles seem valid from a quick glance over them, so I'm guessing there's something off with the AppArmor package versions.

But again, as this is essentially an extremly weird FrankenDebian and totally unsupported configuration, don't expect this to necessarily work in the end.
 
Thanks cheiss, anything you can do is appreciated.

Code: 
root@ThinNode:~# apt-cache policy apparmor apparmor-profiles apparmor-profiles-extra apparmor-utils
apparmor:
  Installed: 2.13.6-10
  Candidate: 2.13.6-10
  Version table:
 *** 2.13.6-10 500
        500 http://deb.debian.org/debian bullseye/main amd64 Packages
        100 /var/lib/dpkg/status
apparmor-profiles:
  Installed: (none)
  Candidate: 2.13.6-10
  Version table:
     2.13.6-10 500
        500 http://deb.debian.org/debian bullseye/main amd64 Packages
apparmor-profiles-extra:
  Installed: (none)
  Candidate: 1.34
  Version table:
     1.34 500
        500 http://deb.debian.org/debian bullseye/main amd64 Packages
apparmor-utils:
  Installed: (none)
  Candidate: 2.13.6-10
  Version table:
     2.13.6-10 500
        500 http://deb.debian.org/debian bullseye/main amd64 Packages
 
I just ran a test on another computer with unlocked bootloader. I installed via Debian 11 route as before and then switched back and forth between 6.2-pve and 6.1 debian kernels. I only encountered AppArmor issues with the debian kernel. When loading the PVE kernel everything works as expected.
 
Thanks for confirming it. Seems do be indeed in issue with the stock Debian kernel. The Proxmox VE kernel is based on the Ubuntu one, with some extra patches on top. I guess these are some differences for AppArmor between Debian <-> Ubuntu.

Thus you could try a stock Ubuntu kernel, given that they are signed properly for Secure Boot too, I presume. Although how one could install them on top of Debian is another thing (I'd start by downloading the deb(s) and installing them by hand probably).
 
cheiss said:
Can you please additionally provide the output of apt-cache policy apparmor apparmor-profiles apparmor-profiles-extra apparmor-utils?

The profiles seem valid from a quick glance over them, so I'm guessing there's something off with the AppArmor package versions.

But again, as this is essentially an extremly weird FrankenDebian and totally unsupported configuration, don't expect this to necessarily work in the end.
Click to expand...
I have this same issue in my RPI4B after installing PiMox.
Is it possible that I can get some help to get apparmor to run? The failure is caused by bits that got installed by pimox.


root@hass:/home/pi# systemctl --state=failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● apparmor.service loaded failed failed Load AppArmor profiles

LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
1 loaded units listed.
root@hass:/home/pi# systemctl status apparmor.service
● apparmor.service - Load AppArmor profiles
Loaded: loaded (/lib/systemd/system/apparmor.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2023-06-06 20:30:24 EAT; 3min 6s ago
Docs: man:apparmor(7)
https://gitlab.com/apparmor/apparmor/wikis/home/
Process: 386 ExecStart=/lib/apparmor/apparmor.systemd reload (code=exited, status=1/FAILURE)
Main PID: 386 (code=exited, status=1/FAILURE)
CPU: 16.235s

Jun 06 20:30:23 hass apparmor.systemd[551]: /sbin/apparmor_parser: Unable to replace "/usr/sbin/cups-browsed". Profile does not conform to protocol
Jun 06 20:30:23 hass apparmor.systemd[510]: /sbin/apparmor_parser: Unable to replace "chromium". Profile does not conform to protocol
Jun 06 20:30:23 hass apparmor.systemd[510]: /sbin/apparmor_parser: Unable to replace "/usr/lib/aarch64-linux-gnu/lightdm/lightdm-guest-session". Profile does not conform to protocol
Jun 06 20:30:23 hass apparmor.systemd[522]: /sbin/apparmor_parser: Unable to replace "lxc-container-default". Profile does not conform to protocol
Jun 06 20:30:24 hass apparmor.systemd[553]: /sbin/apparmor_parser: Unable to replace "/usr/lib/cups/backend/cups-pdf". Profile does not conform to protocol
Jun 06 20:30:24 hass apparmor.systemd[386]: Error: At least one profile failed to load
Jun 06 20:30:24 hass systemd[1]: apparmor.service: Main process exited, code=exited, status=1/FAILURE
Jun 06 20:30:24 hass systemd[1]: apparmor.service: Failed with result 'exit-code'.
Jun 06 20:30:24 hass systemd[1]: Failed to start Load AppArmor profiles.
Jun 06 20:30:24 hass systemd[1]: apparmor.service: Consumed 16.235s CPU time.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom