After upgrade CT won't start

K

kenren98

New Member
Nov 24, 2021
7
0
1
27
Code: 
INFO     cgfsng - ../src/lxc/cgroups/cgfsng.c:unpriv_systemd_create_scope:1227 - Running privileged, not using a systemd unit
DEBUG    seccomp - ../src/lxc/seccomp.c:parse_config_v2:656 - Host native arch is [3221225534]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "[all]"
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "kexec_load errno 1"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[246:kexec_load] action[327681:errno] arch[0]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741827]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741886]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "open_by_handle_at errno 1"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[304:open_by_handle_at] action[327681:errno] arch[0]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741827]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741886]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "init_module errno 1"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[175:init_module] action[327681:errno] arch[0]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741827]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741886]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "finit_module errno 1"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[313:finit_module] action[327681:errno] arch[0]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741827]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741886]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "delete_module errno 1"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[176:delete_module] action[327681:errno] arch[0]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741827]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741886]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "ioctl errno 1 [1,0x9400,SCMP_CMP_MASKED_EQ,0xff00]"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:547 - arg_cmp[0]: SCMP_CMP(1, 7, 65280, 37888)
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[16:ioctl] action[327681:errno] arch[0]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:547 - arg_cmp[0]: SCMP_CMP(1, 7, 65280, 37888)
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[16:ioctl] action[327681:errno] arch[1073741827]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:547 - arg_cmp[0]: SCMP_CMP(1, 7, 65280, 37888)
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[16:ioctl] action[327681:errno] arch[1073741886]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:1017 - Merging compat seccomp contexts into main context
ERROR    apparmor - ../src/lxc/lsm/apparmor.c:run_apparmor_parser:916 - Failed to run apparmor_parser on "/var/lib/lxc/200/apparmor/lxc-200_<-var-lib-lxc>": Failed setting up policy cache (/var/cache/lxc/apparmor): Input/output error
ERROR    apparmor - ../src/lxc/lsm/apparmor.c:apparmor_prepare:1088 - Failed to load generated AppArmor profile
ERROR    start - ../src/lxc/start.c:lxc_init:876 - Failed to initialize LSM
ERROR    start - ../src/lxc/start.c:__lxc_start:2027 - Failed to initialize container "200"
WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_payload_destroy:555 - Uninitialized limit cgroup
WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_monitor_destroy:881 - Uninitialized monitor cgroup
INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "200", config section "lxc"


Debug info
No sure if its a packgae issue


root@node5:~# apparmor_parser --version
AppArmor parser version 2.13.6
Copyright (C) 1999-2008 Novell Inc.
Copyright 2009-2018 Canonical Ltd.


Config is checked, reinstall tried, all CT is not starting, this CT is a debian CT
installed linux packages

root@node5:~# apt list --installed | grep linux

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

binutils-x86-64-linux-gnu/stable,now 2.35.2-2 amd64 [installed,automatic]
console-setup-linux/stable,now 1.205 all [installed]
liblinux-inotify2-perl/stable,now 1:2.2-2+b1 amd64 [installed]
libnvpair3linux/stable,now 2.1.9-pve1 amd64 [installed]
libselinux1/stable,now 3.1-3 amd64 [installed]
libuutil3linux/stable,now 2.1.9-pve1 amd64 [installed]
libzfs4linux/stable,now 2.1.9-pve1 amd64 [installed]
libzpool5linux/stable,now 2.1.9-pve1 amd64 [installed]
linux-base/stable,now 4.6 all [installed]
util-linux/stable,stable-security,now 2.36.1-8+deb11u1 amd64 [installed]
zfsutils-linux/stable,now 2.1.9-pve1 amd64 [installed]
root@node5:~#
 
Hi,
please post the output of pveversion -v and pct config 200.
 
  • Like
Reactions: kenren98
Code: 
root@node5:~# pveversion -v
proxmox-ve: 7.4-1 (running kernel: 5.15.104-1-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.15: 7.4-1
pve-kernel-5.15.104-1-pve: 5.15.104-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
pve-kernel-5.15.85-1-pve: 5.15.85-1
pve-kernel-5.15.83-1-pve: 5.15.83-1
pve-kernel-5.15.64-1-pve: 5.15.64-1
ceph: 16.2.11-pve1
ceph-fuse: 16.2.11-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-4
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-1
libpve-rs-perl: 0.7.5
libpve-storage-perl: 7.4-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.1-1
proxmox-backup-file-restore: 2.4.1-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-widget-toolkit: 3.6.5
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-1
pve-firewall: 4.3-1
pve-firmware: 3.6-4
pve-ha-manager: 3.6.0
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1
root@node5:~#

temporarily moved to another node and tested working

root@node8:~# pct config 200
arch: amd64
cores: 2
features: nesting=1,keyctl=1
hostname: DNS-AdGuard
memory: 2048
net0: name=eth0,bridge=vmbr0,gw=192.168.1.1,hwaddr=36:35:FB:A7:C2:02,ip=192.168.3.200/22,type=veth
onboot: 1
ostype: debian
rootfs: Node8-V1:vm-200-disk-0,size=20G
swap: 1024
unprivileged: 1
root@node8:~#

Thank you so much!
 
  • Like
Reactions: kenren98
aa-status
Code: 
root@node5:~# aa-status
apparmor module is loaded.
14 profiles are loaded.
14 profiles are in enforce mode.
   /usr/bin/lxc-start
   /usr/bin/man
   /usr/sbin/chronyd
   lsb_release
   lxc-container-default
   lxc-container-default-cgns
   lxc-container-default-with-mounting
   lxc-container-default-with-nesting
   man_filter
   man_groff
   nvidia_modprobe
   nvidia_modprobe//kmod
   swtpm
   tcpdump
0 profiles are in complain mode.
2 processes have profiles defined.
2 processes are in enforce mode.
   /usr/sbin/chronyd (989)
   /usr/sbin/chronyd (995)
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.
root@node5:~#

systemctl
Code: 
● apparmor.service - Load AppArmor profiles
     Loaded: loaded (/lib/systemd/system/apparmor.service; enabled; vendor preset: enabled)
     Active: active (exited) since Mon 2023-04-03 11:39:31 EDT; 1 day 18h ago
       Docs: man:apparmor(7)
             https://gitlab.com/apparmor/apparmor/wikis/home/
    Process: 766 ExecStart=/lib/apparmor/apparmor.systemd reload (code=exited, status=0/SUCCESS)
    Process: 642257 ExecReload=/lib/apparmor/apparmor.systemd reload (code=exited, status=0/SUCCESS)
   Main PID: 766 (code=exited, status=0/SUCCESS)
        CPU: 28ms

Apr 03 11:39:31 node5 systemd[1]: Starting Load AppArmor profiles...
Apr 03 11:39:31 node5 apparmor.systemd[766]: Restarting AppArmor
Apr 03 11:39:31 node5 apparmor.systemd[766]: Reloading AppArmor profiles
Apr 03 11:39:31 node5 systemd[1]: Finished Load AppArmor profiles.
Apr 05 06:02:51 node5 systemd[1]: Reloading Load AppArmor profiles.
Apr 05 06:02:51 node5 apparmor.systemd[642257]: Restarting AppArmor
Apr 05 06:02:51 node5 apparmor.systemd[642257]: Reloading AppArmor profiles
Apr 05 06:02:51 node5 systemd[1]: Reloaded Load AppArmor profiles.


I dont think there is any difference between them cuz I usually update them all together. And both of them are Intel CPUS, 8700K for node5, E5-2680V3 for node8. And I am pretty sure it works fine before the update, but after the update, I tried to reinstall. It would not boot even for a new CT, I would assume it is some config problems. The new kernel is updated in two days ago, after the

1680690341222.png
1680690351767.png

I just made cloned the working CT200 to CT777 and same error occurs
1680690579239.png
 
fiona said:
Anything obviously different between the nodes?

I found another thread with a similar issue. Following that thread, can you post the output of aa-status? If it's not mounted, you can try systemctl reload-or-restart apparmor.service.
Click to expand...
Also about the other one with similar issue, I saw that one too, but it seems reboot didnt solve my problem.

journalctl -b -u lxcfs; systemctl status lxcfs
Code: 
root@node5:~# journalctl -b -u lxcfs; systemctl status lxcfs
-- Journal begins at Tue 2022-10-11 16:26:17 EDT, ends at Wed 2023-04-05 06:30:45 E>
Apr 05 06:22:23 node5 systemd[1]: Started FUSE filesystem for LXC.
Apr 05 06:22:23 node5 lxcfs[793]: Running constructor lxcfs_init to reload liblxcfs
Apr 05 06:22:23 node5 lxcfs[793]: mount namespace: 5
Apr 05 06:22:23 node5 lxcfs[793]: hierarchies:
Apr 05 06:22:23 node5 lxcfs[793]:   0: fd:   6: cpuset,cpu,io,memory,hugetlb,pids,r>
Apr 05 06:22:23 node5 lxcfs[793]: Kernel supports pidfds
Apr 05 06:22:23 node5 lxcfs[793]: Kernel supports swap accounting
Apr 05 06:22:23 node5 lxcfs[793]: api_extensions:
Apr 05 06:22:23 node5 lxcfs[793]: - cgroups
Apr 05 06:22:23 node5 lxcfs[793]: - sys_cpu_online
Apr 05 06:22:23 node5 lxcfs[793]: - proc_cpuinfo
Apr 05 06:22:23 node5 lxcfs[793]: - proc_diskstats
Apr 05 06:22:23 node5 lxcfs[793]: - proc_loadavg
Apr 05 06:22:23 node5 lxcfs[793]: - proc_meminfo
Apr 05 06:22:23 node5 lxcfs[793]: - proc_stat
Apr 05 06:22:23 node5 lxcfs[793]: - proc_swaps
Apr 05 06:22:23 node5 lxcfs[793]: - proc_uptime
Apr 05 06:22:23 node5 lxcfs[793]: - proc_slabinfo
Apr 05 06:22:23 node5 lxcfs[793]: - shared_pidns
Apr 05 06:22:23 node5 lxcfs[793]: - cpuview_daemon
Apr 05 06:22:23 node5 lxcfs[793]: - loadavg_daemon
Apr 05 06:22:23 node5 lxcfs[793]: - pidfds
● lxcfs.service - FUSE filesystem for LXC
     Loaded: loaded (/lib/systemd/system/lxcfs.service; enabled; vendor preset: ena>
     Active: active (running) since Wed 2023-04-05 06:22:23 EDT; 8min ago
       Docs: man:lxcfs(1)
   Main PID: 793 (lxcfs)
      Tasks: 3 (limit: 28574)
     Memory: 940.0K
        CPU: 1ms
     CGroup: /system.slice/lxcfs.service
             └─793 /usr/bin/lxcfs /var/lib/lxcfs

Apr 05 06:22:23 node5 lxcfs[793]: - proc_loadavg
Apr 05 06:22:23 node5 lxcfs[793]: - proc_meminfo
Apr 05 06:22:23 node5 lxcfs[793]: - proc_stat
Apr 05 06:22:23 node5 lxcfs[793]: - proc_swaps
Apr 05 06:22:23 node5 lxcfs[793]: - proc_uptime
Apr 05 06:22:23 node5 lxcfs[793]: - proc_slabinfo
Apr 05 06:22:23 node5 lxcfs[793]: - shared_pidns
Apr 05 06:22:23 node5 lxcfs[793]: - cpuview_daemon
Apr 05 06:22:23 node5 lxcfs[793]: - loadavg_daemon
Apr 05 06:22:23 node5 lxcfs[793]: - pidfds


Node5 journalctl -b | grep -i -e apparmor -e aa_ | head -n 15
Code: 
root@node5:~# journalctl -b | grep -i -e apparmor -e aa_ | head -n 15
Apr 05 06:22:22 node5 kernel: AppArmor: AppArmor initialized
Apr 05 06:22:22 node5 kernel: AppArmor: AppArmor Filesystem Enabled
Apr 05 06:22:22 node5 kernel: AppArmor: AppArmor sha1 policy hashing enabled
Apr 05 06:22:22 node5 kernel: evm: security.apparmor
Apr 05 06:22:22 node5 systemd[1]: systemd 247.3-7+1-pmx11u1 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified)
Apr 05 06:22:23 node5 systemd[1]: Starting Load AppArmor profiles...
Apr 05 06:22:23 node5 apparmor.systemd[752]: Restarting AppArmor
Apr 05 06:22:23 node5 apparmor.systemd[752]: Reloading AppArmor profiles
Apr 05 06:22:23 node5 audit[785]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="swtpm" pid=785 comm="apparmor_parser"
Apr 05 06:22:23 node5 kernel: audit: type=1400 audit(1680690143.278:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="swtpm" pid=785 comm="apparmor_parser"
Apr 05 06:22:23 node5 kernel: audit: type=1400 audit(1680690143.278:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=781 comm="apparmor_parser"
Apr 05 06:22:23 node5 audit[781]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=781 comm="apparmor_parser"
Apr 05 06:22:23 node5 audit[778]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=778 comm="apparmor_parser"
Apr 05 06:22:23 node5 audit[784]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=784 comm="apparmor_parser"
Apr 05 06:22:23 node5 audit[784]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=784 comm="apparmor_parser"


Node8 journalctl -b | grep -i -e apparmor -e aa_ | head -n 15
Code: 
root@node8:~# journalctl -b | grep -i -e apparmor -e aa_ | head -n 15
Apr 05 06:24:30 node8 kernel: AppArmor: AppArmor initialized
Apr 05 06:24:30 node8 kernel: AppArmor: AppArmor Filesystem Enabled
Apr 05 06:24:30 node8 kernel: AppArmor: AppArmor sha1 policy hashing enabled
Apr 05 06:24:30 node8 kernel: evm: security.apparmor
Apr 05 06:24:30 node8 systemd[1]: systemd 247.3-7+1-pmx11u1 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified)
Apr 05 06:24:31 node8 systemd[1]: Starting Load AppArmor profiles...
Apr 05 06:24:31 node8 apparmor.systemd[963]: Restarting AppArmor
Apr 05 06:24:31 node8 apparmor.systemd[963]: Reloading AppArmor profiles
Apr 05 06:24:31 node8 audit[991]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="swtpm" pid=991 comm="apparmor_parser"
Apr 05 06:24:31 node8 audit[993]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=993 comm="apparmor_parser"
Apr 05 06:24:31 node8 kernel: audit: type=1400 audit(1680690271.314:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="swtpm" pid=991 comm="apparmor_parser"
Apr 05 06:24:31 node8 kernel: audit: type=1400 audit(1680690271.314:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=993 comm="apparmor_parser"
Apr 05 06:24:31 node8 audit[990]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=990 comm="apparmor_parser"
Apr 05 06:24:31 node8 audit[996]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=996 comm="apparmor_parser"
Apr 05 06:24:31 node8 audit[996]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=996 comm="apparmor_parser"


Thank you again for helping!
 
Is there anything interesting in /var/log/syslog? If the issue only appears on the specific node, you could try to e.g. reinstall lxc-related packages, check if there's some configuration difference to other nodes, check the filesystem and disk that /var/cache/lxc/apparmor belongs to and the permissions for that path.
 
You must log in or register to reply here.
Top Bottom
Back