After upgrade CT won't start

K

kenren98

New Member
Nov 24, 2021
7
0
1
26
Code: 
INFO     cgfsng - ../src/lxc/cgroups/cgfsng.c:unpriv_systemd_create_scope:1227 - Running privileged, not using a systemd unit
DEBUG    seccomp - ../src/lxc/seccomp.c:parse_config_v2:656 - Host native arch is [3221225534]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "[all]"
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "kexec_load errno 1"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[246:kexec_load] action[327681:errno] arch[0]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741827]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741886]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "open_by_handle_at errno 1"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[304:open_by_handle_at] action[327681:errno] arch[0]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741827]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741886]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "init_module errno 1"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[175:init_module] action[327681:errno] arch[0]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741827]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741886]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "finit_module errno 1"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[313:finit_module] action[327681:errno] arch[0]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741827]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741886]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "delete_module errno 1"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[176:delete_module] action[327681:errno] arch[0]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741827]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741886]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "ioctl errno 1 [1,0x9400,SCMP_CMP_MASKED_EQ,0xff00]"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:547 - arg_cmp[0]: SCMP_CMP(1, 7, 65280, 37888)
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[16:ioctl] action[327681:errno] arch[0]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:547 - arg_cmp[0]: SCMP_CMP(1, 7, 65280, 37888)
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[16:ioctl] action[327681:errno] arch[1073741827]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:547 - arg_cmp[0]: SCMP_CMP(1, 7, 65280, 37888)
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[16:ioctl] action[327681:errno] arch[1073741886]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:1017 - Merging compat seccomp contexts into main context
ERROR    apparmor - ../src/lxc/lsm/apparmor.c:run_apparmor_parser:916 - Failed to run apparmor_parser on "/var/lib/lxc/200/apparmor/lxc-200_<-var-lib-lxc>": Failed setting up policy cache (/var/cache/lxc/apparmor): Input/output error
ERROR    apparmor - ../src/lxc/lsm/apparmor.c:apparmor_prepare:1088 - Failed to load generated AppArmor profile
ERROR    start - ../src/lxc/start.c:lxc_init:876 - Failed to initialize LSM
ERROR    start - ../src/lxc/start.c:__lxc_start:2027 - Failed to initialize container "200"
WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_payload_destroy:555 - Uninitialized limit cgroup
WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_monitor_destroy:881 - Uninitialized monitor cgroup
INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "200", config section "lxc"


Debug info
No sure if its a packgae issue


root@node5:~# apparmor_parser --version
AppArmor parser version 2.13.6
Copyright (C) 1999-2008 Novell Inc.
Copyright 2009-2018 Canonical Ltd.


Config is checked, reinstall tried, all CT is not starting, this CT is a debian CT
installed linux packages

root@node5:~# apt list --installed | grep linux

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

binutils-x86-64-linux-gnu/stable,now 2.35.2-2 amd64 [installed,automatic]
console-setup-linux/stable,now 1.205 all [installed]
liblinux-inotify2-perl/stable,now 1:2.2-2+b1 amd64 [installed]
libnvpair3linux/stable,now 2.1.9-pve1 amd64 [installed]
libselinux1/stable,now 3.1-3 amd64 [installed]
libuutil3linux/stable,now 2.1.9-pve1 amd64 [installed]
libzfs4linux/stable,now 2.1.9-pve1 amd64 [installed]
libzpool5linux/stable,now 2.1.9-pve1 amd64 [installed]
linux-base/stable,now 4.6 all [installed]
util-linux/stable,stable-security,now 2.36.1-8+deb11u1 amd64 [installed]
zfsutils-linux/stable,now 2.1.9-pve1 amd64 [installed]
root@node5:~#
 
Hi,
please post the output of pveversion -v and pct config 200.
 
  • Like
Reactions: kenren98
Code: 
root@node5:~# pveversion -v
proxmox-ve: 7.4-1 (running kernel: 5.15.104-1-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.15: 7.4-1
pve-kernel-5.15.104-1-pve: 5.15.104-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
pve-kernel-5.15.85-1-pve: 5.15.85-1
pve-kernel-5.15.83-1-pve: 5.15.83-1
pve-kernel-5.15.64-1-pve: 5.15.64-1
ceph: 16.2.11-pve1
ceph-fuse: 16.2.11-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-4
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-1
libpve-rs-perl: 0.7.5
libpve-storage-perl: 7.4-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.1-1
proxmox-backup-file-restore: 2.4.1-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-widget-toolkit: 3.6.5
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-1
pve-firewall: 4.3-1
pve-firmware: 3.6-4
pve-ha-manager: 3.6.0
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1
root@node5:~#

temporarily moved to another node and tested working

root@node8:~# pct config 200
arch: amd64
cores: 2
features: nesting=1,keyctl=1
hostname: DNS-AdGuard
memory: 2048
net0: name=eth0,bridge=vmbr0,gw=192.168.1.1,hwaddr=36:35:FB:A7:C2:02,ip=192.168.3.200/22,type=veth
onboot: 1
ostype: debian
rootfs: Node8-V1:vm-200-disk-0,size=20G
swap: 1024
unprivileged: 1
root@node8:~#

Thank you so much!
 
  • Like
Reactions: kenren98
aa-status
Code: 
root@node5:~# aa-status
apparmor module is loaded.
14 profiles are loaded.
14 profiles are in enforce mode.
   /usr/bin/lxc-start
   /usr/bin/man
   /usr/sbin/chronyd
   lsb_release
   lxc-container-default
   lxc-container-default-cgns
   lxc-container-default-with-mounting
   lxc-container-default-with-nesting
   man_filter
   man_groff
   nvidia_modprobe
   nvidia_modprobe//kmod
   swtpm
   tcpdump
0 profiles are in complain mode.
2 processes have profiles defined.
2 processes are in enforce mode.
   /usr/sbin/chronyd (989)
   /usr/sbin/chronyd (995)
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.
root@node5:~#

systemctl
Code: 
● apparmor.service - Load AppArmor profiles
     Loaded: loaded (/lib/systemd/system/apparmor.service; enabled; vendor preset: enabled)
     Active: active (exited) since Mon 2023-04-03 11:39:31 EDT; 1 day 18h ago
       Docs: man:apparmor(7)
             https://gitlab.com/apparmor/apparmor/wikis/home/
    Process: 766 ExecStart=/lib/apparmor/apparmor.systemd reload (code=exited, status=0/SUCCESS)
    Process: 642257 ExecReload=/lib/apparmor/apparmor.systemd reload (code=exited, status=0/SUCCESS)
   Main PID: 766 (code=exited, status=0/SUCCESS)
        CPU: 28ms

Apr 03 11:39:31 node5 systemd[1]: Starting Load AppArmor profiles...
Apr 03 11:39:31 node5 apparmor.systemd[766]: Restarting AppArmor
Apr 03 11:39:31 node5 apparmor.systemd[766]: Reloading AppArmor profiles
Apr 03 11:39:31 node5 systemd[1]: Finished Load AppArmor profiles.
Apr 05 06:02:51 node5 systemd[1]: Reloading Load AppArmor profiles.
Apr 05 06:02:51 node5 apparmor.systemd[642257]: Restarting AppArmor
Apr 05 06:02:51 node5 apparmor.systemd[642257]: Reloading AppArmor profiles
Apr 05 06:02:51 node5 systemd[1]: Reloaded Load AppArmor profiles.


I dont think there is any difference between them cuz I usually update them all together. And both of them are Intel CPUS, 8700K for node5, E5-2680V3 for node8. And I am pretty sure it works fine before the update, but after the update, I tried to reinstall. It would not boot even for a new CT, I would assume it is some config problems. The new kernel is updated in two days ago, after the

1680690341222.png
1680690351767.png

I just made cloned the working CT200 to CT777 and same error occurs
1680690579239.png
 
fiona said:
Anything obviously different between the nodes?

I found another thread with a similar issue. Following that thread, can you post the output of aa-status? If it's not mounted, you can try systemctl reload-or-restart apparmor.service.
Click to expand...
Also about the other one with similar issue, I saw that one too, but it seems reboot didnt solve my problem.

journalctl -b -u lxcfs; systemctl status lxcfs
Code: 
root@node5:~# journalctl -b -u lxcfs; systemctl status lxcfs
-- Journal begins at Tue 2022-10-11 16:26:17 EDT, ends at Wed 2023-04-05 06:30:45 E>
Apr 05 06:22:23 node5 systemd[1]: Started FUSE filesystem for LXC.
Apr 05 06:22:23 node5 lxcfs[793]: Running constructor lxcfs_init to reload liblxcfs
Apr 05 06:22:23 node5 lxcfs[793]: mount namespace: 5
Apr 05 06:22:23 node5 lxcfs[793]: hierarchies:
Apr 05 06:22:23 node5 lxcfs[793]:   0: fd:   6: cpuset,cpu,io,memory,hugetlb,pids,r>
Apr 05 06:22:23 node5 lxcfs[793]: Kernel supports pidfds
Apr 05 06:22:23 node5 lxcfs[793]: Kernel supports swap accounting
Apr 05 06:22:23 node5 lxcfs[793]: api_extensions:
Apr 05 06:22:23 node5 lxcfs[793]: - cgroups
Apr 05 06:22:23 node5 lxcfs[793]: - sys_cpu_online
Apr 05 06:22:23 node5 lxcfs[793]: - proc_cpuinfo
Apr 05 06:22:23 node5 lxcfs[793]: - proc_diskstats
Apr 05 06:22:23 node5 lxcfs[793]: - proc_loadavg
Apr 05 06:22:23 node5 lxcfs[793]: - proc_meminfo
Apr 05 06:22:23 node5 lxcfs[793]: - proc_stat
Apr 05 06:22:23 node5 lxcfs[793]: - proc_swaps
Apr 05 06:22:23 node5 lxcfs[793]: - proc_uptime
Apr 05 06:22:23 node5 lxcfs[793]: - proc_slabinfo
Apr 05 06:22:23 node5 lxcfs[793]: - shared_pidns
Apr 05 06:22:23 node5 lxcfs[793]: - cpuview_daemon
Apr 05 06:22:23 node5 lxcfs[793]: - loadavg_daemon
Apr 05 06:22:23 node5 lxcfs[793]: - pidfds
● lxcfs.service - FUSE filesystem for LXC
     Loaded: loaded (/lib/systemd/system/lxcfs.service; enabled; vendor preset: ena>
     Active: active (running) since Wed 2023-04-05 06:22:23 EDT; 8min ago
       Docs: man:lxcfs(1)
   Main PID: 793 (lxcfs)
      Tasks: 3 (limit: 28574)
     Memory: 940.0K
        CPU: 1ms
     CGroup: /system.slice/lxcfs.service
             └─793 /usr/bin/lxcfs /var/lib/lxcfs

Apr 05 06:22:23 node5 lxcfs[793]: - proc_loadavg
Apr 05 06:22:23 node5 lxcfs[793]: - proc_meminfo
Apr 05 06:22:23 node5 lxcfs[793]: - proc_stat
Apr 05 06:22:23 node5 lxcfs[793]: - proc_swaps
Apr 05 06:22:23 node5 lxcfs[793]: - proc_uptime
Apr 05 06:22:23 node5 lxcfs[793]: - proc_slabinfo
Apr 05 06:22:23 node5 lxcfs[793]: - shared_pidns
Apr 05 06:22:23 node5 lxcfs[793]: - cpuview_daemon
Apr 05 06:22:23 node5 lxcfs[793]: - loadavg_daemon
Apr 05 06:22:23 node5 lxcfs[793]: - pidfds


Node5 journalctl -b | grep -i -e apparmor -e aa_ | head -n 15
Code: 
root@node5:~# journalctl -b | grep -i -e apparmor -e aa_ | head -n 15
Apr 05 06:22:22 node5 kernel: AppArmor: AppArmor initialized
Apr 05 06:22:22 node5 kernel: AppArmor: AppArmor Filesystem Enabled
Apr 05 06:22:22 node5 kernel: AppArmor: AppArmor sha1 policy hashing enabled
Apr 05 06:22:22 node5 kernel: evm: security.apparmor
Apr 05 06:22:22 node5 systemd[1]: systemd 247.3-7+1-pmx11u1 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified)
Apr 05 06:22:23 node5 systemd[1]: Starting Load AppArmor profiles...
Apr 05 06:22:23 node5 apparmor.systemd[752]: Restarting AppArmor
Apr 05 06:22:23 node5 apparmor.systemd[752]: Reloading AppArmor profiles
Apr 05 06:22:23 node5 audit[785]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="swtpm" pid=785 comm="apparmor_parser"
Apr 05 06:22:23 node5 kernel: audit: type=1400 audit(1680690143.278:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="swtpm" pid=785 comm="apparmor_parser"
Apr 05 06:22:23 node5 kernel: audit: type=1400 audit(1680690143.278:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=781 comm="apparmor_parser"
Apr 05 06:22:23 node5 audit[781]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=781 comm="apparmor_parser"
Apr 05 06:22:23 node5 audit[778]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=778 comm="apparmor_parser"
Apr 05 06:22:23 node5 audit[784]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=784 comm="apparmor_parser"
Apr 05 06:22:23 node5 audit[784]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=784 comm="apparmor_parser"


Node8 journalctl -b | grep -i -e apparmor -e aa_ | head -n 15
Code: 
root@node8:~# journalctl -b | grep -i -e apparmor -e aa_ | head -n 15
Apr 05 06:24:30 node8 kernel: AppArmor: AppArmor initialized
Apr 05 06:24:30 node8 kernel: AppArmor: AppArmor Filesystem Enabled
Apr 05 06:24:30 node8 kernel: AppArmor: AppArmor sha1 policy hashing enabled
Apr 05 06:24:30 node8 kernel: evm: security.apparmor
Apr 05 06:24:30 node8 systemd[1]: systemd 247.3-7+1-pmx11u1 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified)
Apr 05 06:24:31 node8 systemd[1]: Starting Load AppArmor profiles...
Apr 05 06:24:31 node8 apparmor.systemd[963]: Restarting AppArmor
Apr 05 06:24:31 node8 apparmor.systemd[963]: Reloading AppArmor profiles
Apr 05 06:24:31 node8 audit[991]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="swtpm" pid=991 comm="apparmor_parser"
Apr 05 06:24:31 node8 audit[993]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=993 comm="apparmor_parser"
Apr 05 06:24:31 node8 kernel: audit: type=1400 audit(1680690271.314:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="swtpm" pid=991 comm="apparmor_parser"
Apr 05 06:24:31 node8 kernel: audit: type=1400 audit(1680690271.314:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=993 comm="apparmor_parser"
Apr 05 06:24:31 node8 audit[990]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=990 comm="apparmor_parser"
Apr 05 06:24:31 node8 audit[996]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=996 comm="apparmor_parser"
Apr 05 06:24:31 node8 audit[996]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=996 comm="apparmor_parser"


Thank you again for helping!
 
Is there anything interesting in /var/log/syslog? If the issue only appears on the specific node, you could try to e.g. reinstall lxc-related packages, check if there's some configuration difference to other nodes, check the filesystem and disk that /var/cache/lxc/apparmor belongs to and the permissions for that path.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom