I recently performed an
I found out none of my containers ran after doing a system reboot so I tried
Not entirely sure which package caused it, but this this is the apt/history.log
I tried
Trying to access the apparmor directory shows that it doesn't exist, could the upgrade have deleted the directory?
My filesystem is ext4, many issues I found regarding upgrade failures involves zfs but I don't use zfs
I'm not familiar enough with apparmor to go any deeper and also not entirely sure how to use tools/lxc_start.c directly with the --logfile/--logpriority options either, not sure what other logs/config files would be helpful in finding the issue, but here are a few more:
apt upgrade and my lxc containers stopped working. When starting a container, no error message appears and the web UI responds with "Task OK" but the container doesn't actually start. I should have used apt dist-upgrade instead, but I'm not too sure how to rollback an upgrade and don't have snapshots in placeI found out none of my containers ran after doing a system reboot so I tried
pct start 100 also, and no error message was displayed, but trying to pct enter 100 returns "Error: container '100' not running!"Not entirely sure which package caused it, but this this is the apt/history.log
Code:
# tail /var/log/apt/history.log
Start-Date: 2020-07-11 10:24:37
Commandline: apt upgrade
Install: pve-headers-5.4.44-2-pve:amd64 (5.4.44-2, automatic), proxmox-backup-client:amd64 (0.8.6-1, automatic), pve-kernel-5.4.44-2-pve:amd64 (5.4.44-2, automatic)
Upgrade: proxmox-widget-toolkit:amd64 (2.2-8, 2.2-9), pve-kernel-5.4:amd64 (6.2-3, 6.2-4), corosync:amd64 (3.0.3-pve1, 3.0.4-pve1), libavformat58:amd64 (7:4.1.4-1~deb10u1, 7:4.1.6-1~deb10u1), libcmap4:amd64 (3.0.3-pve1, 3.0.4-pve1), libavfilter7:amd64 (7:4.1.4-1~deb10u1, 7:4.1.6-1~deb10u1), libpve-access-control:amd64 (6.1-1, 6.1-2), libpve-storage-perl:amd64 (6.1-8, 6.2-3), libswresample3:amd64 (7:4.1.4-1~deb10u1, 7:4.1.6-1~deb10u1), libquorum5:amd64 (3.0.3-pve1, 3.0.4-pve1), pve-qemu-kvm:amd64 (5.0.0-4, 5.0.0-9), libmagickwand-6.q16-6:amd64 (8:6.9.10.23+dfsg-2.1, 8:6.9.10.23+dfsg-2.1+deb10u1), pve-container:amd64 (3.1-8, 3.1-10), libpostproc55:amd64 (7:4.1.4-1~deb10u1, 7:4.1.6-1~deb10u1), pve-manager:amd64 (6.2-6, 6.2-9), libvotequorum8:amd64 (3.0.3-pve1, 3.0.4-pve1), libpve-guest-common-perl:amd64 (3.0-10, 3.0-11), libavcodec58:amd64 (7:4.1.4-1~deb10u1, 7:4.1.6-1~deb10u1), libpve-common-perl:amd64 (6.1-3, 6.1-5), libavutil56:amd64 (7:4.1.4-1~deb10u1, 7:4.1.6-1~deb10u1), qemu-server:amd64 (6.2-3, 6.2-8), libcfg7:amd64 (3.0.3-pve1, 3.0.4-pve1), libproxmox-backup-qemu0:amd64 (0.1.6-1, 0.6.1-1), libswscale5:amd64 (7:4.1.4-1~deb10u1, 7:4.1.6-1~deb10u1), libknet1:amd64 (1.15-pve1, 1.16-pve1), libmagickcore-6.q16-6:amd64 (8:6.9.10.23+dfsg-2.1, 8:6.9.10.23+dfsg-2.1+deb10u1), pve-headers-5.4:amd64 (6.2-3, 6.2-4), pve-kernel-helper:amd64 (6.2-3, 6.2-4), libpve-http-server-perl:amd64 (3.0-5, 3.0-6), libcpg4:amd64 (3.0.3-pve1, 3.0.4-pve1), libcorosync-common4:amd64 (3.0.3-pve1, 3.0.4-pve1), imagemagick-6-common:amd64 (8:6.9.10.23+dfsg-2.1, 8:6.9.10.23+dfsg-2.1+deb10u1)
End-Date: 2020-07-11 10:26:03I tried
lxc-start with logs instead, and got these messages:
Code:
# lxc-start -n 100 -F -l DEBUG -o /tmp/lxc-100.log
lxc-start: 100: lsm/apparmor.c: run_apparmor_parser: 892 Failed to run apparmor_parser on "/var/lib/lxc/100/apparmor/lxc-100_<-var-lib-lxc>": apparmor_parser: Unable to replace "lxc-100_</var/lib/lxc>". Profile doesn't conform to protocol
lxc-start: 100: lsm/apparmor.c: apparmor_prepare: 1064 Failed to load generated AppArmor profile
lxc-start: 100: start.c: lxc_init: 845 Failed to initialize LSM
lxc-start: 100: start.c: __lxc_start: 1903 Failed to initialize container "100"
lxc-start: 100: tools/lxc_start.c: main: 308 The container failed to start
lxc-start: 100: tools/lxc_start.c: main: 314 Additional information can be obtained by setting the --logfile and --logpriority options
# tail /tmp/lxc-100.log
lxc-start 100 20200712012140.203 ERROR start - start.c:lxc_init:845 - Failed to initialize LSM
lxc-start 100 20200712012140.203 ERROR start - start.c:__lxc_start:1903 - Failed to initialize container "100"
lxc-start 100 20200712012140.203 DEBUG conf - conf.c:idmaptool_on_path_and_privileged:2642 - The binary "/usr/bin/newuidmap" does have the setuid bit set
lxc-start 100 20200712012140.203 DEBUG conf - conf.c:idmaptool_on_path_and_privileged:2642 - The binary "/usr/bin/newgidmap" does have the setuid bit set
lxc-start 100 20200712012140.203 DEBUG conf - conf.c:lxc_map_ids:2710 - Functional newuidmap and newgidmap binary found
lxc-start 100 20200712012140.208 NOTICE utils - utils.c:lxc_setgroups:1366 - Dropped additional groups
lxc-start 100 20200712012140.208 INFO conf - conf.c:run_script_argv:340 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "100", config section "lxc"
lxc-start 100 20200712012140.893 INFO conf - conf.c:run_script_argv:340 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "100", config section "lxc"
lxc-start 100 20200712012141.395 ERROR lxc_start - tools/lxc_start.c:main:308 - The container failed to start
lxc-start 100 20200712012141.395 ERROR lxc_start - tools/lxc_start.c:main:314 - Additional information can be obtained by setting the --logfile and --logpriority optionsTrying to access the apparmor directory shows that it doesn't exist, could the upgrade have deleted the directory?
Code:
# ls /var/lib/lxc/100/apparmor
ls: cannot access '/var/lib/lxc/100/apparmor': No such file or directory
# ls -l /var/lib/lxc/100/
total 8
-rw-r--r-- 1 root root 977 Jul 12 09:21 config
drwxr-xr-x 2 root root 4096 Jun 15 2019 rootfsMy filesystem is ext4, many issues I found regarding upgrade failures involves zfs but I don't use zfs
I'm not familiar enough with apparmor to go any deeper and also not entirely sure how to use tools/lxc_start.c directly with the --logfile/--logpriority options either, not sure what other logs/config files would be helpful in finding the issue, but here are a few more:
Code:
# pct config 100
arch: amd64
cores: 2
hostname: apache
memory: 512
nameserver: 1.1.1.1
net0: name=eth0,bridge=vmbr0,gw=192.168.0.1,hwaddr=82:B1:0D:3C:47:68,ip=192.168.0.42/16,ip6=dhcp,type=veth
onboot: 1
ostype: ubuntu
parent: upgrade
rootfs: local-lvm:vm-100-disk-0,size=20G
startup: order=1,up=30
swap: 1024
unprivileged: 1
# systemctl status pve-container@100.service
● pve-container@100.service - PVE LXC Container: 100
Loaded: loaded (/lib/systemd/system/pve-container@.service; static; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2020-07-12 09:27:47 +08; 16min ago
Docs: man:lxc-start
man:lxc
man:pct
Process: 30827 ExecStart=/usr/bin/lxc-start -F -n 100 (code=exited, status=1/FAILURE)
Main PID: 30827 (code=exited, status=1/FAILURE)
Jul 12 09:27:44 alpha systemd[1]: Started PVE LXC Container: 100.
Jul 12 09:27:47 alpha systemd[1]: pve-container@100.service: Main process exited, code=exited, status=1/FAILURE
Jul 12 09:27:47 alpha systemd[1]: pve-container@100.service: Failed with result 'exit-code'.
# journalctl -xe
-- The job identifier is 100128.
Jul 12 09:50:16 alpha systemd[1]: Started PVE LXC Container: 100.
-- Subject: A start job for unit pve-container@100.service has finished successfully
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit pve-container@100.service has finished successfully.
--
-- The job identifier is 100210.
Jul 12 09:50:16 alpha kernel: EXT4-fs (dm-13): mounted filesystem with ordered data mode. Opts: (null)
Jul 12 09:50:17 alpha audit[1534]: AVC apparmor="STATUS" info="failed to unpack end of profile" error=-71 profile="unconfined" name="lxc-100_</var/lib/lxc>" pid=1534 comm="apparmor_parser" name="lxc-100_</var/lib/lxc>" offset=151
Jul 12 09:50:17 alpha kernel: audit: type=1400 audit(1594518617.147:54): apparmor="STATUS" info="failed to unpack end of profile" error=-71 profile="unconfined" name="lxc-100_</var/lib/lxc>" pid=1534 comm="apparmor_parser" name="lxc-100_</var/lib/lxc>" offset=151
Jul 12 09:50:18 alpha systemd[1]: pve-container@100.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- An ExecStart= process belonging to unit pve-container@100.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Jul 12 09:50:18 alpha systemd[1]: pve-container@100.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit pve-container@100.service has entered the 'failed' state with result 'exit-code'.