[SOLVED] pct start failed after apparmor update

[volvic]

Hello,

Probably following an update from cron-apt (security only), lxc container does not reponse.

I restarted the server and now, the container will not start.

Someone would see a solution?

I tried :

~# lxc-start -n 100 -F -lDEBUG -o lxc-100.log --logpriority TRACE
lxc-start: 100: lxccontainer.c: do_lxcapi_start: 984 Permission denied - Failed to make / rslave at startup
lxc-start: 100: tools/lxc_start.c: main: 368 The container failed to start.
lxc-start: 100: tools/lxc_start.c: main: 372 Additional information can be obtained by setting the --logfile and --logpriority options.

~# cat lxc-100.log
lxc-start 100 20171204134744.245 INFO     lxc_start_ui - tools/lxc_start.c:main:277 - using rcfile /var/lib/lxc/100/config
lxc-start 100 20171204134744.245 TRACE    lxc_commands - commands.c:lxc_cmd:290 - command get_init_pid tries to connect command socket
lxc-start 100 20171204134744.245 TRACE    lxc_commands - commands.c:lxc_cmd:295 - command get_init_pid failed to connect command socket: Connection refused
lxc-start 100 20171204134744.245 TRACE    lxc_commands - commands.c:lxc_cmd:290 - command get_cgroup tries to connect command socket
lxc-start 100 20171204134744.245 TRACE    lxc_commands - commands.c:lxc_cmd:295 - command get_cgroup failed to connect command socket: Connection refused
lxc-start 100 20171204134744.245 TRACE    lxc_commands - commands.c:do_lxc_cmd_get_cgroup_path:439 - command get_cgroup failed for container "100": Connection refused.
lxc-start 100 20171204134744.245 TRACE    lxc_commands - commands.c:lxc_cmd:290 - command get_state tries to connect command socket
lxc-start 100 20171204134744.245 TRACE    lxc_commands - commands.c:lxc_cmd:295 - command get_state failed to connect command socket: Connection refused
lxc-start 100 20171204134744.245 TRACE    lxc_start - start.c:lxc_init_handler:583 - unix domain socket 4 for command server is ready
lxc-start 100 20171204134744.245 ERROR    lxc_container - lxccontainer.c:do_lxcapi_start:984 - Permission denied - Failed to make / rslave at startup
lxc-start 100 20171204134744.245 ERROR    lxc_start_ui - tools/lxc_start.c:main:368 - The container failed to start.
lxc-start 100 20171204134744.245 ERROR    lxc_start_ui - tools/lxc_start.c:main:372 - Additional information can be obtained by setting the --logfile and --logpriority options.

~# systemctl status pve-container@100.service
● pve-container@100.service - PVE LXC Container: 100
   Loaded: loaded (/lib/systemd/system/pve-container@.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2017-12-04 14:42:31 CET; 10min ago
     Docs: man:lxc-start
           man:lxc
           man:pct
  Process: 32433 ExecStart=/usr/bin/lxc-start -n 100 (code=exited, status=1/FAILURE)

déc. 04 14:42:30 antares systemd[1]: Starting PVE LXC Container: 100...
déc. 04 14:42:30 antares lxc-start[32433]: lxc-start: 100: lxccontainer.c: wait_on_daemonized_start: 751 No such file or directory - Failed to receive the container state
déc. 04 14:42:30 antares lxc-start[32433]: lxc-start: 100: tools/lxc_start.c: main: 368 The container failed to start.
déc. 04 14:42:30 antares lxc-start[32433]: lxc-start: 100: tools/lxc_start.c: main: 370 To get more details, run the container in foreground mode.
déc. 04 14:42:30 antares lxc-start[32433]: lxc-start: 100: tools/lxc_start.c: main: 372 Additional information can be obtained by setting the --logfile and --logpriority options
déc. 04 14:42:30 antares systemd[1]: pve-container@100.service: Control process exited, code=exited status=1
déc. 04 14:42:30 antares systemd[1]: pve-container@100.service: Killing process 32435 (lxc-start) with signal SIGKILL.
déc. 04 14:42:31 antares systemd[1]: Failed to start PVE LXC Container: 100.
déc. 04 14:42:31 antares systemd[1]: pve-container@100.service: Unit entered failed state.
déc. 04 14:42:31 antares systemd[1]: pve-container@100.service: Failed with result 'exit-code'.

~# pveversion -v
proxmox-ve: 5.1-28 (running kernel: 4.13.8-2-pve)
pve-manager: 5.1-36 (running version: 5.1-36/131401db)
pve-kernel-4.13.4-1-pve: 4.13.4-26
pve-kernel-4.13.8-2-pve: 4.13.8-28
pve-kernel-4.10.17-4-pve: 4.10.17-24
pve-kernel-4.10.17-3-pve: 4.10.17-23
pve-kernel-4.10.17-1-pve: 4.10.17-18
libpve-http-server-perl: 2.0-6
lvm2: 2.02.168-pve6
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-15
qemu-server: 5.0-17
pve-firmware: 2.0-3
libpve-common-perl: 5.0-20
libpve-guest-common-perl: 2.0-13
libpve-access-control: 5.0-7
libpve-storage-perl: 5.0-16
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-2
pve-docs: 5.1-12
pve-qemu-kvm: 2.9.1-2
pve-container: 2.0-17
pve-firewall: 3.0-3
pve-ha-manager: 2.0-3
ksm-control-daemon: not correctly installed
glusterfs-client: 3.8.8-1
lxc-pve: 2.1.0-2
lxcfs: 2.0.7-pve4
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.7.3-pve1~bpo9

~# cat /etc/pve/lxc/100.conf
arch: amd64
cores: 1
hostname: cloud
memory: 2048
net0: name=eth0,bridge=vmbr0,gw=gateway,hwaddr=mac,ip=@ip/32,type=veth
onboot: 1
ostype: debian
rootfs: local:100/vm-100-disk-1.raw,acl=1,size=50G
swap: 1024

 

[volvic]

At first, the problem occurs after one of these updates :

apparmor dbus libapparmor-perl libapparmor1 libc-bin libc-dev-bin libc-l10n libc6 libc6-dev libcurl3-gnutls libdbi1 libdbus-1-3 libicu57 libpython2.7 libpython2.7-minimal libpython2.7-stdlib linux-compiler-gcc-6-x86 linux-headers-4.9.0-4-amd64 linux-headers-4.9.0-4-common linux-kbuild-4.9 linux-libc-dev locales multiarch-support python2.7 python2.7-minimal

 

[volvic]

The problem would come from the kernel pve-kernel-4.13.8-2-pve: 4.13.8-28

see :
https://forum.proxmox.com/threads/pve-5-1-kvm-broken-on-old-cpus.37666/

 

[wolfgang]

Hi,

you can try the 4.13.8-3 kernel
http://download.proxmox.com/debian/...4/pve-kernel-4.13.8-3-pve_4.13.8-29_amd64.deb

There is a fix for old cpu flags.

 

[volvic]

Thank Wolfgang, but that does not solve my problem.
I would like to downgrade my kernel, I search how to

 

[wolfgang]

hear is how to do
https://forum.proxmox.com/threads/b..._corruption-proxmox-5-1-36.38128/#post-188276

 

[volvic]

Yes, thanks, but I don't have access to boot grub.
This is a rental dedicated server.

 

[fabian]

the last apparmor package update in Debian was in March (there is one pending on proposed though..). double check which packages got upgraded (and from which version to which version) and also post your configured repositories

 

[fabian]

this is an issue with apparmor in stretch-proposed-updates - we'll find a solution before the upcoming point release on sunday!

 

[volvic]

okay, thanks again

apparmor 2.11.0-3+deb9u1 => not work
apparmor 2.11.0-3 => work fine

 

[marcori]

apparmor 2.11.0-3+deb9u1 => not work
apparmor 2.11.0-3 => work fine

Thank you Volvic, It was just very stressful !
Thanks to you, I will be able to relax.
However, this does not remove the bug with this apparmor version.
Get ready for the next update

 

[fabian]

the apparmor update is no longer slated for the next Debian point release.

if you have installed it because you have stretch-proposed-updates configured as repository, you need to downgrade to the previous version. you probably should not have stretch-proposed-updates configured on production machines

check all installed apparmor packages with broken version (should only list apparmor related packages):

$ dpkg-query --show -f '${Package},${Version}\n' | grep '2.11.0-3+deb9u1$'
apparmor,2.11.0-3+deb9u1
libapparmor-perl,2.11.0-3+deb9u1
libapparmor1,2.11.0-3+deb9u1

verify generated downgrade list

$ dpkg-query --show -f '${Package},${Version}\n' | grep ',2.11.0-3+deb9u1$' | sed -e 's/,.*$/=2.11.0-3/' | tr '\n' ' '
apparmor=2.11.0-3 libapparmor-perl=2.11.0-3 libapparmor1=2.11.0-3 %

downgrade them

$ apt install $(dpkg-query --show -f '${Package},${Version}\n' | grep ',2.11.0-3+deb9u1$' | sed -e 's/,.*$/=2.11.0-3/' | tr '\n' ' ')
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  apparmor-profiles apparmor-profiles-extra apparmor-utils
The following packages will be DOWNGRADED:
  apparmor libapparmor-perl libapparmor1
0 upgraded, 0 newly installed, 3 downgraded, 0 to remove and 0 not upgraded.
Need to get 686 kB of archives.
After this operation, 4,096 B disk space will be freed.
Do you want to continue? [Y/n]
Get:1 https://cdn-aws.deb.debian.org/debian stretch/main amd64 libapparmor1 amd64 2.11.0-3 [78.7 kB]
Get:2 https://cdn-aws.deb.debian.org/debian stretch/main amd64 libapparmor-perl amd64 2.11.0-3 [82.1 kB]
Get:3 https://cdn-aws.deb.debian.org/debian stretch/main amd64 apparmor amd64 2.11.0-3 [525 kB]
Fetched 686 kB in 6s (102 kB/s)
Preconfiguring packages ...
dpkg: warning: downgrading libapparmor1:amd64 from 2.11.0-3+deb9u1 to 2.11.0-3
(Reading database ... 461104 files and directories currently installed.)
Preparing to unpack .../libapparmor1_2.11.0-3_amd64.deb ...
Unpacking libapparmor1:amd64 (2.11.0-3) over (2.11.0-3+deb9u1) ...
dpkg: warning: downgrading libapparmor-perl from 2.11.0-3+deb9u1 to 2.11.0-3
Preparing to unpack .../libapparmor-perl_2.11.0-3_amd64.deb ...
Unpacking libapparmor-perl (2.11.0-3) over (2.11.0-3+deb9u1) ...
dpkg: warning: downgrading apparmor from 2.11.0-3+deb9u1 to 2.11.0-3
Preparing to unpack .../apparmor_2.11.0-3_amd64.deb ...
Unpacking apparmor (2.11.0-3) over (2.11.0-3+deb9u1) ...
Processing triggers for libc-bin (2.24-11+deb9u1) ...
Setting up libapparmor1:amd64 (2.11.0-3) ...
Processing triggers for systemd (232-25+deb9u1) ...
Processing triggers for man-db (2.7.6.1-2) ...
Setting up libapparmor-perl (2.11.0-3) ...
Setting up apparmor (2.11.0-3) ...
Installing new version of config file /etc/apparmor/parser.conf ...
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
Processing triggers for libc-bin (2.24-11+deb9u1) ...

 

[volvic]

Thank you Fabian for the procedure, it works very well.
I removed stretch-proposed-updates from my sources list

Marcori should be really relaxed after that

 

[exochris7]

Hi, same issue with proxmox 5.1.41 and apparmor 2.11.0-3+deb9u2.
Not possible to downgrade to 2.11.0-3
Reading Package lists... done Building depency tree Reading state information... done
E: Version '2.11.0-3' for 'apparmor' was not found
E: Version '2.11.0-3' for 'libapparmor-perl' was not found
E: Version '2.11.0-3' for 'libapparmor1' was not found
Solution was: Add line “deb http://download.proxmox.com/debian/pve stretch pve-no-subscription”. to source.list
Source from: xxxx://forum.level1techs.com/t/proxmox-5-1-apparmor-updated-lxc-containers-now-will-not-start/125687/5

 

[redwoz]

Hi, same issue with proxmox 5.1.41 and apparmor 2.11.0-3+deb9u2.
Not possible to downgrade to 2.11.0-3
Reading Package lists... done Building depency tree Reading state information... done
E: Version '2.11.0-3' for 'apparmor' was not found
E: Version '2.11.0-3' for 'libapparmor-perl' was not found
E: Version '2.11.0-3' for 'libapparmor1' was not found
Solution was: Add line “deb http://download.proxmox.com/debian/pve stretch pve-no-subscription”. to source.list
Source from: xxxx://forum.level1techs.com/t/proxmox-5-1-apparmor-updated-lxc-containers-now-will-not-start/125687/5

confirmed this works for me
issue appeared after performing an update via proxmox gui

 

[mathx]

To be clear, it's the upgraded kernel from that sources list that works, not that that sources list has an old copy of apparmor + libraries to downgrade to.

You must add that line, apt-get update && apt-get dist-upgrade && shutdown -r now to reboot to the new kernel.

Thanks.

 

[Palerius]

Hi there!
Sorry for digging out this thread but I don't want to start a new one for the exact same issue.
I am running proxmox 6.4-13 and from the information that I gathered I seem to have the same issue.

I updated a container with update && dist-upgrade and can since not start the container anymore.
Looking at the syslogs I do get references to issues with apparmor.
My problem is that I don't seem to understand the solution explained here. Is there someone that can elaborate a bit more on how I can fix this?
I am running the server as a hobby so while it is not all that dramatic to lose the container I would prefer not to as it would mean a lot of time to run my backup (which is admittedly also a little older than I would like it to be)

I appreciate any input.

Cheers.

 

[mathx]

Hi there!
Sorry for digging out this thread but I don't want to start a new one for the exact same issue.
I am running proxmox 6.4-13 and from the information that I gathered I seem to have the same issue.

I updated a container with update && dist-upgrade and can since not start the container anymore.
Looking at the syslogs I do get references to issues with apparmor.
My problem is that I don't seem to understand the solution explained here. Is there someone that can elaborate a bit more on how I can fix this?
I am running the server as a hobby so while it is not all that dramatic to lose the container I would prefer not to as it would mean a lot of time to run my backup (which is admittedly also a little older than I would like it to be)

I appreciate any input.

Cheers.

Updating the kernel as we did so long ago probably wont fix it since things have moved far forward in 4 years. You have another issue. You need to find a new related thread, or start a new one, as it's unrelated to this specific issue at that time.

 

[Palerius]

Thanks for the replay. I guess I will create a new post then after all!