LXC problem after PVE kernel update

KenyBDG

New Member
Aug 10, 2019
13
0
1
38
Today I made upgraded my proxmox host pve and unfortunately after upgrade process one of my LXC container stopped working.
Now I'm struggling how to solved trhis issue, but I'm not very familiar with proxmox .....

Code:
Linux pve 4.16.0-041600-generic #201804012230 SMP Sun Apr 1 22:31:39 UTC 2018 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@pve:~# systemctl status lxc@100.service
● lxc@100.service - LXC Container: 100
   Loaded: loaded (/lib/systemd/system/lxc@.service; disabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/lxc@.service.d
           └─pve-reboot.conf
   Active: inactive (dead)
     Docs: man:lxc-start
           man:lxc
root@pve:~# journalctl -xe
--
-- Unit pvesr.service has finished starting up.
--
-- The start-up result is done.
Oct 02 14:26:37 pve pvedaemon[1742]: <root@pam> end task UPID:pve:0000126E:0002A1B9:5D949743:vncshell::root@pam: OK
Oct 02 14:26:38 pve pvedaemon[4818]: starting CT 100: UPID:pve:000012D2:0002B8FC:5D94977E:vzstart:100:root@pam:
Oct 02 14:26:38 pve pvedaemon[1743]: <root@pam> starting task UPID:pve:000012D2:0002B8FC:5D94977E:vzstart:100:root@pam:
Oct 02 14:26:38 pve systemd[1]: Starting PVE LXC Container: 100...
-- Subject: Unit pve-container@100.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pve-container@100.service has begun starting up.
Oct 02 14:26:39 pve kernel: EXT4-fs (dm-7): mounted filesystem with ordered data mode. Opts: (null)
Oct 02 14:26:40 pve audit[4842]: AVC apparmor="STATUS" info="failed to unpack end of profile" error=-71 profile="unconfined" name="lxc-100_</var/lib/lxc>" pid=4842 comm="
Oct 02 14:26:40 pve kernel: audit: type=1400 audit(1570019200.836:11): apparmor="STATUS" info="failed to unpack end of profile" error=-71 profile="unconfined" name="lxc-1
Oct 02 14:26:41 pve lxc-start[4820]: lxc-start: 100: lxccontainer.c: wait_on_daemonized_start: 856 No such file or directory - Failed to receive the container state
Oct 02 14:26:41 pve lxc-start[4820]: lxc-start: 100: tools/lxc_start.c: main: 330 The container failed to start
Oct 02 14:26:41 pve lxc-start[4820]: lxc-start: 100: tools/lxc_start.c: main: 333 To get more details, run the container in foreground mode
Oct 02 14:26:41 pve lxc-start[4820]: lxc-start: 100: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority option
Oct 02 14:26:41 pve pvedaemon[1743]: unable to get PID for CT 100 (not running?)
Oct 02 14:26:41 pve systemd[1]: pve-container@100.service: Control process exited, code=exited status=1
Oct 02 14:26:41 pve pvedaemon[4818]: command 'systemctl start pve-container@100' failed: exit code 1
Oct 02 14:26:41 pve systemd[1]: Failed to start PVE LXC Container: 100.
-- Subject: Unit pve-container@100.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pve-container@100.service has failed.
--
-- The result is failed.
Oct 02 14:26:41 pve pvedaemon[1743]: <root@pam> end task UPID:pve:000012D2:0002B8FC:5D94977E:vzstart:100:root@pam: command 'systemctl start pve-container@100' failed: exi
Oct 02 14:26:41 pve systemd[1]: pve-container@100.service: Unit entered failed state.
Oct 02 14:26:41 pve systemd[1]: pve-container@100.service: Failed with result 'exit-code'.
Oct 02 14:26:41 pve pvedaemon[4845]: starting termproxy UPID:pve:000012ED:0002B9DC:5D949781:vncshell::root@pam:
Oct 02 14:26:41 pve pvedaemon[1744]: <root@pam> starting task UPID:pve:000012ED:0002B9DC:5D949781:vncshell::root@pam:
Oct 02 14:26:41 pve pvedaemon[1742]: <root@pam> successful auth for user 'root@pam'
Oct 02 14:26:41 pve login[4850]: pam_unix(login:session): session opened for user root by root(uid=0)
Oct 02 14:26:41 pve login[4855]: ROOT LOGIN  on '/dev/pts/0
 
Here is CT config and debug.log

Code:
root@pve:/tmp# pveversion -v
proxmox-ve: 5.4-2 (running kernel: 4.16.0-041600-generic)
pve-manager: 5.4-13 (running version: 5.4-13/aee6f0ec)
pve-kernel-4.15: 5.4-9
pve-kernel-4.15.18-21-pve: 4.15.18-48
pve-kernel-4.15.18-19-pve: 4.15.18-45
pve-kernel-4.15.18-12-pve: 4.15.18-36
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-12
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-55
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-14
libpve-storage-perl: 5.0-44
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-7
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-28
pve-cluster: 5.0-38
pve-container: 2.0-40
pve-docs: 5.4-2
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-22
pve-firmware: 2.0-7
pve-ha-manager: 2.0-9
pve-i18n: 1.1-4
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 3.0.1-4
pve-xtermjs: 3.12.0-1
qemu-server: 5.0-54
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.13-pve1~bpo2
 

Attachments

proxmox-ve: 5.4-2 (running kernel: 4.16.0-041600-generic)

why are you running a generic kernel instead of pve kernel? that won't work since we do modifications for apparmor & co
 
hmmm, I don't really know why, today morning one of my collegues tried install new PCIe card, so myabe that's why.
Question is there any way to rollback kernel to pve ?

Edit:

Now i know what he has done:
https://www.tbsdtv.com/forum/viewtopic.php?f=86&t=25046


update the kernel you can do like this :
wget http://kernel.ubuntu.com/~kernel-pp...6.0-041600_4.16.0-041600.201804012230_all.deb
wget http://kernel.ubuntu.com/~kernel-pp...-generic_4.16.0-041600.201804012230_amd64.deb
wget http://kernel.ubuntu.com/~kernel-pp...-generic_4.16.0-041600.201804012230_amd64.deb
sudo dpkg -i *.deb

Is there any way to rollback this karnel upgrade ? I would like avoid install proxmox instance once again
 
Last edited:
just boot into a pve-kernel (the boot loader should let you select which kernel to boot), then remove the extra packages you installed with "apt remove linux-headers-4.16.0-041600 linux-headers-4.16.0-041600-generic linux-image-4.16.0-041600-generic"
 
Thanks for your quick answer Fabian. Unfortunately I can only manage this device remotely, so I would like avoid situation when I removed something and proxmox will stuck on boot procedure. As I mentioned on the begining, I'm not expert on proxmox, so maybe you know how to do it ? :)
 
that's why you should reboot first, then remove ;) you can always verify with "uname -a" that you are not currently running the kernel you are about to remove..
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!