[SOLVED] All LXCs won't start

[Flu]

After doing OS updates today none of my containers will start. I don't have subscription or non-sub repos added.

pve-manager/5.1-41/0b958203 (running kernel: 4.13.13-2-pve)

Start from GUI:

Job for pve-container@5000.service failed because the control process exited with error code.
See "systemctl status pve-container@5000.service" and "journalctl -xe" for details.
TASK ERROR: command 'systemctl start pve-container@5000' failed: exit code 1

# systemctl status pve-container@5000.service

# systemctl status pve-container@5000.service
● pve-container@5000.service - PVE LXC Container: 5000
   Loaded: loaded (/lib/systemd/system/pve-container@.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2018-04-04 21:55:51 EDT; 26min ago
     Docs: man:lxc-start
           man:lxc
           man:pct
  Process: 3144 ExecStart=/usr/bin/lxc-start -n 5000 (code=exited, status=1/FAILURE)

Apr 04 21:55:51 rocksteady systemd[1]: Starting PVE LXC Container: 5000...
Apr 04 21:55:51 rocksteady lxc-start[3144]: lxc-start: 5000: lxccontainer.c: wait_on_daemonized_start: 751 No such file or directory - Failed to receive the container state
Apr 04 21:55:51 rocksteady lxc-start[3144]: lxc-start: 5000: tools/lxc_start.c: main: 371 The container failed to start.
Apr 04 21:55:51 rocksteady lxc-start[3144]: lxc-start: 5000: tools/lxc_start.c: main: 373 To get more details, run the container in foreground mode.
Apr 04 21:55:51 rocksteady lxc-start[3144]: lxc-start: 5000: tools/lxc_start.c: main: 375 Additional information can be obtained by setting the --logfile and --logpriority options.
Apr 04 21:55:51 rocksteady systemd[1]: pve-container@5000.service: Control process exited, code=exited status=1
Apr 04 21:55:51 rocksteady systemd[1]: pve-container@5000.service: Killing process 3146 (lxc-start) with signal SIGKILL.
Apr 04 21:55:51 rocksteady systemd[1]: Failed to start PVE LXC Container: 5000.
Apr 04 21:55:51 rocksteady systemd[1]: pve-container@5000.service: Unit entered failed state.
Apr 04 21:55:51 rocksteady systemd[1]: pve-container@5000.service: Failed with result 'exit-code'.

# journalctl -xe

# journalctl -xe
Apr 04 22:25:18 rocksteady systemd[1]: Starting PVE LXC Container: 5000...
-- Subject: Unit pve-container@5000.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pve-container@5000.service has begun starting up.
Apr 04 22:25:18 rocksteady audit[5573]: AVC apparmor="DENIED" operation="mount" info="failed mntpnt match" error=-13 profile="/usr/bin/lxc-start" name="/" pid=5573 comm="lxc-start" flags="rw, rslave"
Apr 04 22:25:18 rocksteady lxc-start[5571]: lxc-start: 5000: lxccontainer.c: wait_on_daemonized_start: 751 No such file or directory - Failed to receive the container state
Apr 04 22:25:18 rocksteady lxc-start[5571]: lxc-start: 5000: tools/lxc_start.c: main: 371 The container failed to start.
Apr 04 22:25:18 rocksteady lxc-start[5571]: lxc-start: 5000: tools/lxc_start.c: main: 373 To get more details, run the container in foreground mode.
Apr 04 22:25:18 rocksteady lxc-start[5571]: lxc-start: 5000: tools/lxc_start.c: main: 375 Additional information can be obtained by setting the --logfile and --logpriority options.
Apr 04 22:25:18 rocksteady systemd[1]: pve-container@5000.service: Control process exited, code=exited status=1
Apr 04 22:25:18 rocksteady systemd[1]: pve-container@5000.service: Killing process 5573 (lxc-start) with signal SIGKILL.
Apr 04 22:25:18 rocksteady kernel: audit: type=1400 audit(1522895118.917:36): apparmor="DENIED" operation="mount" info="failed mntpnt match" error=-13 profile="/usr/bin/lxc-start" name="/" pid=5573 comm="lxc-start" flags="rw, rslave"
Apr 04 22:25:18 rocksteady systemd[1]: Failed to start PVE LXC Container: 5000.
-- Subject: Unit pve-container@5000.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pve-container@5000.service has failed.
--
-- The result is failed.
Apr 04 22:25:18 rocksteady systemd[1]: pve-container@5000.service: Unit entered failed state.
Apr 04 22:25:18 rocksteady systemd[1]: pve-container@5000.service: Failed with result 'exit-code'.

# lxc-start -n 5000 -F

# lxc-start -n 5000 -F
lxc-start: 5000: lxccontainer.c: do_lxcapi_start: 984 Permission denied - Failed to make / rslave at startup
lxc-start: 5000: tools/lxc_start.c: main: 371 The container failed to start.
lxc-start: 5000: tools/lxc_start.c: main: 375 Additional information can be obtained by setting the --logfile and --logpriority options.

 

[Flu]

I tried downgrading apparmor but I can not find it in the no-subscription repo

 

[Flu]

Fixed by adding the non subscription repo and doing:
# apt-get install lxc-pve

 

[Thomas Stone]

Fixed by adding the non subscription repo and doing:
# apt-get install lxc-pve

This happened to me too. I have subscriptions for two boxes but was doing some testing on VMs before an upgrade, so I didn't apply my key. Created two CTs and neither of them would start! I found your post and added the non-subscription repo and just did apt update apt dist-upgrade. Package lxc-pve was already installed.

After a reboot the CT would start.

I had installed from "proxmox-ve_5.1-3.iso" so I think the conclusion should be something is messed up with that iso and hopefully the devs will agree and update the iso they are distributing so that others won't have this bad initial experience.

 

[tom]

This happened to me too. I have subscriptions for two boxes but was doing some testing on VMs before an upgrade, so I didn't apply my key. Created two CTs and neither of them would start! I found your post and added the non-subscription repo and just did apt update apt dist-upgrade. Package lxc-pve was already installed.

After a reboot the CT would start.

I had installed from "proxmox-ve_5.1-3.iso" so I think the conclusion should be something is messed up with that iso and hopefully the devs will agree and update the iso they are distributing so that others won't have this bad initial experience.

You always need to install updates after installing from the ISO - its a one click tasks via GUI.

We do not re-create the ISO every day.

 

[Thomas Stone]

Hi Tom:

I'm a fan and subscriber to Proxmox and I want more people to be the same. But what you say seems to be incorrect. When a user installs Proxmox VE 5.x the system cannot be updated with one click or many clicks because it comes configured only for the enterprise apt repository. So trying to update from the GUI does not work. This requires some research and a trip to the terminal to edit /etc/apt/sources.list. Please correct me if I am mistaken.

And about the iso, wouldn't you test that the point at which you created the iso a basic function like running a CT would work? Are Flu and I incorrect about CTs not working in the 'frozen' state of the 5.1-3 iso?

 

[Flu]

A default install of 5.1-3.iso works fine, the issue is when you update the base OS without enabling the non-subscription repo. A package in the debian repo breaks a pve package that come from the ISO and you need to add the non-sub repo to update it. I am all for having the non-sub repo disabled by default, because I have had the non-sub repo break a system more that not having it. I would say if the system is in production use enterprise or pull packages from non-sub as needed.

 

[Thomas Stone]

I see. Thanks for clarifying that Flu. So, it would have worked (from the iso) had I not updated the system ... or if I had enabled the non-sub repo OR put in an enterprise key for the update.

 

[okoe]

Hello,

your discussion was very helpful for me. Facing the same issue, I added

# PVE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb http://download.proxmox.com/debian/pve stretch pve-no-subscription

to /etc/apt/sources.list

Then I utilized the update/upgrade function in the Web-UI. Afterwards I was able to star all containers again.

Thanks
Oliver