Containers fail to restart

[upnort]

With the recent Debian packages updates, we experienced containers failing to restart. We've seen this before. The solution is mentioned at the end of this thread:

find /sys/fs/cgroup/*/lxc/$CTID* -type d -print -delete
pct start $CTID

Does anybody know why this happens sporadically?

 

[oguz]

hi,

Does anybody know why this happens sporadically?

these directories are left behind when a container fails to start and doesn't clean up properly afterwards.

With the recent Debian packages updates

did you upgrade from old debian like wheezy? because newer debian versions use systemd instead of init, and upgrading from such versions seem to cause inconsistencies w.r.t. handling of services (which might be the cause of the restart problems).

 

[upnort]

and doesn't clean up properly afterwards

Yes, but why?

did you upgrade from old debian like wheezy?

No, this happens on fully updated systems.

 

[oguz]

hi,

Yes, but why?

it starts with a problem in the container which doesn't start properly.

basically, systemd kills the cleanup process before it can do its job, a bit racy.

No, this happens on fully updated systems.

you could start your container with debug logs[0] and we can take a look why it doesn't boot.

[0]: https://pve.proxmox.com/pve-docs/chapter-pct.html#_obtaining_debugging_logs

 

[upnort]

After updating libexpat1=2.2.0-2+deb9u3 and libpve-common-perl=5.0-55, we again got nailed by this problem.

I ran the following to try to restart one container:

lxc-start -n 160 -F -l DEBUG -o /tmp/lxc-160.log

The results:

lxc-start: 160: network.c: instantiate_veth: 106 Operation not permitted - Failed to create veth pair "veth160i0" and "veth4LU2F2"
lxc-start: 160: network.c: lxc_create_network_priv: 2462 Failed to create network device
lxc-start: 160: start.c: lxc_spawn: 1646 Failed to create the network
lxc-start: 160: start.c: __lxc_start: 1989 Failed to spawn container "160"
lxc-start: 160: tools/lxc_start.c: main: 330 The container failed to start
lxc-start: 160: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options

In the end I rebooted the host. Not a palatable solution.

What would have been a better solution?

 

[upnort]

At the moment, seems like the best route is manually shutdown containers before updating any pve package.

pct list; for ct in $(pct list | awk '{print $1}' | grep -v VMID); do echo $ct; pct shutdown $ct; done; pct list

 

[oguz]

hi,

I ran the following to try to restart one container:

lxc-start -n 160 -F -l DEBUG -o /tmp/lxc-160.log

The results:

this is not the complete result. the command you used writes the full log in /tmp/lxc-160.log. please run it again and send the contents of the file.

 

[upnort]

The host was rebooted. The log no longer exists.

Edit: Next time the log will be redirected to /var/tmp.