[SOLVED] Failed to run lxc.hook.pre-start

[fabian]

it complains about " /usr/share/debmatic/bin/lxc-start-hook.sh" not existing - does it?

 

[PrinzEisenherz1]

Correct, the file no longer exists.
But I don't understand why.

 

[fabian]

well, it's not shipped by PVE, so I can't tell you

 

[LaserGuidedCake]

Fellows,

Seem to be having the same issue starting containers. This is generally after I had to forcefully stop the container, it fails to start until the node is restarted... Definately far from ideal...

When I try to start my container with the following:

lxc-start -n 143 -F -lDEBUG -o lxc-143.log

I can see this in the resulting log:

lxc-start 143 20240321094324.741 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 143 lxc pre-start produced output: failed to remove directory '/sys/fs/cgroup/lxc/143/ns/system.slice/systemd-machine-id-commit.service': Device or resource busy

Not exactly sure what that means, but I see many of the same symptoms that others face:
- lxc_init: 844 Failed to run lxc.hook.pre-start for container
- resolves on reboot of node

This seems to have started being an issue for me after upgrading to 8.1.4.

 

[t.lamprecht]

failed to remove directory '/sys/fs/cgroup/lxc/143/ns/system.slice/systemd-machine-id-commit.service': Device or resource busy

Seems like some services are still left over inside the CT, which makes PVE/LXC fail to ensure a clean start again..

You mention you have to forcefully stop CTs, why's that? Seems like there's something off in general with those CTs and this here are only symptoms..

 

[LaserGuidedCake]

>forcefully stop CTs, why's that?
Let me give you the rundown, this was all over the course of 30 minutes:
1. created a new container, installed a database and webserver
2. rebooted it (via init 6)
3. then, the console became unresponsive and wouldn't reply on the network
4. then, I gave it a systemctl stop pve-container@143
5. took some time to complete, but, the container is listed as 'stopped'

>services are still left over inside the CT
seems to be - I believe when trolling through htop I can see some of what was there hanging out, not sure how to correctly kill off what is there though.

I do see other threads with roughly the same things going on:
doesn't seem to be an answer here - https://forum.proxmox.com/threads/lxc-cant-start-after-force-stop.84697/

EDIT: more information
attempted to migrate the CT to a different host - got this error:
2024-03-21 04:14:21 ERROR: removing local copy of 'local-lvm:vm-143-disk-0' failed - lvremove 'pve/vm-143-disk-0' error: Logical volume pve/vm-143-disk-0 contains a filesystem in use.

backup jobs are also kicking off on the host, but not going anywhre - not sure if its related:
INFO: trying to get global lock - waiting...

 

[t.lamprecht]

1. created a new container, installed a database and webserver

What distro, which DB and webserver are running there?

What's the full config? pct config CTID

EDIT: more information
attempted to migrate the CT to a different host - got this error:
2024-03-21 04:14:21 ERROR: removing local copy of 'local-lvm:vm-143-disk-0' failed - lvremove 'pve/vm-143-disk-0' error: Logical volume pve/vm-143-disk-0 contains a filesystem in use.

backup jobs are also kicking off on the host, but not going anywhre - not sure if its related:
INFO: trying to get global lock - waiting...

These are also follow-up symptoms/errors of the original issue that the CT is doing something odd and cannot stop correctly – potentially having processes in the uninterruptible D state (like e.g. can happen from a hung NFS mount).

Once that is fixed, it's rather likely that all those follow-up issues will be also gone.

 

[LaserGuidedCake]

What distro, which DB and webserver are running there?

Literally an install of Zabbix, followed from their site - https://www.zabbix.com/download?zab...ents=server_frontend_agent&db=mysql&ws=apache

But specifically, Debian 12 as the distro, mariadb as the database, its client, and apache,

What's the full config? pct config CTID

Well this is probably no good -


pct config 143
Configuration file 'nodes/pve/lxc/143.conf' does not exist

Not sure if that was from me tunnning arouund the console attempting fixes that other perscribed, but it would appear that file is gone...

This probably doesnt help, but here is the output from another healthy container that has (roughly) the same things going on inside it:
AGAIN: this is from a Healthy container, not what you had asked for:
pct config 142
arch: amd64
cores: 2
features: nesting=1
hostname: testdvc1
memory: 512
net0: name=eth0,bridge=vmbr0,firewall=1,gw=10.10.12.179,hwaddr=8E:D4:0C:67:CF:3E,ip=10.10.12.10/24,type=veth
ostype: debian
rootfs: local-lvm:vm-142-disk-0,size=8G
swap: 512
unprivileged: 1

 

[t.lamprecht]

Well this is probably no good -


pct config 143
Configuration file 'nodes/pve/lxc/143.conf' does not exist

Not sure if that was from me tunnning arouund the console attempting fixes that other perscribed, but it would appear that file is gone...

And this isn't a cluster with multiple nodes? (that CLI tool only works for CTs that currently are on the same node IIRC).

Is the LXC config that gets generated on the start of a CT still around? cat /var/lib/lxc/143/config

And yeah checking for any process that still run in the CT after stopping it would be good too.

 

[LaserGuidedCake]

And this isn't a cluster with multiple nodes?

It is indeed in a cluster - I tired moving it as an attempt to resolve the issues. Didn't work as the disk was 'in use'.

Figured I would tie out on the issue; had to reboot the node for an unrelated maintenance issue. It did take some time for the node to restart, as there were processes that were seill tryingt to do stuff. After the node came back, attempting to start the container renderd the following error in the console of the node:

So, it looks like my container is corrupt. Not a big deal as it was a fresh install.

I'll still be monitoring this thread, and if it happens again, I'll post more.

Thanks all!