[SOLVED] Failed to run lxc.hook.pre-start

it complains about " /usr/share/debmatic/bin/lxc-start-hook.sh" not existing - does it?
 
well, it's not shipped by PVE, so I can't tell you ;)
 
Fellows,

Seem to be having the same issue starting containers. This is generally after I had to forcefully stop the container, it fails to start until the node is restarted... Definately far from ideal...

When I try to start my container with the following:
Code: 
lxc-start -n 143 -F -lDEBUG -o lxc-143.log

I can see this in the resulting log:
Code: 
lxc-start 143 20240321094324.741 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 143 lxc pre-start produced output: failed to remove directory '/sys/fs/cgroup/lxc/143/ns/system.slice/systemd-machine-id-commit.service': Device or resource busy

Not exactly sure what that means, but I see many of the same symptoms that others face:
- lxc_init: 844 Failed to run lxc.hook.pre-start for container
- resolves on reboot of node

This seems to have started being an issue for me after upgrading to 8.1.4.
 
LaserGuidedCake said:
failed to remove directory '/sys/fs/cgroup/lxc/143/ns/system.slice/systemd-machine-id-commit.service': Device or resource busy
Click to expand...
Seems like some services are still left over inside the CT, which makes PVE/LXC fail to ensure a clean start again..

You mention you have to forcefully stop CTs, why's that? Seems like there's something off in general with those CTs and this here are only symptoms..
 
>forcefully stop CTs, why's that?
Let me give you the rundown, this was all over the course of 30 minutes:
1. created a new container, installed a database and webserver
2. rebooted it (via init 6)
3. then, the console became unresponsive and wouldn't reply on the network
4. then, I gave it a systemctl stop pve-container@143
5. took some time to complete, but, the container is listed as 'stopped'

>services are still left over inside the CT
seems to be - I believe when trolling through htop I can see some of what was there hanging out, not sure how to correctly kill off what is there though.

I do see other threads with roughly the same things going on:
doesn't seem to be an answer here - https://forum.proxmox.com/threads/lxc-cant-start-after-force-stop.84697/

EDIT: more information
attempted to migrate the CT to a different host - got this error:
2024-03-21 04:14:21 ERROR: removing local copy of 'local-lvm:vm-143-disk-0' failed - lvremove 'pve/vm-143-disk-0' error: Logical volume pve/vm-143-disk-0 contains a filesystem in use.

backup jobs are also kicking off on the host, but not going anywhre - not sure if its related:
INFO: trying to get global lock - waiting...
 
Last edited:
LaserGuidedCake said:
1. created a new container, installed a database and webserver
Click to expand...
What distro, which DB and webserver are running there?

What's the full config? pct config CTID

LaserGuidedCake said:
EDIT: more information
attempted to migrate the CT to a different host - got this error:
2024-03-21 04:14:21 ERROR: removing local copy of 'local-lvm:vm-143-disk-0' failed - lvremove 'pve/vm-143-disk-0' error: Logical volume pve/vm-143-disk-0 contains a filesystem in use.

backup jobs are also kicking off on the host, but not going anywhre - not sure if its related:
INFO: trying to get global lock - waiting...
Click to expand...
These are also follow-up symptoms/errors of the original issue that the CT is doing something odd and cannot stop correctly – potentially having processes in the uninterruptible D state (like e.g. can happen from a hung NFS mount).

Once that is fixed, it's rather likely that all those follow-up issues will be also gone.
 
t.lamprecht said:
What distro, which DB and webserver are running there?
Click to expand...
Literally an install of Zabbix, followed from their site - https://www.zabbix.com/download?zab...ents=server_frontend_agent&db=mysql&ws=apache

But specifically, Debian 12 as the distro, mariadb as the database, its client, and apache,

t.lamprecht said:
What's the full config? pct config CTID
Click to expand...
Well this is probably no good -


pct config 143
Configuration file 'nodes/pve/lxc/143.conf' does not exist

Not sure if that was from me tunnning arouund the console attempting fixes that other perscribed, but it would appear that file is gone...

This probably doesnt help, but here is the output from another healthy container that has (roughly) the same things going on inside it:
AGAIN: this is from a Healthy container, not what you had asked for:
pct config 142
arch: amd64
cores: 2
features: nesting=1
hostname: testdvc1
memory: 512
net0: name=eth0,bridge=vmbr0,firewall=1,gw=10.10.12.179,hwaddr=8E:D4:0C:67:CF:3E,ip=10.10.12.10/24,type=veth
ostype: debian
rootfs: local-lvm:vm-142-disk-0,size=8G
swap: 512
unprivileged: 1
 
LaserGuidedCake said:
Well this is probably no good -


pct config 143
Configuration file 'nodes/pve/lxc/143.conf' does not exist

Not sure if that was from me tunnning arouund the console attempting fixes that other perscribed, but it would appear that file is gone...
Click to expand...
And this isn't a cluster with multiple nodes? (that CLI tool only works for CTs that currently are on the same node IIRC).

Is the LXC config that gets generated on the start of a CT still around? cat /var/lib/lxc/143/config

And yeah checking for any process that still run in the CT after stopping it would be good too.
 
t.lamprecht said:
And this isn't a cluster with multiple nodes?
Click to expand...
It is indeed in a cluster - I tired moving it as an attempt to resolve the issues. Didn't work as the disk was 'in use'.

Figured I would tie out on the issue; had to reboot the node for an unrelated maintenance issue. It did take some time for the node to restart, as there were processes that were seill tryingt to do stuff. After the node came back, attempting to start the container renderd the following error in the console of the node:
1711266122484.png
So, it looks like my container is corrupt. Not a big deal as it was a fresh install.

I'll still be monitoring this thread, and if it happens again, I'll post more.

Thanks all!
 
Hello,
I'm migrating (or, well.... trying to) from one proxmox instance to another. I backup every LXC and VM on a NAS that both has access to and restore the backup. When trying to start the first, i get the following error:
Code: 
run_buffer: 571 Script exited with status 2
lxc_init: 845 Failed to run lxc.hook.pre-start for container "101"
__lxc_start: 2034 Failed to initialize container "101"
TASK ERROR: startup for container '101' failed

When searching for the first line, this forum post pops up as the first result. Can someone help me figure out what the error means?

lxc-start -n 101 -F -lDEBUG -o lxc-101.log
Code: 
root@pve:~# lxc-start -n 101 -F -lDEBUG -o lxc-101.log
lxc-start: 101: ../src/lxc/utils.c: run_buffer: 571 Script exited with status 2
lxc-start: 101: ../src/lxc/start.c: lxc_init: 845 Failed to run lxc.hook.pre-start for container "101"
lxc-start: 101: ../src/lxc/start.c: __lxc_start: 2034 Failed to initialize container "101"
lxc-start: 101: ../src/lxc/tools/lxc_start.c: lxc_start_main: 307 The container failed to start
lxc-start: 101: ../src/lxc/tools/lxc_start.c: lxc_start_main: 312 Additional information can be obtained by setting the --logfile and --logpriority options
root@pve:~#

pveversion -v
Code: 
root@pve:~# pveversion -v
proxmox-ve: 8.3.0 (running kernel: 6.8.12-9-pve)
pve-manager: 8.3.5 (running version: 8.3.5/dac3aa88bac3f300)
proxmox-kernel-helper: 8.1.1
proxmox-kernel-6.8: 6.8.12-9
proxmox-kernel-6.8.12-9-pve-signed: 6.8.12-9
proxmox-kernel-6.8.12-4-pve-signed: 6.8.12-4
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2+deb12u1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.1
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.0
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.10
libpve-cluster-perl: 8.0.10
libpve-common-perl: 8.2.9
libpve-guest-common-perl: 5.1.6
libpve-http-server-perl: 5.2.0
libpve-network-perl: 0.10.1
libpve-rs-perl: 0.9.2
libpve-storage-perl: 8.3.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.6.0-2
proxmox-backup-client: 3.3.4-1
proxmox-backup-file-restore: 3.3.4-1
proxmox-firewall: 0.6.0
proxmox-kernel-helper: 8.1.1
proxmox-mail-forward: 0.3.1
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.7
pve-cluster: 8.0.10
pve-container: 5.2.4
pve-docs: 8.3.1
pve-edk2-firmware: 4.2025.02-2
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.1.0
pve-firmware: 3.15-2
pve-ha-manager: 4.0.6
pve-i18n: 3.4.1
pve-qemu-kvm: 9.2.0-3
pve-xtermjs: 5.3.0-3
qemu-server: 8.3.8
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.7-pve2
root@pve:~#

pct config 101:
Code: 
root@pve:~# pct config 101
arch: amd64
cores: 4
features: mount=nfs,nesting=1
hostname: docker-reports
memory: 8192
mp0: /mnt/media,mp=/mnt/media
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:F7:4B:7A,ip=dhcp,type=veth
onboot: 1
ostype: debian
rootfs: cold-data:101/vm-101-disk-0.raw,size=220G
swap: 2048
tags: ip101
lxc.mount.auto: cgroup:rw
lxc.cgroup2.devices.allow: a
lxc.cap.drop:
lxc.cgroup2.devices.allow: c 188:* rwm
lxc.cgroup2.devices.allow: c 189:* rwm
lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.cgroup2.devices.allow: c 29:0 rwm
lxc.mount.entry: /dev/serial/by-id  dev/serial/by-id  none bind,optional,create=dir
lxc.mount.entry: /dev/ttyUSB0       dev/ttyUSB0       none bind,optional,create=file
lxc.mount.entry: /dev/ttyUSB1       dev/ttyUSB1       none bind,optional,create=file
lxc.mount.entry: /dev/ttyACM0       dev/ttyACM0       none bind,optional,create=file
lxc.mount.entry: /dev/ttyACM1       dev/ttyACM1       none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD128       dev/dri/renderD128       none bind,optional,create=file 0,0
root@pve:~#

cat /etc/pve/storage.cfg
Code: 
root@pve:~# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content backup,vztmpl,iso

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images

dir: cold-data
        path /mnt/cold-data
        content import,iso,backup,snippets,images,vztmpl,rootdir
        shared 0

cifs: tempbckp
        path /mnt/pve/tempbckp
        server 192.168.1.4
        share Backups
        content import,iso,snippets,backup,images,vztmpl,rootdir
        prune-backups keep-all=1
        username autouser

root@pve:~#

Thanks in advance
 
probably one of the custom `lxc.XXX` entries is wrong for your new system..
 
Hi Fabian, thanks for your reply.
Can you elaborate on what you mean by "custom lxc.XXX entries"?
How do i verify this?
Is there some kind of checklist for what should be verified, before one tries to migrate?
 
Hey everyone,

I created an account specifically to reply to this post. I came here a couple of times after Google searches for this exact problem - it seemed to happen out of the blue, sometimes more often in a row, sometimes after months without any problems.

My setup is really simple, I only run about 8-10 containers+VMs combined, nothing fancy.
All of the containers were installed with the Proxmox VE Helper-Scripts.

What seemed to be the problem for me was one (or more) of the containers running out of disk space. I found it by accident by looking at lvs on my node - one of the containers had 99+% of disk usage. It only had a 4G disk to begin with, so speaking in absolute terms basically nothing left. It was the exact one that became unresponsive out of the blue, had to be force stopped and wouldn't start again.

I grew the disk with pct resize <CTID> rootfs <new_size- after that it started right away.

Sometimes my issue would solve itself by rebooting my node. I'm guessing by maybe cleaning out caches or some other lvm magic and thereby getting back enough space for the container to be able to boot. At least that's my theory for now.

Hope this maybe solves it for some of you too.

Cheers
Mats
 
Last edited:
This hit me earlier today after upgrading a PVE to v9 and, after rebooting, upgrading its single LXC from Debian 12 to 13. Turned out that Trixie is already at 13.1 in /etc/debian_version and when mounting the LXC and setting its /etc/debian_version to 13.0, the LXC could be started again.

Look like a bug which @fabian might be interested in ...

lxc-start 100 20250907045559.697 DEBUG utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start produced output: unsupported debian version '13.1'
 
I'm running still proxmox 8.4.12 with Non production-ready repository and did yesterday an update for my containers to Debian 13.1 and none of them works anymore.

Code: 
lxc-start 100 20250907091825.509 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2273 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 100 20250907091825.509 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2273 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 100 20250907091825.509 INFO     lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
lxc-start 100 20250907091825.509 INFO     utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "100", config section "lxc"
lxc-start 100 20250907091825.899 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start produced output: unsupported debian version '13.1'

lxc-start 100 20250907091825.911 ERROR    utils - ../src/lxc/utils.c:run_buffer:571 - Script exited with status 25
lxc-start 100 20250907091825.911 ERROR    start - ../src/lxc/start.c:lxc_init:845 - Failed to run lxc.hook.pre-start for container "100"
lxc-start 100 20250907091825.911 ERROR    start - ../src/lxc/start.c:__lxc_start:2034 - Failed to initialize container "100"
lxc-start 100 20250907091825.911 INFO     utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "100", config section "lxc"
lxc-start 100 20250907091826.414 INFO     utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "100", config section "lxc"
lxc-start 100 20250907091826.783 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp0: not mounted.

lxc-start 100 20250907091826.783 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp0' failed: exit code 32

lxc-start 100 20250907091826.786 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp1: not mounted.

lxc-start 100 20250907091826.786 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp1' failed: exit code 32

lxc-start 100 20250907091826.819 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:lxc_start_main:307 - The container failed to start
lxc-start 100 20250907091826.819 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:lxc_start_main:312 - Additional information can be obtained by setting the --logfile and --logpriority options

Restoring a backup with debian 13.1 does not work.

Code: 
recovering backed-up configuration from 'ssdstorage:backup/vzdump-lxc-100-2025_09_07-01_00_05.tar.zst'
restoring 'ssdstorage:backup/vzdump-lxc-100-2025_09_07-01_00_05.tar.zst' now..
extracting archive '/mnt/ssdstorage/dump/vzdump-lxc-100-2025_09_07-01_00_05.tar.zst'
Total bytes read: 4304291840 (4.1GiB, 68MiB/s)
merging backed-up and given configuration..
TASK ERROR: unable to restore CT 100 - unsupported debian version '13.1'
 
Last edited:
Last edited:
  • Like
Reactions: ojaksch
You must log in or register to reply here.
Top Bottom