Container doesn't start : "Failed to run lxc.hook.pre-start"

[NaWaKaK]

Hello everyone,

One of my proxmox containers hasn't started up for a few days (probably due to a power cut).

Here's the error displayed at startup:

run_buffer: 321 Script exited with status 255
lxc_init: 847 Failed to run lxc.hook.pre-start for container "100"
__lxc_start: 2008 Failed to initialize container "100"
TASK ERROR: startup for container '100' failed

I also tried to start it with this command :

lxc-start -n 100 -F -lDEBUG -o lxc-100.log

And here is the output :

lxc-start 100 20230925074946.307 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2267 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 100 20230925074946.307 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2267 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 100 20230925074946.308 INFO     lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
lxc-start 100 20230925074946.308 INFO     conf - ../src/lxc/conf.c:run_script_argv:337 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "100", config section "lxc"
lxc-start 100 20230925074946.631 DEBUG    conf - ../src/lxc/conf.c:run_buffer:310 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start produced output: mount: /var/lib/lxc/.pve-staged-mounts/mp0: cannot mount; probably corrupted filesystem on /dev/loop0.

lxc-start 100 20230925074946.696 DEBUG    conf - ../src/lxc/conf.c:run_buffer:310 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start produced output: command 'mount /dev/loop0 /var/lib/lxc/.pve-staged-mounts/mp0' failed: exit code 32

lxc-start 100 20230925074946.703 ERROR    conf - ../src/lxc/conf.c:run_buffer:321 - Script exited with status 255
lxc-start 100 20230925074946.703 ERROR    start - ../src/lxc/start.c:lxc_init:847 - Failed to run lxc.hook.pre-start for container "100"
lxc-start 100 20230925074946.703 ERROR    start - ../src/lxc/start.c:__lxc_start:2008 - Failed to initialize container "100"
lxc-start 100 20230925074946.703 INFO     conf - ../src/lxc/conf.c:run_script_argv:337 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "100", config section "lxc"
lxc-start 100 20230925074946.954 DEBUG    conf - ../src/lxc/conf.c:run_buffer:310 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp0: not mounted.

lxc-start 100 20230925074946.954 DEBUG    conf - ../src/lxc/conf.c:run_buffer:310 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp0' failed: exit code 32

lxc-start 100 20230925074946.980 INFO     conf - ../src/lxc/conf.c:run_script_argv:337 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "100", config section "lxc"
lxc-start 100 20230925074947.481 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:306 - The container failed to start
lxc-start 100 20230925074947.481 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:311 - Additional information can be obtained by setting the --logfile and --logpriority options

It seems to be a problem with the storage mounting, so I tried to do it manually :

pct mount 100

And here is the error displayed :

mount: /var/lib/lxc/100/rootfs/data: cannot mount; probably corrupted filesystem on /dev/loop0.
mounting container failed
command 'mount /dev/loop0 /var/lib/lxc/100/rootfs//data/' failed: exit code 32

I tried to launch fsck :

pct fsck 100

Here is the output :

fsck from util-linux 2.36.1
/dev/mapper/pve-vm--100--disk--0: clean, 61507/3276800 files, 1232934/13107200 blocks

But that didn't solve the problem.

Now I'm not sure what to try to solve this problem, could you please help me ?

Thank you in advance for your answers.

 

[cheiss]

Hi,

lxc-start -n 100 -F -lDEBUG -o lxc-100.log

Only ever use pct start <vmid> -debug to start a LXC on Proxmox VE, don't do it directly using lxc-start.
Edit: Can you re-run the container with pct start 100 -debug and post the full output of that command?

Can you also please post the output of pveversion -v and pct config 100?

 

[leesteken]

Only ever use pct start <vmid> -debug to start a LXC on Proxmox VE, don't do it directly using lxc-start.

Off-topic: please update the Proxmox manual if people should not run this command to output the log to a file.

 

[cheiss]

Off-topic: please update the Proxmox manual if people should not run this command to output the log to a file.

Well, didn't know that is in there, can't remember the whole manual after all .
Then, I'll apologize and phrase it differently: There are certain times (debugging) cases where it can be useful, but most of the time (or at least, for more unexperienced users), it can have unintended effects (like e.g. not updating the LXC config or activating the storage properly).

 

[NaWaKaK]

Hi,


Only ever use pct start <vmid> -debug to start a LXC on Proxmox VE, don't do it directly using lxc-start.
Edit: Can you re-run the container with pct start 100 -debug and post the full output of that command?

Can you also please post the output of pveversion -v and pct config 100?

Hi cheiss,

Thanks a lot for your answer and for the information about lxc-start, I didn't know it can have unintended effects.

Here is the output of pct start :

root@pve:~# pct start 100 --debug
run_buffer: 321 Script exited with status 255
lxc_init: 847 Failed to run lxc.hook.pre-start for container "100"
__lxc_start: 2008 Failed to initialize container "100"
0 hostid 100000 range 65536
INFO     lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
INFO     conf - ../src/lxc/conf.c:run_script_argv:337 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "100", config section "lxc"
DEBUG    conf - ../src/lxc/conf.c:run_buffer:310 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start produced output: mount: /var/lib/lxc/.pve-staged-mounts/mp0: cannot mount; probably corrupted filesystem on /dev/loop0.

DEBUG    conf - ../src/lxc/conf.c:run_buffer:310 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start produced output: command 'mount /dev/loop0 /var/lib/lxc/.pve-staged-mounts/mp0' failed: exit code 32

ERROR    conf - ../src/lxc/conf.c:run_buffer:321 - Script exited with status 255
ERROR    start - ../src/lxc/start.c:lxc_init:847 - Failed to run lxc.hook.pre-start for container "100"
ERROR    start - ../src/lxc/start.c:__lxc_start:2008 - Failed to initialize container "100"
INFO     conf - ../src/lxc/conf.c:run_script_argv:337 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "100", config section "lxc"
startup for container '100' failed

You will also find below the outputs of the two requested commands :

root@pve:~# pveversion -v
proxmox-ve: 7.3-1 (running kernel: 5.15.74-1-pve)
pve-manager: 7.3-3 (running version: 7.3-3/c3928077)
pve-kernel-5.15: 7.2-14
pve-kernel-helper: 7.2-14
pve-kernel-5.15.74-1-pve: 5.15.74-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-8
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.2-12
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.3
pve-cluster: 7.3-1
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.5-6
pve-ha-manager: 3.5.1
pve-i18n: 2.8-1
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-1
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1

root@pve:~# pct config 100
arch: amd64
cores: 6
features: nesting=1
hostname: mycontainer
memory: 8192
mp0: data1:100/vm-100-disk-0.raw,mp=/data/,backup=1,size=3000G
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.1.1,hwaddr=36:9C:CB:41:DE:49,ip=192.168.1.85/24,type=veth
onboot: 1
ostype: ubuntu
rootfs: local-lvm:vm-100-disk-0,size=50G
swap: 4096
unprivileged: 1

Thanks a lot for your help.

 

[NaWaKaK]

Hello everybody,

I'm still stuck on this problem and have no idea how to solve it.

Does anyone have any ideas ?

Thanks.

 

[jokas0311]

Hello
I have the same problem, any idea what could cause that problem?

 

[jokas0311]

Sep 27 23:57:26 pve1 pvesh[1501]: Starting CT 101
Sep 27 23:57:26 pve1 pve-guests[1502]: <root@pam> starting task UPIDve1:000005DF:000004C7:651507B6:vzstart:101:root@pam:
Sep 27 23:57:26 pve1 pve-guests[1503]: starting CT 101: UPIDve1:000005DF:000004C7:651507B6:vzstart:101:root@pam:
Sep 27 23:57:27 pve1 systemd[1]: Created slice system-pve\x2dcontainer.slice - PVE LXC Container Slice.
Sep 27 23:57:27 pve1 systemd[1]: Started pve-container@101.service - PVE LXC Container: 101.
Sep 27 23:57:27 pve1 kernel: EXT4-fs (dm-6): mounted filesystem 4ecb2d0c-0ed5-4169-bb42-520a0e033edc with ordered data mode. Quota mode: none.
Sep 27 23:57:27 pve1 audit[1523]: AVC apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-101_</var/lib/lxc>" pid=1523 comm="apparmor_parser"
Sep 27 23:57:27 pve1 kernel: kauditd_printk_skb: 13 callbacks suppressed
Sep 27 23:57:27 pve1 kernel: audit: type=1400 audit(1695877047.767:25): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-101_</var/lib/lxc>" pid=1523 comm="apparmor_parser"
Sep 27 23:57:28 pve1 kernel: vmbr0: port 2(veth101i0) entered blocking state
Sep 27 23:57:28 pve1 kernel: vmbr0: port 2(veth101i0) entered disabled state
Sep 27 23:57:28 pve1 kernel: device veth101i0 entered promiscuous mode
Sep 27 23:57:28 pve1 kernel: eth0: renamed from vethmvy6J8
Sep 27 23:57:28 pve1 audit[1737]: AVC apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=1737 comm="(networkd)" srcname="/" flags="rw, rbind"
Sep 27 23:57:28 pve1 kernel: audit: type=1400 audit(1695877048.727:26): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=1737 comm="(networkd)" srcname="/" flags="rw, rbind"
Sep 27 23:57:28 pve1 audit[1751]: AVC apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=1751 comm="(d-logind)" srcname="/" flags="rw, rbind"
Sep 27 23:57:28 pve1 audit[1753]: AVC apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=1753 comm="(ionclean)" srcname="/" flags="rw, rbind"
Sep 27 23:57:28 pve1 kernel: audit: type=1400 audit(1695877048.751:27): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=1751 comm="(d-logind)" srcname="/" flags="rw, rbind"
Sep 27 23:57:28 pve1 kernel: audit: type=1400 audit(1695877048.751:28): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=1753 comm="(ionclean)" srcname="/" flags="rw, rbind"
Sep 27 23:57:28 pve1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Sep 27 23:57:28 pve1 kernel: vmbr0: port 2(veth101i0) entered blocking state
Sep 27 23:57:28 pve1 kernel: vmbr0: port 2(veth101i0) entered forwarding state
Sep 27 23:57:28 pve1 audit[1765]: AVC apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=1765 comm="(resolved)" srcname="/" flags="rw, rbind"
Sep 27 23:57:28 pve1 kernel: audit: type=1400 audit(1695877048.763:29): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=1765 comm="(resolved)" srcname="/" flags="rw, rbind"
Sep 27 23:57:28 pve1 audit[1778]: AVC apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=1778 comm="(pachectl)" srcname="/" flags="rw, rbind"
Sep 27 23:57:28 pve1 kernel: audit: type=1400 audit(1695877048.807:30): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=1778 comm="(pachectl)" srcname="/" flags="rw, rbind"
Sep 27 23:57:28 pve1 audit[1776]: AVC apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=1776 comm="(install)" srcname="/" flags="rw, rbind"
Sep 27 23:57:28 pve1 kernel: audit: type=1400 audit(1695877048.815:31): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=1776 comm="(install)" srcname="/" flags="rw, rbind"
Sep 27 23:57:28 pve1 audit[1798]: AVC apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=1798 comm="(sh)" srcname="/" flags="rw, rbind"
Sep 27 23:57:28 pve1 kernel: audit: type=1400 audit(1695877048.823:32): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=1798 comm="(sh)" srcname="/" flags="rw, rbind"
Sep 27 23:57:28 pve1 audit[1810]: AVC apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=1810 comm="(sh)" srcname="/" flags="rw, rbind"
Sep 27 23:57:28 pve1 kernel: audit: type=1400 audit(1695877048.835:33): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=1810 comm="(sh)" srcname="/" flags="rw, rbind"
Sep 27 23:57:28 pve1 audit[1856]: AVC apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=1856 comm="(mariadbd)" srcname="/" flags="rw, rbind"
Sep 27 23:57:28 pve1 kernel: audit: type=1400 audit(1695877048.863:34): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=1856 comm="(mariadbd)" srcname="/" flags="rw, rbind"
Sep 27 23:57:28 pve1 pve-guests[1501]: <root@pam> end task UPIDve1:000005DE:000004C6:651507B6:startall::root@pam: OK
Sep 27 23:57:28 pve1 systemd[1]: Finished pve-guests.service - PVE guests.
Sep 27 23:57:29 pve1 systemd[1]: Starting pvescheduler.service - Proxmox VE scheduler...
Sep 27 23:57:29 pve1 audit[2436]: AVC apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=2436 comm="(sh)" srcname="/" flags="rw, rbind"
Sep 27 23:57:29 pve1 audit[2443]: AVC apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=2443 comm="(an-start)" srcname="/" flags="rw, rbind"
Sep 27 23:57:29 pve1 pvescheduler[2739]: starting server
Sep 27 23:57:29 pve1 systemd[1]: Started pvescheduler.service - Proxmox VE scheduler.
Sep 27 23:57:29 pve1 systemd[1]: Reached target multi-user.target - Multi-User System.
Sep 27 23:57:29 pve1 systemd[1]: Reached target graphical.target - Graphical Interface.
Sep 27 23:57:29 pve1 systemd[1]: Starting systemd-update-utmp-runlevel.service - Record Runlevel Change in UTMP...
Sep 27 23:57:29 pve1 systemd[1]: systemd-update-utmp-runlevel.service: Deactivated successfully.
Sep 27 23:57:29 pve1 systemd[1]: Finished systemd-update-utmp-runlevel.service - Record Runlevel Change in UTMP.
Sep 27 23:57:29 pve1 systemd[1]: Startup finished in 5.113s (kernel) + 9.955s (userspace) = 15.068s.
Sep 27 23:57:31 pve1 audit[2762]: AVC apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=2762 comm="(dovecot)" srcname="/" flags="rw, rbind"
Sep 27 23:57:36 pve1 pvestatd[1453]: modified cpu set for lxc/101: 0-3
Sep 27 23:57:50 pve1 systemd[1]: systemd-fsckd.service: Deactivated successfully.
Sep 27 23:58:02 pve1 pvedaemon[3624]: shutdown CT 101: UPIDve1:00000E28:000012DB:651507DA:vzshutdown:101:root@pam:
Sep 27 23:58:02 pve1 pvedaemon[1485]: <root@pam> starting task UPIDve1:00000E28:000012DB:651507DA:vzshutdown:101:root@pam:
Sep 27 23:58:03 pve1 audit[3628]: AVC apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=3628 comm="(doveadm)" srcname="/" flags="rw, rbind"
Sep 27 23:58:03 pve1 kernel: kauditd_printk_skb: 3 callbacks suppressed
Sep 27 23:58:03 pve1 kernel: audit: type=1400 audit(1695877083.039:38): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=3628 comm="(doveadm)" srcname="/" flags="rw, rbind"
Sep 27 23:58:03 pve1 chronyd[1278]: Selected source 200.25.3.17 (2.debian.pool.ntp.org)
Sep 27 23:58:03 pve1 chronyd[1278]: System clock TAI offset set to 37 seconds
Sep 27 23:58:04 pve1 kernel: vmbr0: port 2(veth101i0) entered disabled state
Sep 27 23:58:04 pve1 kernel: device veth101i0 left promiscuous mode
Sep 27 23:58:04 pve1 kernel: vmbr0: port 2(veth101i0) entered disabled state
Sep 27 23:58:04 pve1 audit[3876]: AVC apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc-101_</var/lib/lxc>" pid=3876 comm="apparmor_parser"
Sep 27 23:58:04 pve1 kernel: audit: type=1400 audit(1695877084.672:39): apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc-101_</var/lib/lxc>" pid=3876 comm="apparmor_parser"