[SOLVED] LXC container suddenly not starting

[Areckee]

Hi,
one of my LXC containers (with Docker on Debian) refuses to start since yesterday. I have some unprivileged and some privileged ones and only this one causes problems.
I get these error messages:

run_buffer: 322 Script exited with status 1
lxc_init: 844 Failed to run lxc.hook.pre-start for container "203"
__lxc_start: 2027 Failed to initialize container "203"
TASK ERROR: startup for container '203' failed

# lxc-start -n 203 -F -lDEBUG -o lxc-203.log
lxc-start: 203: ../src/lxc/conf.c: run_buffer: 322 Script exited with status 1
lxc-start: 203: ../src/lxc/start.c: lxc_init: 844 Failed to run lxc.hook.pre-start for container "203"
lxc-start: 203: ../src/lxc/start.c: __lxc_start: 2027 Failed to initialize container "203"
lxc-start: 203: ../src/lxc/tools/lxc_start.c: main: 306 The container failed to start
lxc-start: 203: ../src/lxc/tools/lxc_start.c: main: 311 Additional information can be obtained by setting the --logfile and --logpriority options

# cat lxc-203.log
lxc-start 203 20230807142635.422 INFO     lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
lxc-start 203 20230807142635.422 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "203", config section "lxc"
lxc-start 203 20230807142636.103 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 203 lxc pre-start produced output: closing file '/etc/network/interfaces.tmp.283215' failed - No space left on device

lxc-start 203 20230807142636.105 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 203 lxc pre-start produced output: error in setup task PVE::LXC::Setup::pre_start_hook

lxc-start 203 20230807142636.116 ERROR    conf - ../src/lxc/conf.c:run_buffer:322 - Script exited with status 1
lxc-start 203 20230807142636.116 ERROR    start - ../src/lxc/start.c:lxc_init:844 - Failed to run lxc.hook.pre-start for container "203"
lxc-start 203 20230807142636.116 ERROR    start - ../src/lxc/start.c:__lxc_start:2027 - Failed to initialize container "203"
lxc-start 203 20230807142636.116 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "203", config section "lxc"
lxc-start 203 20230807142636.617 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "203", config section "lxc"
lxc-start 203 20230807142637.178 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 203 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp0: not mounted.

lxc-start 203 20230807142637.178 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 203 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp0' failed: exit code 32

lxc-start 203 20230807142637.240 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:306 - The container failed to start
lxc-start 203 20230807142637.240 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:311 - Additional information can be obtained by setting the --logfile and --logpriority options

# lxc-start --name 203 --foreground
lxc-start: 203: ../src/lxc/conf.c: run_buffer: 322 Script exited with status 1
lxc-start: 203: ../src/lxc/start.c: lxc_init: 844 Failed to run lxc.hook.pre-start for container "203"
lxc-start: 203: ../src/lxc/start.c: __lxc_start: 2027 Failed to initialize container "203"
lxc-start: 203: ../src/lxc/tools/lxc_start.c: main: 306 The container failed to start
lxc-start: 203: ../src/lxc/tools/lxc_start.c: main: 311 Additional information can be obtained by setting the --logfile and --logpriority options

I've read many threads on this forum as well on reddit, applied the solutions (i.e. installed binutils, checked for Debian version in /etc and it's 11.7, # pct fsck 203) but none helped.

# pveversion -v
proxmox-ve: 7.4-1 (running kernel: 5.15.108-1-pve)
pve-manager: 7.4-16 (running version: 7.4-16/0f39f621)
pve-kernel-5.15: 7.4-4
pve-kernel-5.15.108-1-pve: 5.15.108-2
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-3
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.3-1
proxmox-backup-file-restore: 2.4.3-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.7.3
pve-cluster: 7.3-3
pve-container: 4.4-6
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+1
pve-firewall: 4.3-5
pve-firmware: 3.6-5
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.11-pve1

# pct config 203
arch: amd64
cores: 2
features: nesting=1
hostname: docker
memory: 2048
net0: name=eth0,bridge=vmbr0,firewall=1,gw=10.10.30.1,hwaddr=3A:1A:56:0B:F2:69,ip=10.10.30.203/24,tag=30,type=veth
onboot: 1
ostype: debian
rootfs: local:203/vm-203-disk-0.raw,size=8G
swap: 512

I'd very much appreciate your help.

 

[Chris]

pre-start produced output: closing file '/etc/network/interfaces.tmp.283215' failed - No space left on device

Hi,
looks like you ran out of storage space. You can mount the containers root filesystem with pct mount 203 and check with df -h && df -hi.

 

[Areckee]

Hi,
looks like you ran out of storage space. You can mount the containers root filesystem with pct mount 203 and check with df -h && df -hi.

You are absolutely right. I didn't see it coming Thank you for your help. You're doing great job. Keep it that way.

 

[jimklimov]

Thanks for the discussion here, helped me get on the right track with an Ubuntu 23.04 container `do-release-upgrade -d` bumped to (pre-release of) 23.10 - and it failed to boot.

In my case the issue logged in Proxmox UI was very unhelpful:

run_buffer: 321 Script exited with status 255
lxc_init: 847 Failed to run lxc.hook.pre-start for container "118"
__lxc_start: 2008 Failed to initialize container "118"
TASK ERROR: startup for container '118' failed

However the `lxc-start -n 118 -F -lDEBUG -o lxc-118.log` suggested above helped with getting a log file which stated clearly:

lxc-start 118 20231002100729.303 INFO     conf - ../src/lxc/conf.c:run_script_argv:337 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "118", config section "lxc"
lxc-start 118 20231002100729.607 DEBUG    conf - ../src/lxc/conf.c:run_buffer:310 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 118 lxc pre-start produced output: unsupported Ubuntu version '23.10'
lxc-start 118 20231002100729.615 ERROR    conf - ../src/lxc/conf.c:run_buffer:321 - Script exited with status 255

That led me on a grepping spree across host filesystem, with `/usr/share/perl5/PVE/LXC/Setup/Ubuntu.pm` in my case being the culprit - it starts with a list of known release IDs

Hope this helps someone else,
Jim Klimov

 

[cetjunior]

Thanks for the discussion here, helped me get on the right track with an Ubuntu 23.04 container `do-release-upgrade -d` bumped to (pre-release of) 23.10 - and it failed to boot.

In my case the issue logged in Proxmox UI was very unhelpful:

run_buffer: 321 Script exited with status 255
lxc_init: 847 Failed to run lxc.hook.pre-start for container "118"
__lxc_start: 2008 Failed to initialize container "118"
TASK ERROR: startup for container '118' failed

However the `lxc-start -n 118 -F -lDEBUG -o lxc-118.log` suggested above helped with getting a log file which stated clearly:

lxc-start 118 20231002100729.303 INFO     conf - ../src/lxc/conf.c:run_script_argv:337 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "118", config section "lxc"
lxc-start 118 20231002100729.607 DEBUG    conf - ../src/lxc/conf.c:run_buffer:310 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 118 lxc pre-start produced output: unsupported Ubuntu version '23.10'
lxc-start 118 20231002100729.615 ERROR    conf - ../src/lxc/conf.c:run_buffer:321 - Script exited with status 255

That led me on a grepping spree across host filesystem, with `/usr/share/perl5/PVE/LXC/Setup/Ubuntu.pm` in my case being the culprit - it starts with a list of known release IDs

Hope this helps someone else,
Jim Klimov

Hi @jimklimov!

Your answer gave me the way! I was testing upgrades (containers and VMs) in a new cluster, and got 2 CTs dead after running "do-release-upgrade -d" in Ubuntu containers to validate, but they don't boot again. Checking the file you gave, I confirmed that Ubuntu 24.04 isn't listed yet, but at least I can do a temporary fix.

EDIT: just manually added "numble" and version to the file did the trick, CTs are boot again.

Best regards.