LXC suddenly wont start after the node crashed

[scarletdevil19]

I suddenly lost internet connectivity with the "name not resolved" error when had me logging into my cluster to find one of my nodes down. This is the node that hosts my PI-Hole instance.

I tried just bringing it all back up and it did boot however the Pi-Hole LXC simply won't start

running pct start 105 --debug I get the following output

root@:~# pct start 105 --debug
run_buffer: 571 Script exited with status 1
lxc_init: 845 Failed to run lxc.hook.pre-start for container "105"
__lxc_start: 2047 Failed to initialize container "105"
d 0 hostid 100000 range 65536
INFO     lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
INFO     utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "105", config section "lxc"
DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 105 lxc pre-start produced output: mkdir /etc/systemd: Input/output error at /usr/share/perl5/PVE/LXC/Setup/Base.pm line 779.

DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 105 lxc pre-start produced output: error in setup task PVE::LXC::Setup::pre_start_hook

ERROR    utils - ../src/lxc/utils.c:run_buffer:571 - Script exited with status 1
ERROR    start - ../src/lxc/start.c:lxc_init:845 - Failed to run lxc.hook.pre-start for container "105"
ERROR    start - ../src/lxc/start.c:__lxc_start:2047 - Failed to initialize container "105"
INFO     utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "105", config section "lxc"
startup for container '105' failed

Any help would be appreciated

 

[Robert Obkircher]

What version is reported by apt show pve-container? The output of pct config 105 might also help.

Input/output error sounds like some problem with the file system. Is there anything suspicious in the system log?

 

[scarletdevil19]

What version is reported by apt show pve-container? The output of pct config 105 might also help.

Input/output error sounds like some problem with the file system. Is there anything suspicious in the system log?

Here's the output.

root@:~# apt show pve-container
Package: pve-container
Version: 6.1.1
Priority: optional
Section: perl
Maintainer: Proxmox Support Team <support@proxmox.com>
Installed-Size: 717 kB
Depends: binutils, file, isc-dhcp-client, libpve-access-control (>= 8.0.0~), libpve-c>
Recommends: libpve-network-perl (>= 0.8.3)
Download-Size: 156 kB
APT-Manual-Installed: yes
APT-Sources: http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 >
Description: Proxmox VE Container management tool
 Tool to manage Linux Containers on Proxmox VE.

root@:~#pct config 105
arch: amd64
cores: 4
features: nesting=1
hostname: Pi-Hole
memory: 1024
net0: name=eth0,bridge=vmbr0,gw=192.168.10.1,hwaddr=BC:24:11:F8:21:0E,ip=192.168.10.2/24,ip6=dhcp,type=veth
onboot: 0
ostype: debian
rootfs: local:105/vm-105-disk-0.raw,size=8G
startup: order=1
swap: 0
unprivileged: 1
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

The systlog complains about not being able to mount the filesystem but fsck shows that there are no issues

 

[Robert Obkircher]

Thank you for the details.

The systlog complains about not being able to mount the filesystem but fsckshows that there are no issues

Can you also provide the relevant system log messages? I assume pct mount <id> (and pct unmount <id>) don't work either?

Did you run fsck directly or via pct fsck <id>?