Container doesn not start after Power Loss

J

jahknem

New Member
Dec 2, 2019
15
0
1
28
I cannot start a container after a power loss.

This is the error I get when starting the container in the front:
root@pve:~# lxc-start -o lxc-start.log -lDEBUG -F -n 112 lxc-start: 112: conf.c: run_buffer: 352 Script exited with status 2 lxc-start: 112: start.c: lxc_init: 897 Failed to run lxc.hook.pre-start for container "112" lxc-start: 112: start.c: __lxc_start: 2032 Failed to initialize container "112" Segmentation fault

My Container Config:
arch: amd64 cores: 2 hostname: Radarr memory: 2048 mp0: /RAIDZ_1/shared,mp=/shared nameserver: 9.9.9.9 net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.2.33,hwaddr=12:D9:A8:99:0B:98,ip=192.168.2.32/24,type=veth ostype: debian rootfs: RAIDZ_1:subvol-112-disk-0,size=128G searchdomain: 1.1.1.1 swap: 2048 #lxc.cgroup.devices.allow: c 10:200 rwm #lxc.hook.autodev: sh -c "modprobe tun; cd ${LXC_ROOTFS_MOUNT}/dev; mkdir net; mknod net/tun c 10 200; chmod 0666 net/tun"

Also the Containers which cannot be started also have access to the tun device as well as to a host directory.

Can someone help me?
 
Hi,

if you run LXC in debug mode the debug log would be helpful.
please send the lxc-start.log
 
Of course:
Bash: 
lxc-start 113 20200127210901.936 ERROR    conf - conf.c:run_buffer:352 - Script exited with status 2
lxc-start 113 20200127210901.936 ERROR    start - start.c:lxc_init:897 - Failed to run lxc.hook.pre-start for container "113"
lxc-start 113 20200127210901.936 ERROR    start - start.c:__lxc_start:2032 - Failed to initialize container "113"
lxc-start 112 20200127211509.192 INFO     lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "[all]"
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "kexec_load errno 1"
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "open_by_handle_at errno 1"
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "init_module errno 1"
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "finit_module errno 1"
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "delete_module errno 1"
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start 112 20200127211509.192 INFO     seccomp - seccomp.c:parse_config_v2:1008 - Merging compat seccomp contexts into main context
lxc-start 112 20200127211509.192 INFO     conf - conf.c:run_script_argv:372 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "112", config section "lxc"
lxc-start 112 20200127211509.514 DEBUG    conf - conf.c:run_buffer:340 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 112 lxc pre-start produced output: unable to detect OS distribution

lxc-start 112 20200127211509.520 ERROR    conf - conf.c:run_buffer:352 - Script exited with status 2
lxc-start 112 20200127211509.520 ERROR    start - start.c:lxc_init:897 - Failed to run lxc.hook.pre-start for container "112"
lxc-start 112 20200127211509.520 ERROR    start - start.c:__lxc_start:2032 - Failed to initialize container "112"
 
hi,

Code: 
lxc-start 112 20200127211509.514 DEBUG    conf - conf.c:run_buffer:340 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 112 lxc pre-start produced output: unable to detect OS distribution

your rootfs is probably not mounted (your subvol).

check if the zfs pool is mounted.

also see this post[0]

[0]: https://forum.proxmox.com/threads/update-broke-lxc.59776/#post-277303
 
Nope, the subvolume is already mounted... tried that first :/
root@pve:~# zfs mount -O RAIDZ_1/subvol-112-disk-0
cannot mount 'RAIDZ_1/subvol-112-disk-0': filesystem already mounted
Click to expand...

Also after doing what you advised in the post you linked the situation sadly did not change:
root@pve:~# update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-5.3.13-1-pve
Running hook script 'zz-pve-efiboot'..
Re-executing '/etc/kernel/postinst.d/zz-pve-efiboot' in new private mount namespace..
No /etc/kernel/pve-efiboot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.0.21-5-pve
Running hook script 'zz-pve-efiboot'..
Re-executing '/etc/kernel/postinst.d/zz-pve-efiboot' in new private mount namespace..
No /etc/kernel/pve-efiboot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.0.21-2-pve
Running hook script 'zz-pve-efiboot'..
Re-executing '/etc/kernel/postinst.d/zz-pve-efiboot' in new private mount namespace..
No /etc/kernel/pve-efiboot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.0.15-1-pve
Running hook script 'zz-pve-efiboot'..
Re-executing '/etc/kernel/postinst.d/zz-pve-efiboot' in new private mount namespace..
No /etc/kernel/pve-efiboot-uuids found, skipping ESP sync.
Click to expand...
root@pve:~# pct start 112
Job for pve-container@112.service failed because the control process exited with error code.
See "systemctl status pve-container@112.service" and "journalctl -xe" for details.
command 'systemctl start pve-container@112' failed: exit code 1
root@pve:~# lxc-start -o lxc-start.log -lDEBUG -F -n 112
lxc-start: 112: conf.c: run_buffer: 352 Script exited with status 2
lxc-start: 112: start.c: lxc_init: 897 Failed to run lxc.hook.pre-start for container "112"
lxc-start: 112: start.c: __lxc_start: 2032 Failed to initialize container "112"
Segmentation fault
root@pve:~#
Click to expand...
 
are you able to mount the container with pct mount CTID? do you see /etc/debian_version in it?

does the container start if you disable the mp0?

since it's a power loss it's hard to tell if there was any damage done to the disk. that can possibly cause corruption. it'd be good to check the disk health with smartmontools and also the zpool health
 
I can mount the subvolume
root@pve:~# pct mount 112
mounted CT 112 in '/var/lib/lxc/112/rootfs'
Click to expand...

After commenting out mp0 and trying to start the container I still get the same error.

Also the short smartctl tests completed successfully and do not show any errors. Also no errors on zfs side
 
can you rerun the debug command with the mp commented?
 
Yes, still getting the same error :/
root@pve:~# lxc-start -o lxc-start.log -lDEBUG -F -n 112
lxc-start: 112: conf.c: run_buffer: 352 Script exited with status 2
lxc-start: 112: start.c: lxc_init: 897 Failed to run lxc.hook.pre-start for container "112"
lxc-start: 112: start.c: __lxc_start: 2032 Failed to initialize container "112"
Segmentation fault
Click to expand...
 
can you post the new debug log as well please
 
Sorry, I first did not find it as I had forgotten that I saved it in a different place..
root@pve:~# cat lxc-start.log
lxc-start 112 20200128140449.459 INFO lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:789 - Processing "reject_force_umount # comment this to allow umount -f; not recommended"
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:789 - Processing "[all]"
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:789 - Processing "kexec_load errno 1"
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:789 - Processing "open_by_handle_at errno 1"
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:789 - Processing "init_module errno 1"
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:789 - Processing "finit_module errno 1"
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:789 - Processing "delete_module errno 1"
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start 112 20200128140449.459 INFO seccomp - seccomp.c:parse_config_v2:1008 - Merging compat seccomp contexts into main context
lxc-start 112 20200128140449.459 INFO conf - conf.c:run_script_argv:372 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "112", config section "lxc"
lxc-start 112 20200128140449.779 DEBUG conf - conf.c:run_buffer:340 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 112 lxc pre-start produced output: unable to detect OS distribution

lxc-start 112 20200128140449.786 ERROR conf - conf.c:run_buffer:352 - Script exited with status 2
lxc-start 112 20200128140449.786 ERROR start - start.c:lxc_init:897 - Failed to run lxc.hook.pre-start for container "112"
lxc-start 112 20200128140449.786 ERROR start - start.c:__lxc_start:2032 - Failed to initialize container "112"
Click to expand...
 
oguz said:
do you see /etc/debian_version in it?
Click to expand...
do you see that file in the container when you mount it?
what are the contents?
 
I had a backup a few days ago. I haven't got any new data so I decided to restore the backup instead of troubleshooting the problem.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back