[SOLVED] command 'systemctl start pve-container@104' failed: exit code 1

[DomF]

Hi,

My Proxmox server experienced a crash and on bootup one container and VM failed to start-up. From looking at the message for the VM it couldn't find the .iso of the ubuntu distribution which was specified. Then looking into my zfsiso directory, it was empty. I got the VM running again by removing the iso entry from my configuration file.

However, I am stuck in bringing up the turnkey mediaserver container. I tried to create a new container by downloading the template and running the wizard to Create CT but that failed to create the container.

On the existing CT I ran this command to capture debug information:
lxc-start 104 -o 104.log -l debug

lxc-start 104 20190312233429.915 INFO     lxccontainer - lxccontainer.c:do_lxcapi_start:984 - Set process title to [lxc monitor] /var/lib/lxc 104
lxc-start 104 20190312233429.916 INFO     lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start 104 20190312233429.916 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
lxc-start 104 20190312233429.916 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 104 20190312233429.916 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start 104 20190312233429.916 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 104 20190312233429.916 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start 104 20190312233429.916 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 104 20190312233429.916 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start 104 20190312233429.916 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 104 20190312233429.916 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start 104 20190312233429.916 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "[all]"
lxc-start 104 20190312233429.916 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "kexec_load errno 1"
lxc-start 104 20190312233429.916 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start 104 20190312233429.916 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start 104 20190312233429.916 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start 104 20190312233429.916 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start 104 20190312233429.916 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "open_by_handle_at errno 1"
lxc-start 104 20190312233429.916 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start 104 20190312233429.916 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start 104 20190312233429.916 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start 104 20190312233429.916 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start 104 20190312233429.916 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "init_module errno 1"
lxc-start 104 20190312233429.916 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start 104 20190312233429.916 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start 104 20190312233429.917 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start 104 20190312233429.917 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start 104 20190312233429.917 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "finit_module errno 1"
lxc-start 104 20190312233429.917 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start 104 20190312233429.917 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start 104 20190312233429.917 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start 104 20190312233429.917 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start 104 20190312233429.917 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "delete_module errno 1"
lxc-start 104 20190312233429.917 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start 104 20190312233429.917 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start 104 20190312233429.917 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start 104 20190312233429.917 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start 104 20190312233429.917 INFO     seccomp - seccomp.c:parse_config_v2:970 - Merging compat seccomp contexts into main context
lxc-start 104 20190312233429.917 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "104", config section "lxc"
lxc-start 104 20190312233430.300 DEBUG    conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 104 lxc pre-start with output: unable to detect OS distri
bution

lxc-start 104 20190312233430.305 ERROR    conf - conf.c:run_buffer:335 - Script exited with status 2
lxc-start 104 20190312233430.305 ERROR    start - start.c:lxc_init:861 - Failed to run lxc.hook.pre-start for container "104"
lxc-start 104 20190312233430.306 ERROR    start - start.c:__lxc_start:1944 - Failed to initialize container "104"
lxc-start 104 20190312233430.332 DEBUG    lxccontainer - lxccontainer.c:wait_on_daemonized_start:853 - First child 18813 exited
lxc-start 104 20190312233430.332 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:856 - No such file or directory - Failed to receive the container state
lxc-start 104 20190312233430.332 ERROR    lxc_start - tools/lxc_start.c:main:330 - The container failed to start
lxc-start 104 20190312233430.332 ERROR    lxc_start - tools/lxc_start.c:main:333 - To get more details, run the container in foreground mode
lxc-start 104 20190312233430.332 ERROR    lxc_start - tools/lxc_start.c:main:336 - Additional information can be obtained by setting the --logfile and --logpriority options

Then trying to create a new CT this was the message I got:

Using default stripesize 64.00 KiB.
Logical volume "vm-109-disk-0" created.
mke2fs 1.43.4 (31-Jan-2017)
Discarding device blocks: 4096/2621440 done
Creating filesystem with 2621440 4k blocks and 655360 inodes
Filesystem UUID: e3f9787f-032e-4b50-bb33-d24bd9aa8799
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: 0/80 done
Writing inode tables: 0/80 done
Creating journal (16384 blocks): done
Multiple mount protection is enabled with update interval 5 seconds.
Writing superblocks and filesystem accounting information: 0/80 done

extracting archive '/osdisks/iso/template/cache/debian-9-turnkey-mediaserver_15.0-1_amd64.tar.gz'
tar: ./var/spool/postfix/dev/urandom: Cannot mknod: Operation not permitted
tar: ./var/spool/postfix/dev/random: Cannot mknod: Operation not permitted
Total bytes read: 1119948800 (1.1GiB, 2.2MiB/s)
tar: Exiting with failure status due to previous errors
Logical volume "vm-109-disk-0" successfully removed
TASK ERROR: unable to create CT 109 - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar xpf - -z --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' -C /var/lib/lxc/109/rootfs --skip-old-files --anchored --exclude './dev/*'' failed: exit code 2

I updated my packages and I am running now:
proxmox-ve: 5.3-1 (running kernel: 4.15.18-11-pve)
pve-manager: 5.3-11 (running version: 5.3-11/d4907f84)

Any ideas what happened and how to fix?

Thanks.
Dominic

 

[dietmar]

What kind of OS Distribution is that mediaserver template?

 

[DomF]

It's one of the built-in templates in proxmox from turnkey linux. It is Debian based.

 

[dietmar]

OK, I see. Seems this container contains device nodes, so you need to create/run that 'privileged'. Please can you try?

 

[DomF]

I re-ran the create CT wizard with the "unprivileged container" unticked. It successfully created the mediaserver container.

 

[DomF]

I checked my notes from the past and it seems that previously my mediaserver container was created with the unpriviledged container unticked. But the wizard now has that as ticked by default. So, now I can create a new mediaserver container but will I be able to start-up the old server?

 

[oguz]

But the wizard now has that as ticked by default.

This was changed in a recent version, only the GUI wizard is affected.

So, now I can create a new mediaserver container but will I be able to start-up the old server?

In the startup log from your old container:

lxc-start 104 20190312233429.917 INFO conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "104", config section "lxc" lxc-start 104 20190312233430.300 DEBUG conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 104 lxc pre-start with output: unable to detect OS distri bution lxc-start 104 20190312233430.305 ERROR conf - conf.c:run_buffer:335 - Script exited with status 2 lxc-start 104 20190312233430.305 ERROR start - start.c:lxc_init:861 - Failed to run lxc.hook.pre-start for container "104" lxc-start 104 20190312233430.306 ERROR start - start.c:__lxc_start:1944 - Failed to initialize container "104"

"unable to detect OS distribution". This can happen because there's no corresponding file in container's /etc/ directory for auto-detecting the distribution. e.g. /etc/debian_version or /etc/arch-release and so on.

 

[DomF]

I think that container disk is corrupt as the content in very minimal compared to the new disk associated with the new mediaserver container.

root@pve:/osdisks/image/subvol-104-disk-1# ls
bignas dev nasdrive

(bignas, nasdrive) are mountpoints. And dev is empty.

Comparing with the new disk there are more directories and files within them:
root@pve:/osdisks/image/subvol-109-disk-0# ls
bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var

I will continue and setup the new container.

Thanks for your help.
Dominic

 

[willybong]

if useful, I had the same issue. I had a problem with my NIC interface. I changed the vmbr and the container started..