All lxc fails to launch after apt update and upgrade.

PmUserZFS

Well-Known Member
Feb 2, 2018
96
5
48
Task viewer: CT 107 - Start

OutputStatus

Stop
Job for pve-container@107.service failed because the control process exited with error code.
See "systemctl status pve-container@107.service" and "journalctl -xe" for details.
TASK ERROR: command 'systemctl start pve-container@107' failed: exit code 1


i have to restart a old backup of the lxcs then I can launch them.



Code:
root@pve:~# systemctl status pve-container@107.service
● pve-container@107.service - PVE LXC Container: 107
   Loaded: loaded (/lib/systemd/system/pve-container@.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sat 2019-12-07 22:10:04 CET; 3min 29s ago
     Docs: man:lxc-start
           man:lxc
           man:pct
  Process: 20482 ExecStart=/usr/bin/lxc-start -n 107 (code=exited, status=1/FAILURE)

Dec 07 22:10:04 pve systemd[1]: Starting PVE LXC Container: 107...
Dec 07 22:10:04 pve lxc-start[20482]: lxc-start: 107: lxccontainer.c: wait_on_daemonized_start: 865 No such file or directory - Failed to receive the container state
Dec 07 22:10:04 pve lxc-start[20482]: lxc-start: 107: tools/lxc_start.c: main: 329 The container failed to start
Dec 07 22:10:04 pve lxc-start[20482]: lxc-start: 107: tools/lxc_start.c: main: 332 To get more details, run the container in foreground mode
Dec 07 22:10:04 pve lxc-start[20482]: lxc-start: 107: tools/lxc_start.c: main: 335 Additional information can be obtained by setting the --logfile and --logpriority options
Dec 07 22:10:04 pve systemd[1]: pve-container@107.service: Control process exited, code=exited, status=1/FAILURE
Dec 07 22:10:04 pve systemd[1]: pve-container@107.service: Failed with result 'exit-code'.
Dec 07 22:10:04 pve systemd[1]: Failed to start PVE LXC Container: 107.


Code:
-- A start job for unit pve-container@107.service has begun execution.
--
-- The job identifier is 2440.
Dec 07 22:10:04 pve lxc-start[20482]: lxc-start: 107: lxccontainer.c: wait_on_daemonized_start: 865 No such file or directory - Failed to receive the container state
Dec 07 22:10:04 pve lxc-start[20482]: lxc-start: 107: tools/lxc_start.c: main: 329 The container failed to start
Dec 07 22:10:04 pve lxc-start[20482]: lxc-start: 107: tools/lxc_start.c: main: 332 To get more details, run the container in foreground mode
Dec 07 22:10:04 pve lxc-start[20482]: lxc-start: 107: tools/lxc_start.c: main: 335 Additional information can be obtained by setting the --logfile and --logpriority options
Dec 07 22:10:04 pve systemd[1]: pve-container@107.service: Control process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- An ExecStart= process belonging to unit pve-container@107.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Dec 07 22:10:04 pve systemd[1]: pve-container@107.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit pve-container@107.service has entered the 'failed' state with result 'exit-code'.
Dec 07 22:10:04 pve systemd[1]: Failed to start PVE LXC Container: 107.
-- Subject: A start job for unit pve-container@107.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit pve-container@107.service has finished with a failure.
--
-- The job identifier is 2440 and the job result is failed.
Dec 07 22:10:04 pve pvedaemon[20480]: command 'systemctl start pve-container@107' failed: exit code 1
Dec 07 22:10:04 pve kernel: lxc-start[20486]: segfault at 50 ip 00007f6295531f8b sp 00007ffd04d71270 error 4 in liblxc.so.1.6.0[7f62954d8000+8a000]
Dec 07 22:10:04 pve kernel: Code: 9b c0 ff ff 4d 85 ff 0f 85 82 02 00 00 66 90 48 8b 73 50 48 8b bb f8 00 00 00 e8 80 78 fa ff 4c 8b 74 24 10 48 89 de 4c 89 f7 <41> ff 56 50 4c 89 f7 48 89 de 41 ff 56 58 48 8b 83 f8 00 00 00 8b
Dec 07 22:10:04 pve pvedaemon[7584]: <root@pam> end task UPID:pve:00005000:00024F75:5DEC152C:vzstart:107:root@pam: command 'systemctl start pve-container@107' failed: exit code 1
 
Code:
root@pve:~# pveversion
pve-manager/6.1-3/37248ce6 (running kernel: 5.3.10-1-pve)
 
seems as if your container is started before the storage is available (seeing the
865 No such file or directory - Failed to receive the container state)

Have you checked if all your storage is avaiable to PVE ?
 
seems as if your container is started before the storage is available (seeing the
865 No such file or directory - Failed to receive the container state)

Have you checked if all your storage is avaiable to PVE ?

Hi, yes its a ZFS fs on a zpool that is availble.

Can one get more info about this output
ec 07 22:10:04 pve lxc-start[20482]: lxc-start: 107: lxccontainer.c: wait_on_daemonized_start: 865 No such file or directory - Failed to receive the container state
 
I had the same issue following an upgrade from 5.4 (i think?) to 6.1 today.

The issue was the dataset for the container was not mounted. When I tried to mount it manually, it turned out the directory wasn't empty (random "dev" device directory). I removed the directories existing in the mount point and remounted. Problem solved.

Hopefully that's your situation as well.
 
1576440533044.png

root@pve:~# lxc-start -n 107 -F -l DEBUG -o /tmp/lxc-107.log
lxc-start: 107: tools/lxc_start.c: main: 289 No container config specified

Code:
root@pve:~# cat /tmp/lxc-107.log
lxc-start 107 20191215200733.668 ERROR    lxc_start - tools/lxc_start.c:main:289 - No container config specified
lxc-start 107 20191215200925.759 ERROR    lxc_start - tools/lxc_start.c:main:289 - No container config specified
lxc-start 107 20191215200931.391 ERROR    lxc_start - tools/lxc_start.c:main:289 - No container config specified
lxc-start 107 20191215200936.995 ERROR    lxc_start - tools/lxc_start.c:main:289 - No container config specified
 
Last edited:
I have tired to restart backups of the 107 before launching it with debug options.

my nextcloud container which I really have to get up and running I havent touched at all.

root@pve:~# lxc-start -n 116 -l DEBUG -o /tmp/lxc-116.log
lxc-start: 116: lxccontainer.c: wait_on_daemonized_start: 865 No such file or directory - Failed to receive the container state
lxc-start: 116: tools/lxc_start.c: main: 329 The container failed to start
lxc-start: 116: tools/lxc_start.c: main: 332 To get more details, run the container in foreground mode
lxc-start: 116: tools/lxc_start.c: main: 335 Additional information can be obtained by setting the --logfile and --logpriority options

Code:
root@pve:~# cat /tmp/lxc-116.log
lxc-start 116 20191215201549.764 INFO     lxccontainer - lxccontainer.c:do_lxcapi_start:994 - Set process title to [lxc monitor] /var/lib/lxc 116
lxc-start 116 20191215201549.764 INFO     lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "reject_force_umount  # comment this to allow umount -f;  not recomme                          nded"
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "[all]"
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "kexec_load errno 1"
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "open_by_handle_at errno 1"
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errn                          o)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "init_module errno 1"
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "finit_module errno 1"
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "delete_module errno 1"
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start 116 20191215201549.765 INFO     seccomp - seccomp.c:parse_config_v2:1008 - Merging compat seccomp contexts into main context
lxc-start 116 20191215201549.765 INFO     conf - conf.c:run_script_argv:372 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "116", con                          fig section "lxc"
lxc-start 116 20191215201550.194 DEBUG    conf - conf.c:run_buffer:340 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 116 lxc pre-start produced output: u                          nable to detect OS distribution

lxc-start 116 20191215201550.201 ERROR    conf - conf.c:run_buffer:352 - Script exited with status 2
lxc-start 116 20191215201550.201 ERROR    start - start.c:lxc_init:897 - Failed to run lxc.hook.pre-start for container "116"
lxc-start 116 20191215201550.202 ERROR    start - start.c:__lxc_start:2032 - Failed to initialize container "116"
lxc-start 116 20191215201550.202 DEBUG    lxccontainer - lxccontainer.c:wait_on_daemonized_start:862 - First child 26563 exited
lxc-start 116 20191215201550.202 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:865 - No such file or directory - Failed to receive the container s                          tate
lxc-start 116 20191215201550.202 ERROR    lxc_start - tools/lxc_start.c:main:329 - The container failed to start
lxc-start 116 20191215201550.202 ERROR    lxc_start - tools/lxc_start.c:main:332 - To get more details, run the container in foreground mode
lxc-start 116 20191215201550.202 ERROR    lxc_start - tools/lxc_start.c:main:335 - Additional information can be obtained by setting the --logfile and --logpriority                           options
 
is your zfs volume mounted?