Can't start LXC after reboot

wit4r7

Member
Sep 22, 2018
31
1
8
32
Hello everyone,

I'm new to PVE and tryed today my first lxc (InfluxDB, Grafana). After everything worked, I shut down the server. When I just restarted the server, I couldn't start both containers anymore. With the hint that Systemctl terminated with error 1.

The containers are ubuntu 18.04 bionic.

Here are (hopefully) some useful logs:
Code:
root@pve:~# systemctl status pve-container@107.service
● pve-container@107.service - PVE LXC Container: 107
   Loaded: loaded (/lib/systemd/system/pve-container@.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2018-10-16 20:02:58 CEST; 5min ago
     Docs: man:lxc-start
           man:lxc
           man:pct
  Process: 8472 ExecStart=/usr/bin/lxc-start -n 107 (code=exited, status=1/FAILURE)

Oct 16 20:02:57 pve systemd[1]: Starting PVE LXC Container: 107...
Oct 16 20:02:58 pve systemd[1]: pve-container@107.service: Control process exited, code=exited status=1
Oct 16 20:02:58 pve systemd[1]: Failed to start PVE LXC Container: 107.
Oct 16 20:02:58 pve systemd[1]: pve-container@107.service: Unit entered failed state.
Oct 16 20:02:58 pve systemd[1]: pve-container@107.service: Failed with result 'exit-code'.
Code:
root@pve:~# journalctl -xe
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pvesr.service has finished starting up.
--
-- The start-up result is done.
Oct 16 20:05:00 pve systemd[1]: Starting Proxmox VE replication runner...
-- Subject: Unit pvesr.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pvesr.service has begun starting up.
Oct 16 20:05:01 pve systemd[1]: Started Proxmox VE replication runner.
-- Subject: Unit pvesr.service has finished start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pvesr.service has finished starting up.
--
-- The start-up result is done.
Oct 16 20:06:00 pve systemd[1]: Starting Proxmox VE replication runner...
-- Subject: Unit pvesr.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pvesr.service has begun starting up.
Oct 16 20:06:01 pve systemd[1]: Started Proxmox VE replication runner.
-- Subject: Unit pvesr.service has finished start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pvesr.service has finished starting up.
--
-- The start-up result is done.
Oct 16 20:07:00 pve systemd-logind[4057]: Creating /run/nologin, blocking further logins...
Oct 16 20:07:00 pve systemd[1]: Starting Proxmox VE replication runner...
-- Subject: Unit pvesr.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pvesr.service has begun starting up.
Oct 16 20:07:00 pve systemd[1]: Started Proxmox VE replication runner.
-- Subject: Unit pvesr.service has finished start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pvesr.service has finished starting up.
--
-- The start-up result is done.
Oct 16 20:08:00 pve systemd[1]: Starting Proxmox VE replication runner...
-- Subject: Unit pvesr.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pvesr.service has begun starting up.
Oct 16 20:08:01 pve systemd[1]: Started Proxmox VE replication runner.
-- Subject: Unit pvesr.service has finished start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pvesr.service has finished starting up.

Code:
root@pve:~# pveversion -v
proxmox-ve: 5.2-2 (running kernel: 4.15.18-7-pve)
pve-manager: 5.2-9 (running version: 5.2-9/4b30e8f9)
pve-kernel-4.15: 5.2-10
pve-kernel-4.15.18-7-pve: 4.15.18-27
pve-kernel-4.15.17-1-pve: 4.15.17-9
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-40
libpve-guest-common-perl: 2.0-18
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-30
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-2
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-20
pve-cluster: 5.0-30
pve-container: 2.0-28
pve-docs: 5.2-8
pve-firewall: 3.0-14
pve-firmware: 2.0-5
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.2-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-36
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.11-pve1~bpo1
Code:
root@pve:~#lxc-start -n 101 -F -l DEBUG -o lxc.log

lxc.log:

  GNU nano 2.7.4                                                                                  File: lxc.log                                                                                             

lxc-start 101 20181016180342.174 ERROR    lxc_start - tools/lxc_start.c:main:288 - No container config specified
lxc-start 106 20181016180405.190 INFO     lsm - lsm/lsm.c:lsm_init:47 - LSM security driver AppArmor
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:757 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:do_resolve_add_rule:503 - Set seccomp rule to reject force umounts
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:934 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:do_resolve_add_rule:503 - Set seccomp rule to reject force umounts
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:943 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:do_resolve_add_rule:503 - Set seccomp rule to reject force umounts
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:953 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:do_resolve_add_rule:503 - Set seccomp rule to reject force umounts
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:963 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:757 - Processing "[all]"
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:757 - Processing "kexec_load errno 1"
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:934 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:943 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:953 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:963 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:757 - Processing "open_by_handle_at errno 1"
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:934 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:943 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:953 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:963 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:757 - Processing "init_module errno 1"
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:934 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:943 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:953 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:963 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:757 - Processing "finit_module errno 1"
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:934 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:943 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:953 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:963 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:757 - Processing "delete_module errno 1"
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:934 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:943 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:953 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:963 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start 106 20181016180405.191 INFO     seccomp - seccomp.c:parse_config_v2:967 - Merging compat seccomp contexts into main context
lxc-start 106 20181016180405.191 INFO     conf - conf.c:run_script_argv:374 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "106", config section "lxc"
lxc-start 106 20181016180405.682 DEBUG    conf - conf.c:run_buffer:344 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 106 lxc pre-start with output: unable to detect OS distribution

lxc-start 106 20181016180405.689 ERROR    conf - conf.c:run_buffer:353 - Script exited with status 2
lxc-start 106 20181016180405.689 ERROR    start - start.c:lxc_init:828 - Failed to run lxc.hook.pre-start for container "106"
lxc-start 106 20181016180405.689 ERROR    start - start.c:__lxc_start:1914 - Failed to initialize container "106"
lxc-start 106 20181016180405.690 ERROR    lxc_start - tools/lxc_start.c:main:330 - The container failed to start
lxc-start 106 20181016180405.690 ERROR    lxc_start - tools/lxc_start.c:main:336 - Additional information can be obtained by setting the --logfile and --logpriority options

Thank you very much for your help!

best regards from a pve beginner
wit4r7
 
Good Morning,

I dug a little deeper and found a mistake. The ZFS discs could not be mounted because the directories were not empty.
See code:
Code:
root@pve:/# zfs mount -a
cannot mount '/vm-storage': directory is not empty
cannot mount '/vm-storage/subvol-106-disk-0': directory is not empty
cannot mount '/vm-storage/subvol-107-disk-0': directory is not empty
cannot mount '/vm-storage/subvol-108-disk-0': directory is not empty

I then found the following command that currently solved the problem for me.
Code:
root@pve:/# zfs set overlay=on vm-storage
But I don't know yet how it behaves after the next restart.

best regards
 
Update: This doesn't fixed my problem.

After a reboot the lxc container have the same problem and can't start automaticly.
Error:
Code:
()
Job for pve-container@108.service failed because the control process exited with error code.
See "systemctl status pve-container@108.service" and "journalctl -xe" for details.
TASK ERROR: command 'systemctl start pve-container@108' failed: exit code 1
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!