Update broke LXC

RXLegacy

New Member
Nov 4, 2019
2
0
1
26
I ran a proxmox update and Now i'm getting this error when trying to load my Fileserver container. I am now fully up to date to proxmox 6.0-9-6.0

Code:
Job for pve-container@101.service failed because the control process exited with error code.
See "systemctl status pve-container@101.service" and "journalctl -xe" for details.
TASK ERROR: command 'systemctl start pve-container@101' failed: exit code 1

Pve-container
Code:
root@Athena:~# systemctl status pve-container@101.service
● pve-container@101.service - PVE LXC Container: 101
   Loaded: loaded (/lib/systemd/system/pve-container@.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2019-11-04 09:37:07 CST; 2min 26s ago
     Docs: man:lxc-start
           man:lxc
           man:pct
  Process: 26358 ExecStart=/usr/bin/lxc-start -n 101 (code=exited, status=1/FAILURE)

Nov 04 09:37:07 Athena systemd[1]: Starting PVE LXC Container: 101...
Nov 04 09:37:07 Athena lxc-start[26358]: lxc-start: 101: lxccontainer.c: wait_on_daemonized_start: 856 No such file or directory - Fai
Nov 04 09:37:07 Athena lxc-start[26358]: lxc-start: 101: tools/lxc_start.c: main: 330 The container failed to start
Nov 04 09:37:07 Athena lxc-start[26358]: lxc-start: 101: tools/lxc_start.c: main: 333 To get more details, run the container in foregr
Nov 04 09:37:07 Athena lxc-start[26358]: lxc-start: 101: tools/lxc_start.c: main: 336 Additional information can be obtained by settin
Nov 04 09:37:07 Athena systemd[1]: pve-container@101.service: Control process exited, code=exited, status=1/FAILURE
Nov 04 09:37:07 Athena systemd[1]: pve-container@101.service: Failed with result 'exit-code'.
Nov 04 09:37:07 Athena systemd[1]: Failed to start PVE LXC Container: 101.
lines 1-16/16 (END)

Code:
oot@Athena:~# journalctl -xe
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- An ExecStart= process belonging to unit pve-container@101.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Nov 04 10:08:40 Athena systemd[1]: pve-container@101.service: Failed with result 'exit
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit pve-container@101.service has entered the 'failed' state with result 'exit
Nov 04 10:08:40 Athena pvedaemon[23842]: unable to get PID for CT 101 (not running?)
Nov 04 10:08:40 Athena systemd[1]: Failed to start PVE LXC Container: 101.
-- Subject: A start job for unit pve-container@101.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit pve-container@101.service has finished with a failure.
--
-- The job identifier is 58059 and the job result is failed.
Nov 04 10:08:40 Athena pvedaemon[31809]: command 'systemctl start pve-container@101' f
Nov 04 10:08:40 Athena pvedaemon[23842]: <root@pam> end task UPID:Athena:00007C41:003F
lines 5607-5630/5630 (END)
 
Code:
root@Athena:~# pveversion -v
proxmox-ve: 6.0-2 (running kernel: 5.0.21-3-pve)
pve-manager: 6.0-9 (running version: 6.0-9/508dcee0)
pve-kernel-5.0: 6.0-9
pve-kernel-helper: 6.0-9
pve-kernel-4.15: 5.4-9
pve-kernel-5.0.21-3-pve: 5.0.21-7
pve-kernel-4.15.18-21-pve: 4.15.18-48
pve-kernel-4.15.18-12-pve: 4.15.18-36
ceph-fuse: 12.2.12-pve1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-5
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.0-9
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-65
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-8
pve-cluster: 6.0-7
pve-container: 3.0-7
pve-docs: 6.0-7
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-7
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-3
pve-qemu-kvm: 4.0.1-3
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-9
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve1
root@Athena:~#

PCT Config
root@Athena:~# pct config 101 arch: amd64 cores: 5 hostname: FileServer memory: 2048 mp0: HDDStor:subvol-101-disk-1,mp=/Athena/HDDStor,size=8600G net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.1.1,hwaddr=CE:71:24:39:76:B2,ip=192.168.1.8/24,type=veth onboot: 1 ostype: debian rootfs: HDDStor:subvol-101-disk-0,size=8G swap: 512 root@Athena:~#

Troubles loading whole debug file so pics for now.
1572885369222.png
1572885386419.png
 
hi,

it's not completely visible on the screenshot, but i'm guessing it says unable to detect OS distribution.

can you try: pct mount CTID and check the content of /etc/debian_version file?
 
I'm having the exact same problem and indeed it says:

Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 104 lxc pre-start produced output: unable to detect OS distribution

I did a restore of a backup I had of the container and it started. Unfortunately, after a reboot of the proxmox-host the problem reappeared and it won't start.. with the same 'unable to detect OS distribution'.

My OS inside the LXC is ubuntu - but if i do pct mount CTID I get:
mounted CT 104 in '/var/lib/ lxc/104/rootfs'

and if I navigate to that folder with cd /var/lib/lxc/1004/rootfs all I see is an empty dev folder and no /etc/debian_version of some sort.

Any suggestions ?

Code:
pve-version -v
proxmox-ve: 6.0-2 (running kernel: 5.0.21-4-pve)
pve-manager: 6.0-11 (running version: 6.0-11/2140ef37)
pve-kernel-helper: 6.0-11
pve-kernel-5.0: 6.0-10
pve-kernel-5.0.21-4-pve: 5.0.21-9
pve-kernel-5.0.21-3-pve: 5.0.21-7
pve-kernel-5.0.21-2-pve: 5.0.21-7
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-3
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-7
libpve-guest-common-perl: 3.0-2
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.0-9
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-8
pve-cluster: 6.0-7
pve-container: 3.0-10
pve-docs: 6.0-8
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-7
pve-firmware: 3.0-4
pve-ha-manager: 3.0-2
pve-i18n: 2.0-3
pve-qemu-kvm: 4.0.1-5
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-13
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2

pct config:
arch: amd64
cores: 2
hostname: pihole
memory: 2048
net0: name=eth0,bridge=vmbr0,gw=192.168.1.1,hwaddr=FE:08:73:E3:A6:C2,ip=192.168.1.2/24,type=veth
onboot: 1
ostype: ubuntu
rootfs: ssd2:subvol-104-disk-1
swap: 512
unprivileged: 1

lxc debug log:
lxc-start 104 20191113103725.163 INFO     confile - confile.c:set_config_idmaps:2003 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 104 20191113103725.163 INFO     confile - confile.c:set_config_idmaps:2003 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 104 20191113103725.164 INFO     lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start 104 20191113103725.164 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
lxc-start 104 20191113103725.164 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 104 20191113103725.164 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "[all]"
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "kexec_load errno 1"
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "open_by_handle_at errno 1"
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "init_module errno 1"
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "finit_module errno 1"
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "delete_module errno 1"
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start 104 20191113103725.166 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start 104 20191113103725.166 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start 104 20191113103725.166 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start 104 20191113103725.166 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "keyctl errno 38"
lxc-start 104 20191113103725.166 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for keyctl action 327718(errno)
lxc-start 104 20191113103725.166 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for keyctl action 327718(errno)
lxc-start 104 20191113103725.166 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for keyctl action 327718(errno)
lxc-start 104 20191113103725.166 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for keyctl action 327718(errno)
lxc-start 104 20191113103725.166 INFO     seccomp - seccomp.c:parse_config_v2:1008 - Merging compat seccomp contexts into main context
lxc-start 104 20191113103725.166 INFO     conf - conf.c:run_script_argv:372 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "104", config section "lxc"
lxc-start 104 20191113103726.548 DEBUG    conf - conf.c:run_buffer:340 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 104 lxc pre-start produced output: unable to detect OS distribution

lxc-start 104 20191113103726.624 ERROR    conf - conf.c:run_buffer:352 - Script exited with status 2
lxc-start 104 20191113103726.721 ERROR    start - start.c:lxc_init:897 - Failed to run lxc.hook.pre-start for container "104"
lxc-start 104 20191113103726.817 ERROR    start - start.c:__lxc_start:2032 - Failed to initialize container "104"
 
My OS inside the LXC is ubuntu - but if i do pct mount CTID I get:
mounted CT 104 in '/var/lib/ lxc/104/rootfs'

and if I navigate to that folder with cd /var/lib/lxc/1004/rootfs all I see is an empty dev folder and no /etc/debian_version of some sort.

Any suggestions ?

is your zfs dataset mounted?
 
It is mounted now - since I did a restore currently so I can at least run my containers. Inside the file /etc/debian_version I see:
Code:
buster/sid
 
try starting the container
 
I think I found the problem. The containers where using a ZFS dataset which was not mounted. When I manually mounted them - my containers started as normal.

However, why my ZFS dataset is not automatically mounted is unknown to me. But that is something for another topic :)
 
However, why my ZFS dataset is not automatically mounted is unknown to me. But that is something for another topic :)

can you post your /etc/pve/storage.cfg ?
 
Yes:

Code:
dir: local
        path /var/lib/vz
        content iso,vztmpl,backup

zfspool: local-zfs
        pool rpool/data
        content images,rootdir
        sparse 1

dir: BackupWD
        path /BackupWD
        content vztmpl,iso,backup,rootdir,snippets,images
        maxfiles 50
        shared 0

zfspool: tank
        pool tank
        content rootdir,images
        sparse 0

zfspool: ssd2
        pool ssd2
        content rootdir,images
        nodes proxmox

The container(s) are on ssd2 as sub-vol containers
 
hi,

for each pool run: zpool set cachefile=/etc/zfs/zpool.cache POOLNAME.
then update initramfs: update-initramfs -u -k all.
then reboot.


sometimes the zfs cachefile can get corrupted/outdated, and therefore zfs-import-cache.service doesn't import pools that aren't present the cachefile.
you can also try enabling zfs-import-scan.service, which searches and imports pools via device scanning (but is slower).
 
I did update the zfs cache-file as you instructed on all my pools and that fixed it - ssd2 now got mounted during boot and containers would automatically start again.

Thanks for your support, much appreciated!
 
I did update the zfs cache-file as you instructed on all my pools and that fixed it - ssd2 now got mounted during boot and containers would automatically start again.

Thanks for your support, much appreciated!

awesome, glad to be of help!

@RXLegacy , i guess you're having the same or a very similar issue with @jelzinga. could you try the solution i posted?
 
Hi, I've got the same problem and I've tried your solutions, of course.

root@vhost:~# cat /tmp/lxc-ID.log
lxc-start 110 20191219091326.797 INFO lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:789 - Processing "reject_force_umount # comment this to allow umount -f; not recommended"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:789 - Processing "[all]"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:789 - Processing "kexec_load errno 1"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:789 - Processing "open_by_handle_at errno 1"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:789 - Processing "init_module errno 1"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:789 - Processing "finit_module errno 1"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:789 - Processing "delete_module errno 1"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:1008 - Merging compat seccomp contexts into main context
lxc-start 110 20191219091326.797 INFO conf - conf.c:run_script_argv:372 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "110", config section "lxc"
lxc-start 110 20191219091327.882 DEBUG conf - conf.c:run_buffer:340 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 110 lxc pre-start produced output: unable to detect OS distribution

lxc-start 110 20191219091327.940 ERROR conf - conf.c:run_buffer:352 - Script exited with status 2
lxc-start 110 20191219091327.940 ERROR start - start.c:lxc_init:897 - Failed to run lxc.hook.pre-start for container "110"
lxc-start 110 20191219091327.941 ERROR start - start.c:__lxc_start:2032 - Failed to initialize container "110"

I can't mount by subvolumens:
root@vhost:~# zfs mount rpool/vm_pool/subvol-110-disk-0
cannot mount '/rpool/vm_pool/subvol-110-disk-0': directory is not empty
root@vhost:~# rm -r /rpool/vm_pool/subvol-110-disk-0/dev/
root@vhost:~# zfs mount rpool/vm_pool/subvol-110-disk-0

After deleting the dev-folder and mounting the subvol manually, I can start my LXC. But this will only work until next reboot.

Have you got any idea, why ZFS isn't mounting the subvls automatically?

Thanks!
nigi
 
  • Like
Reactions: PmUserZFS
Hi, I've got the same problem and I've tried your solutions, of course.

root@vhost:~# cat /tmp/lxc-ID.log
lxc-start 110 20191219091326.797 INFO lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:789 - Processing "reject_force_umount # comment this to allow umount -f; not recommended"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:789 - Processing "[all]"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:789 - Processing "kexec_load errno 1"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:789 - Processing "open_by_handle_at errno 1"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:789 - Processing "init_module errno 1"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:789 - Processing "finit_module errno 1"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:789 - Processing "delete_module errno 1"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:parse_config_v2:1008 - Merging compat seccomp contexts into main context
lxc-start 110 20191219091326.797 INFO conf - conf.c:run_script_argv:372 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "110", config section "lxc"
lxc-start 110 20191219091327.882 DEBUG conf - conf.c:run_buffer:340 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 110 lxc pre-start produced output: unable to detect OS distribution

lxc-start 110 20191219091327.940 ERROR conf - conf.c:run_buffer:352 - Script exited with status 2
lxc-start 110 20191219091327.940 ERROR start - start.c:lxc_init:897 - Failed to run lxc.hook.pre-start for container "110"
lxc-start 110 20191219091327.941 ERROR start - start.c:__lxc_start:2032 - Failed to initialize container "110"

I can't mount by subvolumens:
root@vhost:~# zfs mount rpool/vm_pool/subvol-110-disk-0
cannot mount '/rpool/vm_pool/subvol-110-disk-0': directory is not empty
root@vhost:~# rm -r /rpool/vm_pool/subvol-110-disk-0/dev/
root@vhost:~# zfs mount rpool/vm_pool/subvol-110-disk-0

After deleting the dev-folder and mounting the subvol manually, I can start my LXC. But this will only work until next reboot.

Have you got any idea, why ZFS isn't mounting the subvls automatically?

Thanks!
nigi

I have the same issues with CT.
 
Is there still no solution for this topic?
I can't believe we all have the same problem since the last major update but nobody can't tell us how to solve it.
 
Is there still no solution for this topic?
As far as I see the one issue where enough information was provided (complete debug-log and description that only the /dev folder is present) was resolved (corrupt zfs cache file for the pool where the containers were stored)

Check if this is your issue as well - then update the cache file and run `update-initramfs -u -k all` and reboot as described by @oguz
If you have a different issue - please open a new Thread

I hope this helps!
 
  • Like
Reactions: oguz
Hi Stoiko,

thank you for your support! I've tried this steps a second time, just to be safe making everything right. As far as I see I've got a very similar (or the same) problem, but can't solve it with the steps mentioned above.
This is what I've tried:

root@vhost:~# mv /etc/zfs/zpool.cache /etc/zfs/zpool.cache_20200302

root@vhost:~# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
rpool 5,44T 2,93T 2,51T - - 37% 53% 1.00x ONLINE -

root@vhost:~# zpool set cachefile=/etc/zfs/zpool.cache rpool

root@vhost:~# update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-5.3.18-2-pve
Running hook script 'zz-pve-efiboot'..
Re-executing '/etc/kernel/postinst.d/zz-pve-efiboot' in new private mount namespace..
No /etc/kernel/pve-efiboot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.3.18-1-pve
Running hook script 'zz-pve-efiboot'..
Re-executing '/etc/kernel/postinst.d/zz-pve-efiboot' in new private mount namespace..
No /etc/kernel/pve-efiboot-uuids found, skipping ESP sync.

root@vhost:~# reboot now

Maybe we can try to localize my problem? I'm not a ZFS-expert, so I would be very grateful if you can give me some hints.


root@vhost:~# pveversion -v
proxmox-ve: 6.1-2 (running kernel: 5.3.18-2-pve)
pve-manager: 6.1-7 (running version: 6.1-7/13e58d5e)
pve-kernel-5.3: 6.1-5
pve-kernel-helper: 6.1-5
pve-kernel-5.3.18-2-pve: 5.3.18-2
pve-kernel-5.3.18-1-pve: 5.3.18-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.14-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-12
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-4
libpve-storage-perl: 6.1-4
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-19
pve-docs: 6.1-6
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-10
pve-firmware: 3.0-5
pve-ha-manager: 3.0-8
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-3
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-6
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

root@vhost:~# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content iso,vztmpl,backup

zfspool: local-zfs
pool rpool/data
content rootdir,images
sparse 1

dir: backup
path /rpool/backup
content backup
maxfiles 2
shared 0

zfspool: vm_pool
pool rpool/vm_pool
content rootdir,images
sparse 0

dir: vm_pool-dir
path /rpool/vm_pool
content vztmpl,rootdir,images
shared 0

dir: timecapsule_backup
path /media/timecapsule
content backup
maxfiles 2
shared 0

dir: ext-usb3
path /media/ext-usb3
content backup
maxfiles 5
shared 0

Thanks
nigi
 
check the journal from the last boot - for messages from container starts - and messages from ZFS/Zpool - these should give us a hint where the problem is
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!