Update broke LXC

[RXLegacy]

I ran a proxmox update and Now i'm getting this error when trying to load my Fileserver container. I am now fully up to date to proxmox 6.0-9-6.0

Job for pve-container@101.service failed because the control process exited with error code.
See "systemctl status pve-container@101.service" and "journalctl -xe" for details.
TASK ERROR: command 'systemctl start pve-container@101' failed: exit code 1

Pve-container

root@Athena:~# systemctl status pve-container@101.service
● pve-container@101.service - PVE LXC Container: 101
   Loaded: loaded (/lib/systemd/system/pve-container@.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2019-11-04 09:37:07 CST; 2min 26s ago
     Docs: man:lxc-start
           man:lxc
           man:pct
  Process: 26358 ExecStart=/usr/bin/lxc-start -n 101 (code=exited, status=1/FAILURE)

Nov 04 09:37:07 Athena systemd[1]: Starting PVE LXC Container: 101...
Nov 04 09:37:07 Athena lxc-start[26358]: lxc-start: 101: lxccontainer.c: wait_on_daemonized_start: 856 No such file or directory - Fai
Nov 04 09:37:07 Athena lxc-start[26358]: lxc-start: 101: tools/lxc_start.c: main: 330 The container failed to start
Nov 04 09:37:07 Athena lxc-start[26358]: lxc-start: 101: tools/lxc_start.c: main: 333 To get more details, run the container in foregr
Nov 04 09:37:07 Athena lxc-start[26358]: lxc-start: 101: tools/lxc_start.c: main: 336 Additional information can be obtained by settin
Nov 04 09:37:07 Athena systemd[1]: pve-container@101.service: Control process exited, code=exited, status=1/FAILURE
Nov 04 09:37:07 Athena systemd[1]: pve-container@101.service: Failed with result 'exit-code'.
Nov 04 09:37:07 Athena systemd[1]: Failed to start PVE LXC Container: 101.
lines 1-16/16 (END)

oot@Athena:~# journalctl -xe
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- An ExecStart= process belonging to unit pve-container@101.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Nov 04 10:08:40 Athena systemd[1]: pve-container@101.service: Failed with result 'exit
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit pve-container@101.service has entered the 'failed' state with result 'exit
Nov 04 10:08:40 Athena pvedaemon[23842]: unable to get PID for CT 101 (not running?)
Nov 04 10:08:40 Athena systemd[1]: Failed to start PVE LXC Container: 101.
-- Subject: A start job for unit pve-container@101.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit pve-container@101.service has finished with a failure.
--
-- The job identifier is 58059 and the job result is failed.
Nov 04 10:08:40 Athena pvedaemon[31809]: command 'systemctl start pve-container@101' f
Nov 04 10:08:40 Athena pvedaemon[23842]: <root@pam> end task UPID:Athena:00007C41:003F
lines 5607-5630/5630 (END)

 

[oguz]

hi,

can you please post your:
* pveversion -v
* pct config CTID
* lxc debug logs[0]

[0]: https://pve.proxmox.com/pve-docs/chapter-pct.html#_obtaining_debugging_logs

 

[RXLegacy]

root@Athena:~# pveversion -v
proxmox-ve: 6.0-2 (running kernel: 5.0.21-3-pve)
pve-manager: 6.0-9 (running version: 6.0-9/508dcee0)
pve-kernel-5.0: 6.0-9
pve-kernel-helper: 6.0-9
pve-kernel-4.15: 5.4-9
pve-kernel-5.0.21-3-pve: 5.0.21-7
pve-kernel-4.15.18-21-pve: 4.15.18-48
pve-kernel-4.15.18-12-pve: 4.15.18-36
ceph-fuse: 12.2.12-pve1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-5
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.0-9
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-65
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-8
pve-cluster: 6.0-7
pve-container: 3.0-7
pve-docs: 6.0-7
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-7
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-3
pve-qemu-kvm: 4.0.1-3
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-9
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve1
root@Athena:~#

PCT Config
root@Athena:~# pct config 101 arch: amd64 cores: 5 hostname: FileServer memory: 2048 mp0: HDDStor:subvol-101-disk-1,mp=/Athena/HDDStor,size=8600G net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.1.1,hwaddr=CE:71:24:39:76:B2,ip=192.168.1.8/24,type=veth onboot: 1 ostype: debian rootfs: HDDStor:subvol-101-disk-0,size=8G swap: 512 root@Athena:~#

Troubles loading whole debug file so pics for now.

 

[oguz]

hi,

it's not completely visible on the screenshot, but i'm guessing it says unable to detect OS distribution.

can you try: pct mount CTID and check the content of /etc/debian_version file?

 

[jelzinga]

I'm having the exact same problem and indeed it says:

Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 104 lxc pre-start produced output: unable to detect OS distribution

I did a restore of a backup I had of the container and it started. Unfortunately, after a reboot of the proxmox-host the problem reappeared and it won't start.. with the same 'unable to detect OS distribution'.

My OS inside the LXC is ubuntu - but if i do pct mount CTID I get:
mounted CT 104 in '/var/lib/ lxc/104/rootfs'

and if I navigate to that folder with cd /var/lib/lxc/1004/rootfs all I see is an empty dev folder and no /etc/debian_version of some sort.

Any suggestions ?

pve-version -v
proxmox-ve: 6.0-2 (running kernel: 5.0.21-4-pve)
pve-manager: 6.0-11 (running version: 6.0-11/2140ef37)
pve-kernel-helper: 6.0-11
pve-kernel-5.0: 6.0-10
pve-kernel-5.0.21-4-pve: 5.0.21-9
pve-kernel-5.0.21-3-pve: 5.0.21-7
pve-kernel-5.0.21-2-pve: 5.0.21-7
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-3
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-7
libpve-guest-common-perl: 3.0-2
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.0-9
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-8
pve-cluster: 6.0-7
pve-container: 3.0-10
pve-docs: 6.0-8
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-7
pve-firmware: 3.0-4
pve-ha-manager: 3.0-2
pve-i18n: 2.0-3
pve-qemu-kvm: 4.0.1-5
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-13
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2

pct config:
arch: amd64
cores: 2
hostname: pihole
memory: 2048
net0: name=eth0,bridge=vmbr0,gw=192.168.1.1,hwaddr=FE:08:73:E3:A6:C2,ip=192.168.1.2/24,type=veth
onboot: 1
ostype: ubuntu
rootfs: ssd2:subvol-104-disk-1
swap: 512
unprivileged: 1

lxc debug log:
lxc-start 104 20191113103725.163 INFO     confile - confile.c:set_config_idmaps:2003 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 104 20191113103725.163 INFO     confile - confile.c:set_config_idmaps:2003 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 104 20191113103725.164 INFO     lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start 104 20191113103725.164 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
lxc-start 104 20191113103725.164 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 104 20191113103725.164 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "[all]"
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "kexec_load errno 1"
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "open_by_handle_at errno 1"
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "init_module errno 1"
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "finit_module errno 1"
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "delete_module errno 1"
lxc-start 104 20191113103725.165 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start 104 20191113103725.166 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start 104 20191113103725.166 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start 104 20191113103725.166 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start 104 20191113103725.166 INFO     seccomp - seccomp.c:parse_config_v2:789 - Processing "keyctl errno 38"
lxc-start 104 20191113103725.166 INFO     seccomp - seccomp.c:parse_config_v2:975 - Added native rule for arch 0 for keyctl action 327718(errno)
lxc-start 104 20191113103725.166 INFO     seccomp - seccomp.c:parse_config_v2:984 - Added compat rule for arch 1073741827 for keyctl action 327718(errno)
lxc-start 104 20191113103725.166 INFO     seccomp - seccomp.c:parse_config_v2:994 - Added compat rule for arch 1073741886 for keyctl action 327718(errno)
lxc-start 104 20191113103725.166 INFO     seccomp - seccomp.c:parse_config_v2:1004 - Added native rule for arch -1073741762 for keyctl action 327718(errno)
lxc-start 104 20191113103725.166 INFO     seccomp - seccomp.c:parse_config_v2:1008 - Merging compat seccomp contexts into main context
lxc-start 104 20191113103725.166 INFO     conf - conf.c:run_script_argv:372 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "104", config section "lxc"
lxc-start 104 20191113103726.548 DEBUG    conf - conf.c:run_buffer:340 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 104 lxc pre-start produced output: unable to detect OS distribution

lxc-start 104 20191113103726.624 ERROR    conf - conf.c:run_buffer:352 - Script exited with status 2
lxc-start 104 20191113103726.721 ERROR    start - start.c:lxc_init:897 - Failed to run lxc.hook.pre-start for container "104"
lxc-start 104 20191113103726.817 ERROR    start - start.c:__lxc_start:2032 - Failed to initialize container "104"

 

[oguz]

My OS inside the LXC is ubuntu - but if i do pct mount CTID I get:
mounted CT 104 in '/var/lib/ lxc/104/rootfs'

and if I navigate to that folder with cd /var/lib/lxc/1004/rootfs all I see is an empty dev folder and no /etc/debian_version of some sort.

Any suggestions ?

is your zfs dataset mounted?

 

[jelzinga]

It is mounted now - since I did a restore currently so I can at least run my containers. Inside the file /etc/debian_version I see:

buster/sid

 

[oguz]

try starting the container

 

[jelzinga]

I think I found the problem. The containers where using a ZFS dataset which was not mounted. When I manually mounted them - my containers started as normal.

However, why my ZFS dataset is not automatically mounted is unknown to me. But that is something for another topic

 

[oguz]

However, why my ZFS dataset is not automatically mounted is unknown to me. But that is something for another topic

can you post your /etc/pve/storage.cfg ?

 

[jelzinga]

Yes:

dir: local
        path /var/lib/vz
        content iso,vztmpl,backup

zfspool: local-zfs
        pool rpool/data
        content images,rootdir
        sparse 1

dir: BackupWD
        path /BackupWD
        content vztmpl,iso,backup,rootdir,snippets,images
        maxfiles 50
        shared 0

zfspool: tank
        pool tank
        content rootdir,images
        sparse 0

zfspool: ssd2
        pool ssd2
        content rootdir,images
        nodes proxmox

The container(s) are on ssd2 as sub-vol containers

 

[oguz]

hi,

for each pool run: zpool set cachefile=/etc/zfs/zpool.cache POOLNAME.
then update initramfs: update-initramfs -u -k all.
then reboot.


sometimes the zfs cachefile can get corrupted/outdated, and therefore zfs-import-cache.service doesn't import pools that aren't present the cachefile.
you can also try enabling zfs-import-scan.service, which searches and imports pools via device scanning (but is slower).

 

[jelzinga]

I did update the zfs cache-file as you instructed on all my pools and that fixed it - ssd2 now got mounted during boot and containers would automatically start again.

Thanks for your support, much appreciated!

 

[oguz]

I did update the zfs cache-file as you instructed on all my pools and that fixed it - ssd2 now got mounted during boot and containers would automatically start again.

Thanks for your support, much appreciated!

awesome, glad to be of help!

@RXLegacy , i guess you're having the same or a very similar issue with @jelzinga. could you try the solution i posted?

 

[nigi]

Hi, I've got the same problem and I've tried your solutions, of course.

Spoiler: Debugging Log

root@vhost:~# cat /tmp/lxc-ID.log
lxc-start 110 20191219091326.797 INFO lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:789 - Processing "reject_force_umount # comment this to allow umount -f; not recommended"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:975 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:984 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:994 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:1004 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:789 - Processing "[all]"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:789 - Processing "kexec_load errno 1"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:975 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:984 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:994 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:1004 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:789 - Processing "open_by_handle_at errno 1"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:975 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:984 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:994 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:1004 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:789 - Processing "init_module errno 1"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:975 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:984 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:994 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:1004 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:789 - Processing "finit_module errno 1"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:975 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:984 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:994 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:1004 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:789 - Processing "delete_module errno 1"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:975 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:984 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:994 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:1004 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:1008 - Merging compat seccomp contexts into main context
lxc-start 110 20191219091326.797 INFO conf - conf.c:run_script_argv:372 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "110", config section "lxc"
lxc-start 110 20191219091327.882 DEBUG conf - conf.c:run_buffer:340 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 110 lxc pre-start produced output: unable to detect OS distribution

lxc-start 110 20191219091327.940 ERROR conf - conf.c:run_buffer:352 - Script exited with status 2
lxc-start 110 20191219091327.940 ERROR start - start.c:lxc_init:897 - Failed to run lxc.hook.pre-start for container "110"
lxc-start 110 20191219091327.941 ERROR start - start.c:__lxc_start:2032 - Failed to initialize container "110"

I can't mount by subvolumens:
root@vhost:~# zfs mount rpool/vm_pool/subvol-110-disk-0
cannot mount '/rpool/vm_pool/subvol-110-disk-0': directory is not empty
root@vhost:~# rm -r /rpool/vm_pool/subvol-110-disk-0/dev/
root@vhost:~# zfs mount rpool/vm_pool/subvol-110-disk-0

After deleting the dev-folder and mounting the subvol manually, I can start my LXC. But this will only work until next reboot.

Have you got any idea, why ZFS isn't mounting the subvls automatically?

Thanks!
nigi

 

[PmUserZFS]

Hi, I've got the same problem and I've tried your solutions, of course.

Spoiler: Debugging Log

root@vhost:~# cat /tmp/lxc-ID.log
lxc-start 110 20191219091326.797 INFO lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:789 - Processing "reject_force_umount # comment this to allow umount -f; not recommended"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:975 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:984 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:994 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.c:do_resolve_add_rule:535 - Set seccomp rule to reject force umounts
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:1004 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:789 - Processing "[all]"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:789 - Processing "kexec_load errno 1"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:975 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:984 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:994 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:1004 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:789 - Processing "open_by_handle_at errno 1"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:975 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:984 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:994 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:1004 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:789 - Processing "init_module errno 1"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:975 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:984 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:994 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:1004 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:789 - Processing "finit_module errno 1"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:975 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:984 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:994 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:1004 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:789 - Processing "delete_module errno 1"
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:975 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:984 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:994 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:1004 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start 110 20191219091326.797 INFO seccomp - seccomp.carse_config_v2:1008 - Merging compat seccomp contexts into main context
lxc-start 110 20191219091326.797 INFO conf - conf.c:run_script_argv:372 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "110", config section "lxc"
lxc-start 110 20191219091327.882 DEBUG conf - conf.c:run_buffer:340 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 110 lxc pre-start produced output: unable to detect OS distribution

lxc-start 110 20191219091327.940 ERROR conf - conf.c:run_buffer:352 - Script exited with status 2
lxc-start 110 20191219091327.940 ERROR start - start.c:lxc_init:897 - Failed to run lxc.hook.pre-start for container "110"
lxc-start 110 20191219091327.941 ERROR start - start.c:__lxc_start:2032 - Failed to initialize container "110"

I can't mount by subvolumens:
root@vhost:~# zfs mount rpool/vm_pool/subvol-110-disk-0
cannot mount '/rpool/vm_pool/subvol-110-disk-0': directory is not empty
root@vhost:~# rm -r /rpool/vm_pool/subvol-110-disk-0/dev/
root@vhost:~# zfs mount rpool/vm_pool/subvol-110-disk-0

After deleting the dev-folder and mounting the subvol manually, I can start my LXC. But this will only work until next reboot.

Have you got any idea, why ZFS isn't mounting the subvls automatically?

Thanks!
nigi

I have the same issues with CT.

 

[nigi]

Is there still no solution for this topic?
I can't believe we all have the same problem since the last major update but nobody can't tell us how to solve it.

 

[Stoiko Ivanov]

Is there still no solution for this topic?

As far as I see the one issue where enough information was provided (complete debug-log and description that only the /dev folder is present) was resolved (corrupt zfs cache file for the pool where the containers were stored)

Check if this is your issue as well - then update the cache file and run `update-initramfs -u -k all` and reboot as described by @oguz
If you have a different issue - please open a new Thread

I hope this helps!

 

[nigi]

Hi Stoiko,

thank you for your support! I've tried this steps a second time, just to be safe making everything right. As far as I see I've got a very similar (or the same) problem, but can't solve it with the steps mentioned above.
This is what I've tried:

Spoiler: tried solution

root@vhost:~# mv /etc/zfs/zpool.cache /etc/zfs/zpool.cache_20200302

root@vhost:~# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
rpool 5,44T 2,93T 2,51T - - 37% 53% 1.00x ONLINE -

root@vhost:~# zpool set cachefile=/etc/zfs/zpool.cache rpool

root@vhost:~# update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-5.3.18-2-pve
Running hook script 'zz-pve-efiboot'..
Re-executing '/etc/kernel/postinst.d/zz-pve-efiboot' in new private mount namespace..
No /etc/kernel/pve-efiboot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.3.18-1-pve
Running hook script 'zz-pve-efiboot'..
Re-executing '/etc/kernel/postinst.d/zz-pve-efiboot' in new private mount namespace..
No /etc/kernel/pve-efiboot-uuids found, skipping ESP sync.

root@vhost:~# reboot now

Maybe we can try to localize my problem? I'm not a ZFS-expert, so I would be very grateful if you can give me some hints.

Spoiler: pveversion -v

root@vhost:~# pveversion -v
proxmox-ve: 6.1-2 (running kernel: 5.3.18-2-pve)
pve-manager: 6.1-7 (running version: 6.1-7/13e58d5e)
pve-kernel-5.3: 6.1-5
pve-kernel-helper: 6.1-5
pve-kernel-5.3.18-2-pve: 5.3.18-2
pve-kernel-5.3.18-1-pve: 5.3.18-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.14-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-12
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-4
libpve-storage-perl: 6.1-4
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-19
pve-docs: 6.1-6
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-10
pve-firmware: 3.0-5
pve-ha-manager: 3.0-8
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-3
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-6
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

Spoiler: /etc/pve/storage.cfg

root@vhost:~# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content iso,vztmpl,backup

zfspool: local-zfs
pool rpool/data
content rootdir,images
sparse 1

dir: backup
path /rpool/backup
content backup
maxfiles 2
shared 0

zfspool: vm_pool
pool rpool/vm_pool
content rootdir,images
sparse 0

dir: vm_pool-dir
path /rpool/vm_pool
content vztmpl,rootdir,images
shared 0

dir: timecapsule_backup
path /media/timecapsule
content backup
maxfiles 2
shared 0

dir: ext-usb3
path /media/ext-usb3
content backup
maxfiles 5
shared 0

Thanks
nigi

 

[Stoiko Ivanov]

check the journal from the last boot - for messages from container starts - and messages from ZFS/Zpool - these should give us a hint where the problem is