lxc start doesn't work

[Rikard]

Hi,

I can't start one of the lxc's any longer. I can mount it properly via pct mount 100.

# lxc-start -F -f /etc/pve/lxc/100.conf --name olle --logfile /tmp/lxc.log --logpriority TRACE
lxc-start: olle: utils.c: safe_mount: 1187 No such device - Failed to mount "file" onto "/"
                                                                                               lxc-start: olle: conf.c: mount_entry: 2045 No such device - Failed to mount "file" on "/"
                                                 lxc-start: olle: conf.c: lxc_setup: 3665 Failed to setup mount entries
                                                                                                                           lxc-start: olle: start.c: do_start: 1279 Failed to setup container "olle"
                                                                 lxc-start: olle: sync.c: __sync_wait: 62 An error occurred in another process (expected sequence number 5)
                                    lxc-start: olle: start.c: __lxc_start: 1989 Failed to spawn container "olle"
                                                                                                                        lxc-start: olle: tools/lxc_start.c: main: 330 The container failed to start
lxc-start: olle: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options
#

pveversion:

pveversion -verbose
proxmox-ve: 6.0-2 (running kernel: 5.0.21-2-pve)
pve-manager: 6.0-7 (running version: 6.0-7/28984024)
pve-kernel-5.0: 6.0-8
pve-kernel-helper: 6.0-8
pve-kernel-5.0.21-2-pve: 5.0.21-3
pve-kernel-5.0.18-1-pve: 5.0.18-3
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.12-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-4
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-8
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-65
lxcfs: 3.0.3-pve60
novnc-pve: 1.0.0-60
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-7
pve-cluster: 6.0-7
pve-container: 3.0-7
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-7
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-3
pve-qemu-kvm: 4.0.0-5
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-7
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.1-pve2

 

[wolfgang]

Hi,

you are sure the mount point of the Container is correct and it is mounted?
Because pct mount uses an alternative mount point.

What storage type does the image of container root use?

 

[Rikard]

@wolfgang thanks for your prompt reply.
mount shows all containers are mounted. subvol-100-disk-0 doesn't start. All other containers start as they should.

storage/containers/subvol-101-disk-1 on /storage/containers/subvol-101-disk-1 type zfs (rw,xattr,noacl)
storage/containers/subvol-100-disk-0 on /storage/containers/subvol-100-disk-0 type zfs (rw,xattr,posixacl)
storage/containers/subvol-102-disk-0 on /storage/containers/subvol-102-disk-0 type zfs (rw,xattr,posixacl)
storage/containers/subvol-103-disk-0 on /storage/containers/subvol-103-disk-0 type zfs (rw,xattr,posixacl)

Containers are stored/setup as pve containers. GUI info below:
Summary - Status
Enabled=YES
Active=YES
Content=Disk Image, Container
Type=ZFS
Usage=0.07% (17.38GB of 22.80TB)

Content - Name
subvol-101-disk-1 format/type: subvol/Container
subvol-100-disk-0 format/type: subvol/Container
subvol-102-disk-0 format/type: subvol/Container
subvol-103-disk-0 format/type: subvol/Container


# more /etc/pve/lxc/100.conf

arch: amd64
cores: 4
features: nesting=1
hostname: olle
memory: 32768
mp0: /mnt/pve/scratch/data,mp=/mnt/download,ro=0
mp1: /atastorage/media/f,mp=/mnt/download/download/Movies,ro=0
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.1.1,hwaddr=02:E1:77:ED:C3:C5,ip=192.168.1.147/24,type=veth
ostype: debian
rootfs: containers:subvol-100-disk-0,replicate=0,size=60G
swap: 512
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

 

[wolfgang]

And what about the two mount points?
Please run again with logpriority DEBUG and send the log.

 

[Rikard]

lxc-start -F -f /etc/pve/lxc/100.conf --name olle --logfile /tmp/lxc.log --logpriority DEBUG

lxc-start olle 20191001063142.695 INFO     lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start olle 20191001063142.695 DEBUG    terminal - terminal.c:lxc_terminal_peer_default:714 - Using terminal "/dev/tty" as proxy
lxc-start olle 20191001063142.695 DEBUG    terminal - terminal.c:lxc_terminal_signal_init:192 - Created signal fd 9
lxc-start olle 20191001063142.695 DEBUG    terminal - terminal.c:lxc_terminal_winsz:90 - Set window size to 139 columns and 37 rows
lxc-start olle 20191001063142.695 INFO     start - start.c:lxc_init:906 - Container "olle" is initialized
lxc-start olle 20191001063142.696 DEBUG    cgfsng - cgroups/cgfsng.c:cg_legacy_handle_cpuset_hierarchy:619 - "cgroup.clone_children" wa
s already set to "1"
lxc-start olle 20191001063142.696 INFO     cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1537 - The container uses "lxc/olle" as
cgroup
lxc-start olle 20191001063142.697 INFO     start - start.c:lxc_spawn:1707 - Cloned CLONE_NEWNS
lxc-start olle 20191001063142.697 INFO     start - start.c:lxc_spawn:1707 - Cloned CLONE_NEWPID
lxc-start olle 20191001063142.697 INFO     start - start.c:lxc_spawn:1707 - Cloned CLONE_NEWUTS
lxc-start olle 20191001063142.697 INFO     start - start.c:lxc_spawn:1707 - Cloned CLONE_NEWIPC
lxc-start olle 20191001063142.697 INFO     start - start.c:lxc_spawn:1707 - Cloned CLONE_NEWNET
lxc-start olle 20191001063142.697 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved mnt namespace via fd 14
lxc-start olle 20191001063142.697 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved pid namespace via fd 15
lxc-start olle 20191001063142.697 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved uts namespace via fd 16
lxc-start olle 20191001063142.697 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved ipc namespace via fd 17
lxc-start olle 20191001063142.697 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved net namespace via fd 18
lxc-start olle 20191001063142.698 DEBUG    start - start.c:lxc_spawn:1761 - Preserved net namespace via fd 10
lxc-start olle 20191001063142.698 INFO     start - start.c:do_start:1258 - Unshared CLONE_NEWCGROUP
lxc-start olle 20191001063142.699 INFO     conf - conf.c:mount_autodev:1144 - Preparing "/dev"
lxc-start olle 20191001063142.699 INFO     conf - conf.c:mount_autodev:1191 - Prepared "/dev"
lxc-start olle 20191001063142.699 INFO     conf - conf.c:lxc_fill_autodev:1235 - Populating "/dev"
lxc-start olle 20191001063142.699 DEBUG    conf - conf.c:lxc_fill_autodev:1250 - Created device node "/dev/full"
lxc-start olle 20191001063142.699 DEBUG    conf - conf.c:lxc_fill_autodev:1250 - Created device node "/dev/null"
lxc-start olle 20191001063142.699 DEBUG    conf - conf.c:lxc_fill_autodev:1250 - Created device node "/dev/random"
lxc-start olle 20191001063142.699 DEBUG    conf - conf.c:lxc_fill_autodev:1250 - Created device node "/dev/tty"
lxc-start olle 20191001063142.699 DEBUG    conf - conf.c:lxc_fill_autodev:1250 - Created device node "/dev/urandom"
lxc-start olle 20191001063142.699 DEBUG    conf - conf.c:lxc_fill_autodev:1250 - Created device node "/dev/zero"
lxc-start olle 20191001063142.699 INFO     conf - conf.c:lxc_fill_autodev:1312 - Populated "/dev"
lxc-start olle 20191001063142.701 ERROR    utils - utils.c:safe_mount:1187 - No such device - Failed to mount "file" onto "/"
lxc-start olle 20191001063142.701 ERROR    conf - conf.c:mount_entry:2045 - No such device - Failed to mount "file" on "/"
lxc-start olle 20191001063142.701 ERROR    conf - conf.c:lxc_setup:3665 - Failed to setup mount entries
lxc-start olle 20191001063142.701 ERROR    start - start.c:do_start:1279 - Failed to setup container "olle"
lxc-start olle 20191001063142.701 ERROR    sync - sync.c:__sync_wait:62 - An error occurred in another process (expected sequence numbe
r 5)
lxc-start olle 20191001063142.701 DEBUG    network - network.c:lxc_delete_network:3185 - Deleted network devices
lxc-start olle 20191001063142.701 ERROR    start - start.c:__lxc_start:1989 - Failed to spawn container "olle"
lxc-start olle 20191001063142.739 ERROR    lxc_start - tools/lxc_start.c:main:330 - The container failed to start
lxc-start olle 20191001063142.739 ERROR    lxc_start - tools/lxc_start.c:main:336 - Additional information can be obtained by setting t
he --logfile and --logpriority options

 

[Rikard]

@wolfgang or others, any feedback what might have happened? All worked fine with restarts and all previously, and then suddenly the lxc refuse to start.
All zpools indicate ONLINE (working).

 

[Rikard]

Is there anything that I can try or look at to make it boot up again? It's too important to re-install from a 10 day old snapshot. Even so, if I reinstall it might happen again so it would be nice if I can somehow try to find out what went wrong...

 

[wolfgang]

You can try the following.
Remove the two mp and try to start only with the root disk.
If this does not work make a backup from this configuration and restore the VM to a new VMID.