[SOLVED] LXC container failing to start

C

crider17

New Member
Mar 7, 2021
19
0
1
56
Hi everyone:

I was playing around with adding another drive and now I cannot get my container to spin up.

root@pve:~# lxc-start -n 100 -F -lDEBUG -o lxc-100.log
Code: 
lxc-start: 100: conf.c: run_buffer: 316 Script exited with status 1
lxc-start: 100: start.c: lxc_init: 816 Failed to run lxc.hook.pre-start for container "100"
lxc-start: 100: start.c: __lxc_start: 2007 Failed to initialize container "100"
lxc-start: 100: tools/lxc_start.c: main: 308 The container failed to start
lxc-start: 100: tools/lxc_start.c: main: 313 Additional information can be obtained by setting the --logfile and --logpriority options

root@pve:~# cat lxc-100.log
Code: 
lxc-start 100 20211022224405.397 INFO     lsm - lsm/lsm.c:lsm_init_static:40 - Initialized LSM security driver AppArmor
lxc-start 100 20211022224405.397 INFO     conf - conf.c:run_script_argv:332 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "100", config section "lxc"
lxc-start 100 20211022224405.985 DEBUG    conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start produced output: closing file '/etc/systemd/network/eth0.network.tmp.9856' failed - No space left on device

lxc-start 100 20211022224405.986 DEBUG    conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start produced output: error in setup task PVE::LXC::Setup::pre_start_hook

lxc-start 100 20211022224405.994 ERROR    conf - conf.c:run_buffer:316 - Script exited with status 1
lxc-start 100 20211022224405.994 ERROR    start - start.c:lxc_init:816 - Failed to run lxc.hook.pre-start for container "100"
lxc-start 100 20211022224405.994 ERROR    start - start.c:__lxc_start:2007 - Failed to initialize container "100"
lxc-start 100 20211022224405.994 INFO     conf - conf.c:run_script_argv:332 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "100", config section "lxc"
lxc-start 100 20211022224406.496 INFO     conf - conf.c:run_script_argv:332 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "100", config section "lxc"
lxc-start 100 20211022224407.255 DEBUG    conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp3: not mounted.

lxc-start 100 20211022224407.256 DEBUG    conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp3' failed: exit code 32

lxc-start 100 20211022224407.283 DEBUG    conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp4: not mounted.

lxc-start 100 20211022224407.284 DEBUG    conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp4' failed: exit code 32

lxc-start 100 20211022224407.311 DEBUG    conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp2: not mounted.

lxc-start 100 20211022224407.312 DEBUG    conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp2' failed: exit code 32

lxc-start 100 20211022224407.545 ERROR    lxc_start - tools/lxc_start.c:main:308 - The container failed to start
lxc-start 100 20211022224407.545 ERROR    lxc_start - tools/lxc_start.c:main:313 - Additional information can be obtained by setting the --logfile and --logpriority options
lxc-start 100 20211022224522.404 INFO     lsm - lsm/lsm.c:lsm_init_static:40 - Initialized LSM security driver AppArmor
lxc-start 100 20211022224522.404 INFO     conf - conf.c:run_script_argv:332 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "100", config section "lxc"
lxc-start 100 20211022224522.993 DEBUG    conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start produced output: closing file '/etc/systemd/network/eth0.network.tmp.10112' failed - No space left on device

lxc-start 100 20211022224522.994 DEBUG    conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start produced output: error in setup task PVE::LXC::Setup::pre_start_hook

lxc-start 100 20211022224523.288 ERROR    conf - conf.c:run_buffer:316 - Script exited with status 1
lxc-start 100 20211022224523.294 ERROR    start - start.c:lxc_init:816 - Failed to run lxc.hook.pre-start for container "100"
lxc-start 100 20211022224523.300 ERROR    start - start.c:__lxc_start:2007 - Failed to initialize container "100"
lxc-start 100 20211022224523.309 INFO     conf - conf.c:run_script_argv:332 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "100", config section "lxc"
lxc-start 100 20211022224523.504 INFO     conf - conf.c:run_script_argv:332 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "100", config section "lxc"
lxc-start 100 20211022224524.360 DEBUG    conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp3: not mounted.

lxc-start 100 20211022224524.361 DEBUG    conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp3' failed: exit code 32

lxc-start 100 20211022224524.388 DEBUG    conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp4: not mounted.

lxc-start 100 20211022224524.390 DEBUG    conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp4' failed: exit code 32

lxc-start 100 20211022224524.416 DEBUG    conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp2: not mounted.

lxc-start 100 20211022224524.418 DEBUG    conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp2' failed: exit code 32

lxc-start 100 20211022224524.646 ERROR    lxc_start - tools/lxc_start.c:main:308 - The container failed to start
lxc-start 100 20211022224524.647 ERROR    lxc_start - tools/lxc_start.c:main:313 - Additional information can be obtained by setting the --logfile and --logpriority options

This line stands out as a possible issue but I have no idea what I did to cause it our how to fix it:
xc-start 100 20211022224405.985 DEBUG conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start produced output: closing file '/etc/systemd/network/eth0.network.tmp.9856' failed - No space left on device

Any help/ideas would be appreciated.
 
Whats your df -h, vgdisplay and zfs list output on your PVE host?

The LXC fails because it can't mount your bind-mounts mp2, mp3 and mp4.
 
Thanks for your reply. My mp2 and mp3 were mapped to two USB drives I think. I included my config file below. I commented out those two mount points and unplugged my USB drives to simplify things. I'm not aware of having an mp4. I don't see it in my config. How do I find it and remove it? The VG "PURPLE" is the new drive I was trying to add.

I appreciate your help. I hope I did not mess this up too bad. It's been pretty solid for a long time.

root@pve:/# df -h
Code: 
Filesystem                  Size  Used Avail Use% Mounted on
udev                        7.8G     0  7.8G   0% /dev
tmpfs                       1.6G  1.1M  1.6G   1% /run
/dev/mapper/pve-root         55G  8.9G   43G  18% /
tmpfs                       7.8G   46M  7.8G   1% /dev/shm
tmpfs                       5.0M     0  5.0M   0% /run/lock
HGST10TB                    1.9T  128K  1.9T   1% /HGST10TB
HGST10TB/subvol-100-disk-0  9.0T  7.2T  1.9T  80% /HGST10TB/subvol-100-disk-0
/dev/fuse                   128M   20K  128M   1% /etc/pve
//192.168.1.17/Plex         5.2T  1.4T  3.8T  28% /mnt/pve/PLEX-ON-WIN7DVR
tmpfs                       1.6G     0  1.6G   0% /run/user/0

root@pve:/# vgdisplay
Code: 
 --- Volume group ---
  VG Name               PURPLE
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  5
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <5.46 TiB
  PE Size               4.00 MiB
  Total PE              1430511
  Alloc PE / Size       0 / 0   
  Free  PE / Size       1430511 / <5.46 TiB
  VG UUID               JehIHN-4uW7-K10B-GCcO-gFsS-3h67-kwqOWg
   
  --- Volume group ---
  VG Name               pve
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  906
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                7
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <931.01 GiB
  PE Size               4.00 MiB
  Total PE              238338
  Alloc PE / Size       140818 / 550.07 GiB
  Free  PE / Size       97520 / <380.94 GiB
  VG UUID               Q3OipC-dKFT-Vogz-VWXM-UBfW-wFMz-E090aZ

root@pve:/# zfs list
Code: 
NAME                         USED  AVAIL     REFER  MOUNTPOINT
HGST10TB                    7.13T  1.84T       96K  /HGST10TB
HGST10TB/subvol-100-disk-0  7.13T  1.84T     7.13T  /HGST10TB/subvol-100-disk-0

root@pve:/etc/pve/lxc# cat 100.conf
Code: 
#lxc.hook.autodev%3A bash -c 'for char_dev in $(find /sys/dev/char -regextype sed  -regex ".*/1%3A1" -o -regex ".*/4%3A\([3-9]\|[1-5][0-9]\|6[0-3]\)" -o -regex ".*/4%3A\(6[4-9]\|[7-9][0-9]\|1[0-9][0-9]\|2[0-4][0-9]\|25[0-5]\)" -o -regex ".*/10%3A200" -o -regex ".*/116%3A.*" -o -regex ".*/166%3A.*" -o -regex ".*/180%3A\([0-9]\|1[0-5]\)" -o -regex ".*/188%3A.*" -o -regex ".*/189%3A.*" -o -regex ".*/24[0-2]%3A.*"); do  dev="/dev/$(sed -n "/DEVNAME/ s/^.*=\(.*\)$/\1/p" ${char_dev}/uevent)";  mkdir -p $(dirname ${LXC_ROOTFS_MOUNT}${dev});  for link in $(udevadm info --query=property $dev | sed -n "s/DEVLINKS=//p"); do    mkdir -p ${LXC_ROOTFS_MOUNT}$(dirname $link);    cp -dpR $link ${LXC_ROOTFS_MOUNT}${link};  done;  cp -dpR $dev ${LXC_ROOTFS_MOUNT}${dev};done;'
#lxc.cgroup.devices.allow%3A c 189%3A* rwm
#lxc.mount.entry%3A /dev/bus/usb/003 dev/bus/usb/003 none bind,optional,create=file
#mp2: /dev/disk/by-uuid/1620-C3AD,mp=/mnt/usb8tb
#mp3: /dev/disk/by-uuid/b2b3eb19-d563-4645-85a6-ef6cd73303b3,mp=/mnt/backups
arch: amd64
cores: 3
hostname: ubuntu-plex
memory: 12288
mp0: HGST10TB:subvol-100-disk-0,mp=/media,size=10000G
mp1: local-lvm:vm-100-disk-1,mp=/transcode,size=15G
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.1.1,hwaddr=56:61:DC:8E:7D:2F,ip=192.168.1.19/24,type=veth
onboot: 1
ostype: ubuntu
rootfs: local-lvm:vm-100-disk-0,size=60G
startup: order=1
swap: 2048
lxc.autodev: 1
 
So why is it trying to mount mp2 and mp3 when they are commented out in my config? Is there somewhere else I need to remove them? Also, as I said, there should be no MP4.

I plugged back in my USB drives, uncommented the mp2 and mp3 mounts in the config and tried again. Now all it shows is the "mp4: not mounted" error.

So I think I need to find out where it's trying to load that mount point and remove it. Any help would be appreciated. Nothing in the GUI. Nothing in the config.
 
Last edited:
Hm, I don't see any of your storages running out of space explaining the "No space left on device".

Not sure where the "Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp4' failed: exit code 32" is then coming from if you don't got a mp4. But its also only a debug message and not error message so maybe the mountpoints aren't that important.
 
Does anyone know how I can find where my container is trying to load mp4 and remove it? Thanks!
 
I re-added the mp4 in hopes that I would be able to start the container. No good. I'm really grasping at straws here. Can anyone point me in the right direction as to why I am unable to mount mp4?

Code: 
lxc-start 100 20211023200146.396 INFO     lsm - lsm/lsm.c:lsm_init_static:40 - Initialized LSM security driver AppArmor
lxc-start 100 20211023200146.396 INFO     conf - conf.c:run_script_argv:332 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "100", config section "lxc"
lxc-start 100 20211023200147.146 DEBUG    conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start produced output: failed to get device path

lxc-start 100 20211023200147.154 ERROR    conf - conf.c:run_buffer:316 - Script exited with status 2
lxc-start 100 20211023200147.154 ERROR    start - start.c:lxc_init:816 - Failed to run lxc.hook.pre-start for container "100"
lxc-start 100 20211023200147.154 ERROR    start - start.c:__lxc_start:2007 - Failed to initialize container "100"
lxc-start 100 20211023200147.155 INFO     conf - conf.c:run_script_argv:332 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "100", config section "lxc"
lxc-start 100 20211023200147.656 INFO     conf - conf.c:run_script_argv:332 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "100", config section "lxc"
lxc-start 100 20211023200148.185 DEBUG    conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp4: not mounted.

lxc-start 100 20211023200148.186 DEBUG    conf - conf.c:run_buffer:305 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp4' failed: exit code 32

lxc-start 100 20211023200148.212 ERROR    lxc_start - tools/lxc_start.c:main:308 - The container failed to start
lxc-start 100 20211023200148.212 ERROR    lxc_start - tools/lxc_start.c:main:313 - Additional information can be obtained by setting the --logfile and --logpriority options
 
Disregard. I just ended up restoring from backup and starting over. For whatever reason I was able to add the new drive without issue this time. Thanks.
 
I had a similar problem and it turned out one of the drives filled its quota (ran out of disk space). I gave it more space and it started right away.
 
You must log in or register to reply here.
Top Bottom