[SOLVED] LXC failing to start after scheduled backup

Mar 8, 2022
43
9
8
36
This morning my normal scheduled backup ran however the lxc would not start after the backup completed. Error after trying to start.

Code:
lxc-start 113 20230704025941.778 INFO     lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
lxc-start 113 20230704025941.778 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "113", config section "lxc"
lxc-start 113 20230704025942.116 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 113 lxc pre-start produced output: failed to get device path

lxc-start 113 20230704025942.123 ERROR    conf - ../src/lxc/conf.c:run_buffer:322 - Script exited with status 2
lxc-start 113 20230704025942.123 ERROR    start - ../src/lxc/start.c:lxc_init:844 - Failed to run lxc.hook.pre-start for container "113"
lxc-start 113 20230704025942.123 ERROR    start - ../src/lxc/start.c:__lxc_start:2027 - Failed to initialize container "113"
lxc-start 113 20230704025942.123 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "113", config section "lxc"
lxc-start 113 20230704025942.430 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 113 lxc post-stop produced output: umount: /var/lib/lxc/113/rootfs: not mounted

lxc-start 113 20230704025942.430 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 113 lxc post-stop produced output: command 'umount --recursive -- /var/lib/lxc/113/rootfs' failed: exit code 1

lxc-start 113 20230704025942.437 ERROR    conf - ../src/lxc/conf.c:run_buffer:322 - Script exited with status 1
lxc-start 113 20230704025942.437 ERROR    start - ../src/lxc/start.c:lxc_end:985 - Failed to run lxc.hook.post-stop for container "113"
lxc-start 113 20230704025942.437 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:306 - The container failed to start
lxc-start 113 20230704025942.437 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:311 - Additional information can be obtained by setting the --logfile and --logpriority options

Version
Code:
Linux 5.15.107-1-pve #1 SMP PVE 5.15.107-1

LXC Config
Code:
arch: amd64
cores: 16
features: nesting=1
hostname: plex
memory: 4096
mp0: /mnt/pve/MEDIA_TV,mp=/media/videos/TV
mp1: /mnt/pve/MEDIA_MOVIES,mp=/media/videos/Movies
mp2: /mnt/pve/MEDIA_MUSIC,mp=/media/Music
mp3: suchfast:vm-113-disk-1,mp=/var/lib/plexmediaserver/Library/Application Support/Plex Media Server,backup=1,size=1T
mp4: suchfast:vm-113-disk-2,mp=/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases,backup=1,size=64G
net0: name=eth0,bridge=vmbr0,hwaddr=EE:D7:B8:82:AD:9C,ip=dhcp,type=veth
onboot: 1
ostype: ubuntu
rootfs: suchfast:vm-113-disk-0,size=64G
swap: 2048
tags: plex
lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.cgroup2.devices.allow: c 29:0 rwm
lxc.mount.entry: /dev/fb0 dev/fb0 none bind,optional,create=file
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
lxc.mount.entry: /dev/dri/renderD128 dev/renderD128 none bind,optional,create=file

Any guidance as to what I am missing would be greatly appreciated. If there are any more details I can provide please let me know! Thank you so much!

EDIT: Edit to add debug flag to start log.
 
Last edited:
I had some pending updates. I rebooted the server and the container started normally. I would still like to learn what happened here if anyone can point me in the right direction. Thank you.
 
Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 113 lxc pre-start produced output: failed to get device path
HI,
according to this error in your output I would assume that one or more of you mount points were not available at the time you tried to start the container. Or the lxc.mount.entry devices were not there.
 
  • Like
Reactions: godsavethequ33n
HI,
according to this error in your output I would assume that one or more of you mount points were not available at the time you tried to start the container. Or the lxc.mount.entry devices were not there.

Thank you for direction. Is there a way I can get more details as to which specific mountpoint was not available at the time?
 
If this happens again what specific logs should I be capturing to help with troubleshooting?
First thing would be to check the storage backing your mountpoints and rootfs. Also the journal from the time of the failure might be interesting, you can get it via journalctl --since <DATETIME> --unitl <DATETIME>
mp0: /mnt/pve/MEDIA_TV,mp=/media/videos/TV mp1: /mnt/pve/MEDIA_MOVIES,mp=/media/videos/Movies mp2: /mnt/pve/MEDIA_MUSIC,mp=/media/Music
Are these mounts of NFS/CIFS shares? Maybe you lost connection there?
 
  • Like
Reactions: godsavethequ33n
First thing would be to check the storage backing your mountpoints and rootfs. Also the journal from the time of the failure might be interesting, you can get it via journalctl --since <DATETIME> --unitl <DATETIME>

I will see what I can dig up!

Are these mounts of NFS/CIFS shares? Maybe you lost connection there?
mp0, mp1, and mp2 are in fact NFS/CIFS shares. I checked to make sure I could access them when I was experiencing this error. In the GUI I believe they looked similar to this with no sort of error or warning bubbles as I've seen before (hope that makes sense)

1688579565490.png

I could click on each one and see the usage and if it was enabled without issue.

Thank you!!!
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!