[SOLVED] LXC failing to start after scheduled backup

[godsavethequ33n]

This morning my normal scheduled backup ran however the lxc would not start after the backup completed. Error after trying to start.

lxc-start 113 20230704025941.778 INFO     lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
lxc-start 113 20230704025941.778 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "113", config section "lxc"
lxc-start 113 20230704025942.116 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 113 lxc pre-start produced output: failed to get device path

lxc-start 113 20230704025942.123 ERROR    conf - ../src/lxc/conf.c:run_buffer:322 - Script exited with status 2
lxc-start 113 20230704025942.123 ERROR    start - ../src/lxc/start.c:lxc_init:844 - Failed to run lxc.hook.pre-start for container "113"
lxc-start 113 20230704025942.123 ERROR    start - ../src/lxc/start.c:__lxc_start:2027 - Failed to initialize container "113"
lxc-start 113 20230704025942.123 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "113", config section "lxc"
lxc-start 113 20230704025942.430 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 113 lxc post-stop produced output: umount: /var/lib/lxc/113/rootfs: not mounted

lxc-start 113 20230704025942.430 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 113 lxc post-stop produced output: command 'umount --recursive -- /var/lib/lxc/113/rootfs' failed: exit code 1

lxc-start 113 20230704025942.437 ERROR    conf - ../src/lxc/conf.c:run_buffer:322 - Script exited with status 1
lxc-start 113 20230704025942.437 ERROR    start - ../src/lxc/start.c:lxc_end:985 - Failed to run lxc.hook.post-stop for container "113"
lxc-start 113 20230704025942.437 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:306 - The container failed to start
lxc-start 113 20230704025942.437 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:311 - Additional information can be obtained by setting the --logfile and --logpriority options

Version

Linux 5.15.107-1-pve #1 SMP PVE 5.15.107-1

LXC Config

arch: amd64
cores: 16
features: nesting=1
hostname: plex
memory: 4096
mp0: /mnt/pve/MEDIA_TV,mp=/media/videos/TV
mp1: /mnt/pve/MEDIA_MOVIES,mp=/media/videos/Movies
mp2: /mnt/pve/MEDIA_MUSIC,mp=/media/Music
mp3: suchfast:vm-113-disk-1,mp=/var/lib/plexmediaserver/Library/Application Support/Plex Media Server,backup=1,size=1T
mp4: suchfast:vm-113-disk-2,mp=/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases,backup=1,size=64G
net0: name=eth0,bridge=vmbr0,hwaddr=EE:D7:B8:82:AD:9C,ip=dhcp,type=veth
onboot: 1
ostype: ubuntu
rootfs: suchfast:vm-113-disk-0,size=64G
swap: 2048
tags: plex
lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.cgroup2.devices.allow: c 29:0 rwm
lxc.mount.entry: /dev/fb0 dev/fb0 none bind,optional,create=file
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
lxc.mount.entry: /dev/dri/renderD128 dev/renderD128 none bind,optional,create=file

Any guidance as to what I am missing would be greatly appreciated. If there are any more details I can provide please let me know! Thank you so much!

EDIT: Edit to add debug flag to start log.

 

[godsavethequ33n]

I had some pending updates. I rebooted the server and the container started normally. I would still like to learn what happened here if anyone can point me in the right direction. Thank you.

 

[Chris]

Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 113 lxc pre-start produced output: failed to get device path

HI,
according to this error in your output I would assume that one or more of you mount points were not available at the time you tried to start the container. Or the lxc.mount.entry devices were not there.

 

[godsavethequ33n]

HI,
according to this error in your output I would assume that one or more of you mount points were not available at the time you tried to start the container. Or the lxc.mount.entry devices were not there.

Thank you for direction. Is there a way I can get more details as to which specific mountpoint was not available at the time?

 

[Chris]

Thank you for direction. Is there a way I can get more details as to which specific mountpoint was not available at the time?

I am afraid with the information at hand this is hard to tell.

 

[godsavethequ33n]

I am afraid with the information at hand this is hard to tell.

If this happens again what specific logs should I be capturing to help with troubleshooting?

 

[Chris]

If this happens again what specific logs should I be capturing to help with troubleshooting?

First thing would be to check the storage backing your mountpoints and rootfs. Also the journal from the time of the failure might be interesting, you can get it via journalctl --since <DATETIME> --unitl <DATETIME>

mp0: /mnt/pve/MEDIA_TV,mp=/media/videos/TV mp1: /mnt/pve/MEDIA_MOVIES,mp=/media/videos/Movies mp2: /mnt/pve/MEDIA_MUSIC,mp=/media/Music

Are these mounts of NFS/CIFS shares? Maybe you lost connection there?

 

[godsavethequ33n]

First thing would be to check the storage backing your mountpoints and rootfs. Also the journal from the time of the failure might be interesting, you can get it via journalctl --since <DATETIME> --unitl <DATETIME>

I will see what I can dig up!

Are these mounts of NFS/CIFS shares? Maybe you lost connection there?

mp0, mp1, and mp2 are in fact NFS/CIFS shares. I checked to make sure I could access them when I was experiencing this error. In the GUI I believe they looked similar to this with no sort of error or warning bubbles as I've seen before (hope that makes sense)



I could click on each one and see the usage and if it was enabled without issue.

Thank you!!!