Diagnosing mount: /var/lib/lxc/.pve-staged-mounts/rootfs: can't read superblock on /dev/loop3

[KenwoodFox]

Hello proxmox fourms!

I'm recently getting this error (and the same error minus line 1 when starting without using HA) when attempting to start this container.
Im a bit stressed, because the backup for this container is a bit old, and id really like to restore this container to functionality. I have no idea what was happening when it initially failed, it seems to have done this randomly in the middle of the night under minimum load.

This CT runs on an NFS share using zfs storage backend. I was able to do a pct mount and pct unmount, as it was the only thing mentioned in a similar forum post I read.

task started by HA resource agent
run_buffer: 323 Script exited with status 255
lxc_init: 797 Failed to run lxc.hook.pre-start for container "102"
__lxc_start: 1896 Failed to initialize container "102"
TASK ERROR: startup for container '102' failed

I'm looking not for an immediate answer, I just want help starting out exploring what went wrong, id really like to know what questions I should be asking myself so I can diagnose this on my own.
Thank you so much for reading!

 

[KenwoodFox]

Just in:
attempting to mount *now* results in this

root@Cobalt:~# pct mount 102
mount: /var/lib/lxc/102/rootfs: can't read superblock on /dev/loop3.
mounting container failed
command 'mount /dev/loop3 /var/lib/lxc/102/rootfs//' failed: exit code 32

and just a note I actually don't know for sure what mount does, is this mounting the virtual lvm disk for the container?

That error sounds bad.. but at least I'm getting output?! if anybody could advise what to google next that would be awesome.

 

[KenwoodFox]

I don't mean to be pushy, I'm just stressed about this is all.

My backup solution failed recently and I've been dealing with a lot of technical issues here I just really don't want another big project. If anybody could just, throw me some keywords I'll get googling and reading the manual. I know you guys are really awesome and thorough.

 

[KenwoodFox]

Thanks to sep on IRC i was able to capture a logfile.

# cat logfile.log
lxc-start 10 20201207134844.572 ERROR    lxc_start - tools/lxc_start.c:main:268 - No container config specified
lxc-start 102 20201207134847.251 INFO     confile - confile.c:set_config_idmaps:2055 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 102 20201207134847.251 INFO     confile - confile.c:set_config_idmaps:2055 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 102 20201207134847.251 INFO     lsm - lsm/lsm.c:lsm_init:29 - LSM security driver AppArmor
lxc-start 102 20201207134847.252 INFO     conf - conf.c:run_script_argv:340 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "102", config section "lxc"
lxc-start 102 20201207134852.623 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 102 lxc pre-start produced output: mount: /var/lib/lxc/.pve-staged-mounts/rootfs: can't read superblock on /dev/loop3.

lxc-start 102 20201207134852.655 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 102 lxc pre-start produced output: command 'mount /dev/loop3 /var/lib/lxc/.pve-staged-mounts/rootfs' failed: exit code 32

lxc-start 102 20201207134852.666 ERROR    conf - conf.c:run_buffer:323 - Script exited with status 255
lxc-start 102 20201207134852.666 ERROR    start - start.c:lxc_init:797 - Failed to run lxc.hook.pre-start for container "102"
lxc-start 102 20201207134852.666 ERROR    start - start.c:__lxc_start:1896 - Failed to initialize container "102"
lxc-start 102 20201207134852.667 INFO     conf - conf.c:run_script_argv:340 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "102", config section "lxc"
lxc-start 102 20201207134853.169 INFO     conf - conf.c:run_script_argv:340 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "102", config section "lxc"
lxc-start 102 20201207134853.808 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 102 lxc post-stop produced output: umount: /var/lib/lxc/102/rootfs: not mounted

lxc-start 102 20201207134853.808 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 102 lxc post-stop produced output: command 'umount --recursive -- /var/lib/lxc/102/rootfs' failed: exit code 1

lxc-start 102 20201207134853.818 ERROR    conf - conf.c:run_buffer:323 - Script exited with status 1
lxc-start 102 20201207134853.818 ERROR    start - start.c:lxc_end:964 - Failed to run lxc.hook.post-stop for container "102"
lxc-start 102 20201207134853.818 ERROR    lxc_start - tools/lxc_start.c:main:308 - The container failed to start
lxc-start 102 20201207134853.818 ERROR    lxc_start - tools/lxc_start.c:main:314 - Additional information can be obtained by setting the --logfile and --logpriority options

 

[KenwoodFox]

zapotah on irc mentioned a possible solution about mounting the vdisk "out of band" as a form of recovery? Is this viable and what could it mean.