I’m at my wits ends and would be incredibly grateful for any help fixing this. I’ve got a situation where lxc containers successfully boot the first time only, upon starting up the Proxmox node. However, once you shutdown a container (or reboot a container), the container simply will not boot again. Instead, you must reboot the entire node to get the containers to boot again. VMs, on the other hand, work as expected. I can successfully shutdown and restart them. I can also reboot them.
I have two containers, 101 (set to auto-start) and 999 (I have to manually start it). In both cases, the containers start the first time after the Proxmox node boots. (I have to start the 999 container manually, and it works.) But once you’ve shut down the containers, you cannot get them to start again unless you reboot the entire node.
(Note that I created the 999 container from scratch to see if the problem was somehow related to my existing 101 container. I thought having a brand new, fresh container might reveal that the problem was specific to one container rather than all containers in general, but alas, that doesn’t seem to be the case.)
So I found this thread, which looked like it might describe my problem. I tried the following:
and
(If I understand correctly, I only have one zfs pool, which 'zpool list' tells me is 'rpool.')
I then rebooted. Didn’t solve the problem.
Based on another thread that looked promising, I edited the following file:
changing the following line:
to add the overlay option:
After rebooting, however, the problem was unchanged. I therefore reverted the file back.
I then realized that maybe this wasn’t the best or only way to add the overlay option. So I tried:
Also didn’t work, so I turned it back off.
I found another thread (hopefully not too old) that recommend editing:
To add the following options to zfs directories:
It didn’t work either. So I reverted back.
Any thoughts? I’m really at a loss.
Also, if you want me to try something, spelling out the commands is very helpful. I’m trying hard to learn but am green.
I wonder if this is a solution? I’m not quite grasping what was done here or how I would go about doing it myself.
Finally, is this not a bug in Proxmox? I’ve done little beyond run updates in the containers and VMs. It seemed to be working fine for a few months but now, it’s just (seemingly and suddenly) stopped. I note that there are a lot of recent forum posts where people are having problems getting lxc containers to start.
Here’s some debugging and configuration information that I’ve collected that may be helpful.
The second container returns exactly the same.
For some reason, running the same command above but writing the output to a file yields a more verbose description:
While trying to start, I see this in the Proxmox GUI:
I tried mounting the container:
I notice an odd double slash (‘//‘) at the end of the path below. Don’t know if that means anything.
Also, I notice there is NO directory /dev/loop1 ... at least when I look for it later. I see loop3, loop4, all the way to loop7 when I list out the directory manually. Does that mean anything?
[It looks like I'm out of space, so I'll paste more logs in subsequent comments...]
I have two containers, 101 (set to auto-start) and 999 (I have to manually start it). In both cases, the containers start the first time after the Proxmox node boots. (I have to start the 999 container manually, and it works.) But once you’ve shut down the containers, you cannot get them to start again unless you reboot the entire node.
(Note that I created the 999 container from scratch to see if the problem was somehow related to my existing 101 container. I thought having a brand new, fresh container might reveal that the problem was specific to one container rather than all containers in general, but alas, that doesn’t seem to be the case.)
So I found this thread, which looked like it might describe my problem. I tried the following:
zpool set cachefile=/etc/zfs/zpool.cache rpooland
update-initramfs -k all -u(If I understand correctly, I only have one zfs pool, which 'zpool list' tells me is 'rpool.')
I then rebooted. Didn’t solve the problem.
Based on another thread that looked promising, I edited the following file:
/lib/systemd/system/zfs-mount.servicechanging the following line:
ExecStart=/sbin/zfs mount -ato add the overlay option:
ExecStart=/sbin/zfs mount -O -aAfter rebooting, however, the problem was unchanged. I therefore reverted the file back.
I then realized that maybe this wasn’t the best or only way to add the overlay option. So I tried:
zfs set overlay=on rpoolAlso didn’t work, so I turned it back off.
I found another thread (hopefully not too old) that recommend editing:
/etc/pve/storage.cfgTo add the following options to zfs directories:
mkdir 0
is_mountpoint 1It didn’t work either. So I reverted back.
Any thoughts? I’m really at a loss.
Also, if you want me to try something, spelling out the commands is very helpful. I’m trying hard to learn but am green.
I wonder if this is a solution? I’m not quite grasping what was done here or how I would go about doing it myself.
Finally, is this not a bug in Proxmox? I’ve done little beyond run updates in the containers and VMs. It seemed to be working fine for a few months but now, it’s just (seemingly and suddenly) stopped. I note that there are a lot of recent forum posts where people are having problems getting lxc containers to start.
Here’s some debugging and configuration information that I’ve collected that may be helpful.
Code:
root@pve:~# lxc-start -n 101 -F -l DEBUG
lxc-start: 101: conf.c: run_buffer: 323 Script exited with status 255
lxc-start: 101: start.c: lxc_init: 804 Failed to run lxc.hook.pre-start for container "101"
lxc-start: 101: start.c: __lxc_start: 1903 Failed to initialize container "101"
lxc-start: 101: conf.c: run_buffer: 323 Script exited with status 1
lxc-start: 101: start.c: lxc_end: 971 Failed to run lxc.hook.post-stop for container "101"
lxc-start: 101: tools/lxc_start.c: main: 308 The container failed to start
lxc-start: 101: tools/lxc_start.c: main: 314 Additional information can be obtained by setting the --logfile and --logpriority optionsThe second container returns exactly the same.
Code:
root@pve:~# lxc-start -n 999 -F -l DEBUG
lxc-start: 999: conf.c: run_buffer: 323 Script exited with status 255
lxc-start: 999: start.c: lxc_init: 804 Failed to run lxc.hook.pre-start for container "999"
lxc-start: 999: start.c: __lxc_start: 1903 Failed to initialize container "999"
lxc-start: 999: conf.c: run_buffer: 323 Script exited with status 1
lxc-start: 999: start.c: lxc_end: 971 Failed to run lxc.hook.post-stop for container "999"
lxc-start: 999: tools/lxc_start.c: main: 308 The container failed to start
lxc-start: 999: tools/lxc_start.c: main: 314 Additional information can be obtained by setting the --logfile and --logpriority optionsFor some reason, running the same command above but writing the output to a file yields a more verbose description:
Code:
root@pve:~# lxc-start -n 999 -F -l DEBUG -o debug.txt
lxc-start 999 20200801163911.590 INFO confile - confile.c:set_config_idmaps:2051 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 999 20200801163911.590 INFO confile - confile.c:set_config_idmaps:2051 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 999 20200801163911.592 INFO lsm - lsm/lsm.c:lsm_init:29 - LSM security driver AppArmor
lxc-start 999 20200801163911.593 INFO conf - conf.c:run_script_argv:340 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "999", config section "lxc"
lxc-start 999 20200801163956.769 DEBUG conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 999 lxc pre-start produced output: mount: /var/lib/lxc/.pve-staged-mounts/rootfs: can't read superblock on /dev/loop1.
lxc-start 999 20200801163956.794 DEBUG conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 999 lxc pre-start produced output: command 'mount /dev/loop1 /var/lib/lxc/.pve-staged-mounts/rootfs' failed: exit code 32
lxc-start 999 20200801163956.801 ERROR conf - conf.c:run_buffer:323 - Script exited with status 255
lxc-start 999 20200801163956.801 ERROR start - start.c:lxc_init:804 - Failed to run lxc.hook.pre-start for container "999"
lxc-start 999 20200801163956.801 ERROR start - start.c:__lxc_start:1903 - Failed to initialize container "999"
lxc-start 999 20200801163956.801 INFO conf - conf.c:run_script_argv:340 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "999", config section "lxc"
lxc-start 999 20200801163957.816 DEBUG conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 999 lxc post-stop produced output: umount: /var/lib/lxc/999/rootfs: not mounted
lxc-start 999 20200801163957.816 DEBUG conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 999 lxc post-stop produced output: command 'umount --recursive -- /var/lib/lxc/999/rootfs' failed: exit code 1
lxc-start 999 20200801163957.872 ERROR conf - conf.c:run_buffer:323 - Script exited with status 1
lxc-start 999 20200801163957.873 ERROR start - start.c:lxc_end:971 - Failed to run lxc.hook.post-stop for container "999"
lxc-start 999 20200801163957.873 ERROR lxc_start - tools/lxc_start.c:main:308 - The container failed to start
lxc-start 999 20200801163957.873 ERROR lxc_start - tools/lxc_start.c:main:314 - Additional information can be obtained by setting the --logfile and --logpriority optionsWhile trying to start, I see this in the Proxmox GUI:
I tried mounting the container:
Code:
root@pve:~# pct mount 999
mount: /var/lib/lxc/999/rootfs: can't read superblock on /dev/loop1.
mounting container failed
command 'mount /dev/loop1 /var/lib/lxc/999/rootfs//' failed: exit code 32I notice an odd double slash (‘//‘) at the end of the path below. Don’t know if that means anything.
Also, I notice there is NO directory /dev/loop1 ... at least when I look for it later. I see loop3, loop4, all the way to loop7 when I list out the directory manually. Does that mean anything?
[It looks like I'm out of space, so I'll paste more logs in subsequent comments...]
