[SOLVED] Need Help troubleshooting LXC not starting issue.

[jim.bond.9862]

Hi.
I have a PVE 5.2-5 install (just updated)
my server have been having issues all last several weeks with ZFS pools loosing drives sporadically,than I start getting CPU lock errors etc...

based on my research it seams like my power supply was giving up on me. it is/was a good 760W PSU but it is over 10 years old so can not complain.
so last night I got a new PSU 1050W modular 80+ Gold and replaced the old one.

turns out I also had a bad disk but I already have a replacement for it as it was one of the first thing I thought when my zfs pool show degraded status. replaced the disk re-silvered all is good.

over all the system seams to be running ok. however I have 3 containers that would not start.

WebUI shows :

Job for pve-container@101.service failed because the control process exited with error code.
See "systemctl status pve-container@101.service" and "journalctl -xe" for details.
TASK ERROR: command 'systemctl start pve-container@101' failed: exit code 1

run the sysctl command
output :

root@pve2:~# systemctl status pve-container@101.service
● pve-container@101.service - PVE LXC Container: 101
   Loaded: loaded (/lib/systemd/system/pve-container@.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2018-07-16 13:31:24 EDT; 7min ago
     Docs: man:lxc-start
           man:lxc
           man:pct
  Process: 14866 ExecStart=/usr/bin/lxc-start -n 101 (code=exited, status=1/FAILURE)

Jul 16 13:31:23 pve2 systemd[1]: Starting PVE LXC Container: 101...
Jul 16 13:31:24 pve2 lxc-start[14866]: lxc-start: 101: lxccontainer.c: wait_on_daemonized_start: 815 No such file or directory - Failed to receive the container state
Jul 16 13:31:24 pve2 lxc-start[14866]: The container failed to start.
Jul 16 13:31:24 pve2 lxc-start[14866]: To get more details, run the container in foreground mode.
Jul 16 13:31:24 pve2 lxc-start[14866]: Additional information can be obtained by setting the --logfile and --logpriority options.
Jul 16 13:31:24 pve2 systemd[1]: pve-container@101.service: Control process exited, code=exited status=1
Jul 16 13:31:24 pve2 systemd[1]: Failed to start PVE LXC Container: 101.
Jul 16 13:31:24 pve2 systemd[1]: pve-container@101.service: Unit entered failed state.
Jul 16 13:31:24 pve2 systemd[1]: pve-container@101.service: Failed with result 'exit-code'.

no issues in journalctl I can see

root@pve2:~# journalctl -xe
-- The start-up result is done.
Jul 16 13:39:00 pve2 systemd[1]: Starting Proxmox VE replication runner...
-- Subject: Unit pvesr.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pvesr.service has begun starting up.
Jul 16 13:39:00 pve2 systemd[1]: Started Proxmox VE replication runner.
-- Subject: Unit pvesr.service has finished start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pvesr.service has finished starting up.
--
-- The start-up result is done.
Jul 16 13:40:00 pve2 systemd[1]: Starting Proxmox VE replication runner...
-- Subject: Unit pvesr.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pvesr.service has begun starting up.
Jul 16 13:40:00 pve2 systemd[1]: Started Proxmox VE replication runner.
-- Subject: Unit pvesr.service has finished start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pvesr.service has finished starting up.
--
-- The start-up result is done.
Jul 16 13:41:00 pve2 systemd[1]: Starting Proxmox VE replication runner...
-- Subject: Unit pvesr.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pvesr.service has begun starting up.
Jul 16 13:41:00 pve2 systemd[1]: Started Proxmox VE replication runner.
-- Subject: Unit pvesr.service has finished start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pvesr.service has finished starting up.
--
-- The start-up result is done.
Jul 16 13:42:00 pve2 systemd[1]: Starting Proxmox VE replication runner...
-- Subject: Unit pvesr.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pvesr.service has begun starting up.
Jul 16 13:42:01 pve2 systemd[1]: Started Proxmox VE replication runner.
-- Subject: Unit pvesr.service has finished start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pvesr.service has finished starting up.
--
-- The start-up result is done.
lines 2532-2588/2588 (END)

how can I figure out what is going on?
I have 2 more containers that work just fine. unfortunately they are not as important as those that don't work.
I want to try to see if I can fix this before I through in a towel and rebuild them.
all my data is on mount points so should not loose anything except configuration.

All containers are Debian 8.
those not working are build using TurnKey templates for file server and Emby server
those that do work are based on Standard Debian 8 template and manually configured by me just before all the issues started. one to run PyLoad and one to convert into my own base template for creating other setups.

can some one help me fix this?

thanks Vl

 

[wolfgang]

Hi,

send the output of the following command.

lxc-start -n 101 -F -l DEBUG

 

[jim.bond.9862]

Hi,

send the output of the following command.

lxc-start -n 101 -F -l DEBUG

this is what I get

root@pve2:~# lxc-start -n 101 -F -l DEBUG
lxc-start: 101: conf.c: run_buffer: 347 Script exited with status 2
lxc-start: 101: start.c: lxc_init: 815 Failed to run lxc.hook.pre-start for container "101"
lxc-start: 101: start.c: __lxc_start: 1849 Failed to initialize container "101"
The container failed to start.
Additional information can be obtained by setting the --logfile and --logpriority options.

 

[wolfgang]

Do you have modified the pre-start-hook?

please run again and sent the lxc.log file

lxc-start -n 101 -F -l DEBUG -o lxc.log

 

[jim.bond.9862]

thank you very much. I was missing a zpool that the CT was bind-mount to.
it did not come up on boot. need to check why.
started the pool and now all CT are starting as expected thanks.


PS. have to save the last command for later use if this happens again.