[SOLVED] Ubuntu container won't restart

E

el_pedr0

Renowned Member
Aug 5, 2016
24
3
68
47
Hi,

One of my four containers refuses to restart after a recent power outage. I'd be grateful of steps I can take to try to resolve it.

I've tried issuing the command:
Code: 
lxc-start -n 115 -F -lDEBUG -o temp/lxc-115-2.log:

Here's the output at the command line
Code: 
lxc-start: 115: conf.c: run_buffer: 323 Script exited with status 1
lxc-start: 115: start.c: lxc_init: 804 Failed to run lxc.hook.pre-start for container "115"
lxc-start: 115: start.c: __lxc_start: 1903 Failed to initialize container "115"
lxc-start: 115: tools/lxc_start.c: main: 308 The container failed to start
lxc-start: 115: tools/lxc_start.c: main: 314 Additional information can be obtained by setting the --logfile and --logpriority options

And
Here's the resulting log file:
Code: 
lxc-start 115 20200910165233.673 INFO     confile - confile.c:set_config_idmaps:2051 - Read uid map: type u nsid 1614000000 hostid 1614000000 range 200000
lxc-start 115 20200910165233.673 INFO     confile - confile.c:set_config_idmaps:2051 - Read uid map: type g nsid 1614000000 hostid 1614000000 range 200000
lxc-start 115 20200910165233.673 INFO     confile - confile.c:set_config_idmaps:2051 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 115 20200910165233.673 INFO     confile - confile.c:set_config_idmaps:2051 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 115 20200910165233.673 INFO     lsm - lsm/lsm.c:lsm_init:29 - LSM security driver AppArmor
lxc-start 115 20200910165233.673 INFO     conf - conf.c:run_script_argv:340 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "115", config section "lxc"
lxc-start 115 20200910165234.154 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 115 lxc pre-start produced output: unable to open file '/fastboot.tm$

lxc-start 115 20200910165234.161 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 115 lxc pre-start produced output: error in setup task PVE::LXC::Set$

lxc-start 115 20200910165234.223 ERROR    conf - conf.c:run_buffer:323 - Script exited with status 1
lxc-start 115 20200910165234.224 ERROR    start - start.c:lxc_init:804 - Failed to run lxc.hook.pre-start for container "115"
lxc-start 115 20200910165234.224 ERROR    start - start.c:__lxc_start:1903 - Failed to initialize container "115"
lxc-start 115 20200910165234.225 INFO     conf - conf.c:run_script_argv:340 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "115", config section "lxc"
lxc-start 115 20200910165234.524 INFO     conf - conf.c:run_script_argv:340 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "115", config section "lxc"
lxc-start 115 20200910165234.816 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 115 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-$

lxc-start 115 20200910165234.816 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 115 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.$

lxc-start 115 20200910165234.825 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 115 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-$

lxc-start 115 20200910165234.825 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 115 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.$

lxc-start 115 20200910165234.828 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 115 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-$

lxc-start 115 20200910165234.828 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 115 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.$

lxc-start 115 20200910165234.830 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 115 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-$

lxc-start 115 20200910165234.830 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 115 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.$

lxc-start 115 20200910165234.837 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 115 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-$

lxc-start 115 20200910165234.837 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 115 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.$

lxc-start 115 20200910165234.875 ERROR    lxc_start - tools/lxc_start.c:main:308 - The container failed to start
lxc-start 115 20200910165234.875 ERROR    lxc_start - tools/lxc_start.c:main:314 - Additional information can be obtained by setting the --logfile and --logpriority options

container config:
Code: 
arch: amd64
cores: 1
hostname: downloader.home.pmer.tk
memory: 1280
mp0: /bodpool/Media/Downloads,mp=/mnt/Downloads
mp1: /bodpool/Media/Library,mp=/mnt/Library
mp2: /bodpool/Media/Library/TVShows,mp=/mnt/Library/TVShows
mp3: /bodpool/Media/Library/Movies,mp=/mnt/Library/Movies
mp4: /bodpool/Media/Library/Music,mp=/mnt/Library/Music
mp5: /bodpool/Media/Library/Pictures,mp=/mnt/Library/Pictures
mp6: /bodpool/Media/Library/HomeVideos,mp=/mnt/Library/HomeVideos
mp7: /bodpool/Media/Library/Videos,mp=/mnt/Library/Videos
mp8: /bodpool/Media/Library/Books,mp=/mnt/Library/Books
mp9: /bodpool/Media/Library/Games,mp=/mnt/Library/Games
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.199.254,hwaddr=82:B3:77:28:EB:44,ip=192.168.199.15/24,type=veth
onboot: 1
ostype: ubuntu
rootfs: local-zfs:subvol-115-disk-0,size=8G
startup: order=3
swap: 0
unprivileged: 1
lxc.idmap: u 1614000000 1614000000 200000
lxc.idmap: g 1614000000 1614000000 200000
lxc.idmap: u 0 100000 65536
lxc.idmap: g 0 100000 65536

Here's the output of journalctl -xe
Code: 
Sep 10 17:48:34 toast.home.pmer.tk audit[4939]: AVC apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd" name="/etc/selinux/semanage.conf" pid=4939 comm="selinux_child" requested_mask="r" d
Sep 10 17:48:34 toast.home.pmer.tk sudo[4938]: proxmoxadmin : TTY=pts/3 ; PWD=/home/proxmoxadmin ; USER=root ; COMMAND=/usr/sbin/pct start 115
Sep 10 17:48:34 toast.home.pmer.tk sudo[4938]: pam_unix(sudo:session): session opened for user root by proxmoxadmin(uid=0)
Sep 10 17:48:34 toast.home.pmer.tk kernel: audit: type=1400 audit(1599756514.714:2349): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd" name="/etc/selinux/semanage.conf" pid=4939 comm=
Sep 10 17:48:35 toast.home.pmer.tk pct[4940]: <root@pam> starting task UPID:toast:0000134E:011ABB16:5F5A58E3:vzstart:115:root@pam:
Sep 10 17:48:35 toast.home.pmer.tk pct[4942]: starting CT 115: UPID:toast:0000134E:011ABB16:5F5A58E3:vzstart:115:root@pam:
Sep 10 17:48:35 toast.home.pmer.tk systemd[1]: Started PVE LXC Container: 115.
-- Subject: A start job for unit pve-container@115.service has finished successfully
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit pve-container@115.service has finished successfully.
--
-- The job identifier is 175285.
Sep 10 17:48:35 toast.home.pmer.tk pct[4940]: <root@pam> end task UPID:toast:0000134E:011ABB16:5F5A58E3:vzstart:115:root@pam: OK
Sep 10 17:48:35 toast.home.pmer.tk sudo[4938]: pam_unix(sudo:session): session closed for user root
Sep 10 17:48:35 toast.home.pmer.tk pvestatd[2713]: unable to get PID for CT 115 (not running?)
Sep 10 17:48:36 toast.home.pmer.tk systemd[1]: pve-container@115.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- An ExecStart= process belonging to unit pve-container@115.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Sep 10 17:48:36 toast.home.pmer.tk systemd[1]: pve-container@115.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit pve-container@115.service has entered the 'failed' state with result 'exit-code'.
Sep 10 17:48:43 toast.home.pmer.tk audit[5226]: AVC apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd" name="/etc/selinux/semanage.conf" pid=5226 comm="selinux_child" requested_mask="r" d
Sep 10 17:48:43 toast.home.pmer.tk sudo[5225]: proxmoxadmin : TTY=pts/3 ; PWD=/home/proxmoxadmin ; USER=root ; COMMAND=/bin/journalctl -xe
Sep 10 17:48:43 toast.home.pmer.tk sudo[5225]: pam_unix(sudo:session): session opened for user root by proxmoxadmin(uid=0)
Sep 10 17:48:43 toast.home.pmer.tk kernel: audit: type=1400 audit(1599756523.826:2350): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd" name="/etc/selinux/semanage.conf" pid=5226 comm=
 
please post the output of `pveversion -v`
if possible also attach the complete debug-logfile of the containerstart (some information seems to be cut-off)

how do you try to restart the container? (executing `reboot` inside the container, `pct reboot 115`? )


Thanks!
 
Thanks for the fast response.

The container used to boot at startup of the host. I've tried to start it from the GUI by selecting the container and clicking start at the top of the page. And I've tried pct start 115 from the command line of the host.

I can't execute reboot at the moment because I can't get it to start. For clarification - I suspect (but cant' be sure) that there was a power failure last time it was running and so it didn't get gracefully shut down.

But all these logs are from using the lxc-start command.

Attached is the debug log file

Here's pveversion -v
Code: 
proxmox-ve: 6.2-1 (running kernel: 5.4.41-1-pve)
pve-manager: 6.2-4 (running version: 6.2-4/9824574a)
pve-kernel-5.4: 6.2-2
pve-kernel-helper: 6.2-2
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-4.15: 5.4-18
pve-kernel-4.15.18-29-pve: 4.15.18-57
pve-kernel-4.15.18-28-pve: 4.15.18-56
pve-kernel-4.15.18-15-pve: 4.15.18-40
pve-kernel-4.15.18-12-pve: 4.15.18-36
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-1
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-2
libpve-guest-common-perl: 3.0-10
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve2
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-1
pve-cluster: 6.1-8
pve-container: 3.1-6
pve-docs: 6.2-4
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-2
pve-qemu-kvm: 5.0.0-2
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-2
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1
 

Attachments

There have been a few updates to your current state of packages - please upgrade and try reproducing the issue with the latest version (after a reboot)
 
Unfortunately it looks like the problem persists after apt full-upgrade and reboot of host.

Code: 
#pct start 115
run_buffer: 323 Script exited with status 1
lxc_init: 797 Failed to run lxc.hook.pre-start for container "115"
__lxc_start: 1896 Failed to initialize container "115"
startup for container '115' failed

Code: 
#pveversion -v
proxmox-ve: 6.2-1 (running kernel: 5.4.60-1-pve)
pve-manager: 6.2-11 (running version: 6.2-11/22fb4983)
pve-kernel-5.4: 6.2-6
pve-kernel-helper: 6.2-6
pve-kernel-5.4.60-1-pve: 5.4.60-2
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-4.15: 5.4-18
pve-kernel-4.15.18-29-pve: 4.15.18-57
pve-kernel-4.15.18-28-pve: 4.15.18-56
pve-kernel-4.15.18-15-pve: 4.15.18-40
pve-kernel-4.15.18-12-pve: 4.15.18-36
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-2
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-6
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-12
pve-cluster: 6.1-8
pve-container: 3.2-1
pve-docs: 6.2-5
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-2
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-1
pve-qemu-kvm: 5.1.0-1
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-14
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve1


Code: 
# lxc-start -n 115 -F -lDEBUG -o temp/lxc-115-2.log:
lxc-start: 115: conf.c: run_buffer: 323 Script exited with status 1
lxc-start: 115: start.c: lxc_init: 797 Failed to run lxc.hook.pre-start for container "115"
lxc-start: 115: start.c: __lxc_start: 1896 Failed to initialize container "115"
lxc-start: 115: tools/lxc_start.c: main: 308 The container failed to start
lxc-start: 115: tools/lxc_start.c: main: 314 Additional information can be obtained by setting the --logfile and --logpriority options

And the log file attached - the actual full one this time I hope without any text truncated.
 

Attachments

Thanks for the complete log - this explains it:
Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 115 lxc pre-start produced output: unable to open file '/fastboot.tmp.11249' - Disk quota exceeded
Click to expand...

seems your container's root-filesystem is over quota (meaning it has no free space left) -> increase the container's rootfs size - boot it and then try to free up some space inside

I hope this helps!
 
  • Like
Reactions: el_pedr0
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom