no container will start

[pmlearner]

After many days of both containers and vm's running fine, I suddenly cannot start any containers (vm's still start).

Fails from both command line and gui with :

Failed to start pve-container@126.service: Connection timed out
See system logs and 'systemctl status pve-container@126.service' for details.
TASK ERROR: command 'systemctl start pve-container@126' failed: exit code 1

however if i start with (as a test)

lxc-start -n 126 -F -l DEBUG -o /tmp/lxc-210.log

it will start.

I plant to use proxmox for the long term so really desire to understand exactly what is going wrong. How should I start debugging this?

Thanks!

 

[dietmar]

Check your strages - maybe a storage is blocking. What is the output of

# pvesm status

Does it display the status of all storages or hang somewhere?

 

[pmlearner]

Thanks for your help, pvesm status returns :


root@Proxmox-VE ~ # pvesm status
Name Type Status Total Used Available %
local dir active 2077376896 209124764 1762657356 10.07%


It takes a good while for the start attempt to return, as if it is waiting for a timeout

Thanks for the further input! I'm sure if I rebooted it might fix things but I really want to fully understand what is going on and how to diagnose and fix in the future.

Another possible data point: the container which I started (as a test) via lxc-start will not stop via pct stop. Perhaps this is normal but thought I would mention that as well. pct stop returns :


trying to acquire lock...
can't lock file '/run/lock/lxc/pve-config-130.lock' - got timeout

I can stop it with lxc-stop 130

If I run :

rm /run/lock/lxc/pve-config-130.lock

it not longer gets the lock timeout but still will not start (returns the same message as my first post)

 

[pmlearner]

pct mount and unmount works correctly for the containers

 

[pmlearner]

perhaps this is a systemd issue as :

systemctl status pve-container@114.service

times out :


Failed to get properties: Connection timed out

I tried to restart systemd without a reboot :


systemctl daemon-reexec


but the containers still will not start

 

[pmlearner]

journalctl -xe

shows:

Jun 11 15:19:22 Proxmox-VE pct[6404]: <root@pam> starting task UPID: Proxmox-VE:00001905:00F83B11:5EE22F5A:vzstart:114:root@pam:
Jun 11 15:19:22 Proxmox-VE pct[6405]: starting CT 114: UPID: Proxmox-VE:00001905:00F83B11:5EE22F5A:vzstart:114:root@pam:
Jun 11 15:20:52 Proxmox-VE pct[6405]: command 'systemctl start pve-container@114' failed: exit code 1
Jun 11 15:20:52 Proxmox-VE pct[6404]: <root@pam> end task UPID: Proxmox-VE:00001905:00F83B11:5EE22F5A:vzstart:114:root@pam: command 'systemctl start pve-container@114' failed: exit code 1

(spaces added to : Proxmox to prevent conversion to emoji by the forum software)

 

[jim.bond.9862]

If you are running zfs as system check your pool space.
I had this issue when my rpool was full.

 

[pmlearner]

Thanks for the reply. I'm not using zfs though. Very good idea though as full disks can indeed cause all sorts of problems. I've checked and all disks look good.