[SOLVED] How to debug container shutdown issue ?

Morphushka

Active Member
Jun 25, 2019
49
5
28
34
Syberia
Hello. I have container with zabbix installed by another employees. This container can be rebooted from inside with reboot command. But It falls into infinity stopping when I want to reboot proxmox. I can't stop caontainer with shutdown button from proxmox web ui, but can stop with "stop" from dropdown menu. During backup with mode "stop" I get reboot command 'lxc-stop -n 102 --nokill --timeout 600' failed: got timeout and backup fail. In the same time from proxmox console with pct stop 102 container stops well. Is it something wrong inside the container ? How to debug this ?
 
Is it something wrong inside the container ?

It seems that the container does not listens to graceful shutdown request send to its init process..
So only "Stop", not "Shutdown" works. We normally do not hard-stop with shutdown, as it could lead to some sort of data loss, depending on what is running inside the CT.

What distro is the container using in which version?

How to debug this ?

As a starter you could follow the logs inside the container and trigger a shutdown, normally the ct should log that it got the request and starts shutting down. Either it never gets that request or a service inside the CT does not plays along and hangs.
 
What distro is the container using in which version?
distro is debian-9.0-standard_9.7-1_amd64.tar.gz

Oh, the problem localized: postregres db shutdown first, then zabbix try to dump data to db before shutdown and cycling on it every 10 seconds

Code:
21878:20191018:153359.199 database is down: reconnecting in 10 seconds
 21878:20191018:153409.200 [Z3001] connection to database 'zabbix' failed: [0] could not connect to server: Connection refused
        Is the server running on host "localhost" (::1) and accepting
        TCP/IP connections on port 5432?
could not connect to server: Connection refused
        Is the server running on host "localhost" (127.0.0.1) and accepting
        TCP/IP connections on port 5432?

Now looking for fix shutsown order.
Thanks for help!
 
With systemd?

As then you could add an overwrite to ensure the ordering is correct for the zabbix.service:
Code:
systemctl edit zabbix.service   # or however it is called
# this opens a editor, here you can add:
[Unit]
After=postregres.service    # ensure to change with real service

save and do systemctl daemon-reload

Then the zabbix should be started After postgres and shutdown Before (stops reverse the ordering statements) postgres.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!