[SOLVED] How to debug container shutdown issue ?

[Morphushka]

Hello. I have container with zabbix installed by another employees. This container can be rebooted from inside with reboot command. But It falls into infinity stopping when I want to reboot proxmox. I can't stop caontainer with shutdown button from proxmox web ui, but can stop with "stop" from dropdown menu. During backup with mode "stop" I get reboot command 'lxc-stop -n 102 --nokill --timeout 600' failed: got timeout and backup fail. In the same time from proxmox console with pct stop 102 container stops well. Is it something wrong inside the container ? How to debug this ?

 

[t.lamprecht]

Is it something wrong inside the container ?

It seems that the container does not listens to graceful shutdown request send to its init process..
So only "Stop", not "Shutdown" works. We normally do not hard-stop with shutdown, as it could lead to some sort of data loss, depending on what is running inside the CT.

What distro is the container using in which version?

How to debug this ?

As a starter you could follow the logs inside the container and trigger a shutdown, normally the ct should log that it got the request and starts shutting down. Either it never gets that request or a service inside the CT does not plays along and hangs.

 

[Morphushka]

What distro is the container using in which version?

distro is debian-9.0-standard_9.7-1_amd64.tar.gz

Oh, the problem localized: postregres db shutdown first, then zabbix try to dump data to db before shutdown and cycling on it every 10 seconds

21878:20191018:153359.199 database is down: reconnecting in 10 seconds
 21878:20191018:153409.200 [Z3001] connection to database 'zabbix' failed: [0] could not connect to server: Connection refused
        Is the server running on host "localhost" (::1) and accepting
        TCP/IP connections on port 5432?
could not connect to server: Connection refused
        Is the server running on host "localhost" (127.0.0.1) and accepting
        TCP/IP connections on port 5432?

Now looking for fix shutsown order.
Thanks for help!

 

[t.lamprecht]

With systemd?

As then you could add an overwrite to ensure the ordering is correct for the zabbix.service:

systemctl edit zabbix.service   # or however it is called
# this opens a editor, here you can add:
[Unit]
After=postregres.service    # ensure to change with real service

save and do systemctl daemon-reload

Then the zabbix should be started After postgres and shutdown Before (stops reverse the ordering statements) postgres.

 

[Morphushka]

With systemd?

Yes.
Hmm, we did it in another way: we edit postgresql.service and add ExecStop=/etc/init.d/zabbix-server stop The both approachs works well, but your probably cleaner and therefore better. Thanks!