LXC container hanging on shutdown

[alexskysilk]

this appears to impact only one container. stop job just sits there, and the following 6 lines continue to repeat in the journal:

pve-ha-lrm[2803559]: Task 'UPID:sky11:002AC76E:058F4534:59B2B74D:vzshutdown:1257:xxx@pam:' still active, waiting
Sep 08 08:32:13 sky11 pvedaemon[2813128]: Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/PVE/Tools.pm line 932.
Sep 08 08:32:13 sky11 pvedaemon[2813128]: Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/PVE/Tools.pm line 932.
Sep 08 08:32:13 sky11 pvedaemon[2813128]: Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/PVE/Tools.pm line 932.
Sep 08 08:32:13 sky11 pvedaemon[2338796]: Argument "" isn't numeric in int at /usr/share/perl5/PVE/Tools.pm line 945, <GEN15304> line 1.
Sep 08 08:32:13 sky11 pvedaemon[2338796]: Argument "" isn't numeric in int at /usr/share/perl5/PVE/Tools.pm line 946, <GEN15304> line 2.
Sep 08 08:32:13 sky11 pvedaemon[2338796]: Argument "" isn't numeric in int at /usr/share/perl5/PVE/Tools.pm line 947, <GEN15304> line 3.

pveversion -v
proxmox-ve: 5.0-20 (running kernel: 4.10.17-2-pve)
pve-manager: 5.0-30 (running version: 5.0-30/5ab26bc)
pve-kernel-4.10.17-2-pve: 4.10.17-20
libpve-http-server-perl: 2.0-6
lvm2: 2.02.168-pve3
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-12
qemu-server: 5.0-15
pve-firmware: 2.0-2
libpve-common-perl: 5.0-16
libpve-guest-common-perl: 2.0-11
libpve-access-control: 5.0-6
libpve-storage-perl: 5.0-14
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-2
pve-docs: 5.0-9
pve-qemu-kvm: 2.9.0-4
pve-container: 2.0-15
pve-firewall: 3.0-2
pve-ha-manager: 2.0-2
ksm-control-daemon: 1.2-2
glusterfs-client: 3.8.8-1
lxc-pve: 2.0.8-3
lxcfs: 2.0.7-pve4
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.6.5.11-pve17~bpo90
ceph: 12.1.2-pve1

 

[t.lamprecht]

Sep 08 08:32:13 sky11 pvedaemon[2813128]: Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/PVE/Tools.pm line 932. Sep 08 08:32:13 sky11 pvedaemon[2813128]: Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/PVE/Tools.pm line 932. Sep 08 08:32:13 sky11 pvedaemon[2813128]: Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/PVE/Tools.pm line 932. Sep 08 08:32:13 sky11 pvedaemon[2338796]: Argument "" isn't numeric in int at /usr/share/perl5/PVE/Tools.pm line 945, <GEN15304> line 1. Sep 08 08:32:13 sky11 pvedaemon[2338796]: Argument "" isn't numeric in int at /usr/share/perl5/PVE/Tools.pm line 946, <GEN15304> line 2. Sep 08 08:32:13 sky11 pvedaemon[2338796]: Argument "" isn't numeric in int at /usr/share/perl5/PVE/Tools.pm line 947, <GEN15304> line 3.

The log outputs are a cosmetic bug, it seems.
But the reason for that to trigger is either a storage which path does not exists or hangs.
Is there such a storage (nfs, gluster?) which is not online or hangs?
Could also be the problem for the hanging LXC CT, it tries to sync its data on shutdown but hangs as the storage is gone?

 

[alexskysilk]

the backing store is a ceph rbd, which is operating normally. Other containers can be shut down normally as well, it appears to impact specific container(s) only.

 

[itvietnam]

Hi Alex,

Did you manage this problem? We have the same case with you. Ceph as backend storage.

 

[alexskysilk]

I'm fairly certain this was a product of an incomplete apt-get upgrade process. make sure you complete apt-get upgrade and reboot the node.

 

[itvietnam]

Hi Alex,

Are your CT managed by HA?

This happen for this CT (my case ID 135) only. We tested other CT and thing never happen.

 

[alexskysilk]

Yeah that was my experience as well. if a ct fails you can normally find out why by running it in debug mode; if you have the perl code bomb its because there is a setting on the ct config that is breaking the respective module (in this case, Tools.pm) which is not supposed to happen under normal conditions- the most likely cause of which is that the proxmox software stack is not in a sane state.