[SOLVED] Unable to get PID for CT on stop

[bootsie123]

Hi. I'm running into an issue where when I try to stop a container. In my syslog I get this error:

Aug 07 18:27:40 vmworld pvestatd[4024]: unable to get PID for CT 122 (not running?)
Aug 07 18:27:41 vmworld pvestatd[4024]: unable to get PID for CT 122 (not running?)
Aug 07 18:27:50 vmworld pvestatd[4024]: unable to get PID for CT 122 (not running?)
Aug 07 18:27:50 vmworld pvestatd[4024]: unable to get PID for CT 122 (not running?)
Aug 07 18:27:53 vmworld pvedaemon[4053]: unable to get PID for CT 122 (not running?)
Aug 07 18:27:55 vmworld pvedaemon[4053]: unable to get PID for CT 122 (not running?)
Aug 07 18:27:57 vmworld pvedaemon[4053]: unable to get PID for CT 122 (not running?)
Aug 07 18:27:58 vmworld pvedaemon[4053]: unable to get PID for CT 122 (not running?)

This is returned by both pvestatd and pvedaemon. This ONLY happens when stopping a container. VMs stop how they would and shutting down a container works properly too.

Let me know if you need anything else. Thanks!

Package Versions:

proxmox-ve: 5.4-1 (running kernel: 4.15.18-17-pve)
pve-manager: 5.4-13 (running version: 5.4-13/aee6f0ec)
pve-kernel-4.15: 5.4-5
pve-kernel-4.15.18-19-pve: 4.15.18-45
pve-kernel-4.15.18-18-pve: 4.15.18-44
pve-kernel-4.15.18-17-pve: 4.15.18-43
pve-kernel-4.15.18-16-pve: 4.15.18-41
pve-kernel-4.15.18-13-pve: 4.15.18-37
pve-kernel-4.15.18-11-pve: 4.15.18-34
pve-kernel-4.15.18-10-pve: 4.15.18-32
pve-kernel-4.15.18-9-pve: 4.15.18-30
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-12
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-54
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-14
libpve-storage-perl: 5.0-44
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-3
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-28
pve-cluster: 5.0-37
pve-container: 2.0-40
pve-docs: 5.4-2
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-22
pve-firmware: 2.0-7
pve-ha-manager: 2.0-9
pve-i18n: 1.1-4
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 3.0.1-4
pve-xtermjs: 3.12.0-1
qemu-server: 5.0-54
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.13-pve1~bpo2

 

[Chris]

HI,
does the container stop correctly? Can you please post the output of `lxc-info -n CTID -p` right before you stop the container?

 

[bootsie123]

It seems that the container does eventually stop. For some reason, it seems that it's only one container that has this issue. Although sometimes other containers take longer then they should to startup after a container has been stopped. I think that issue is related to :

Aug 08 09:19:10 vmworld kernel: unregister_netdevice: waiting for lo to become free. Usage count = 1

although I could be wrong.

Anyway, when I run `lxc-info -n 122 -p` before I stop the container I get:

PID: 50612

When I run `lxc-info 122` right after pressing stop, this is what I get:

Name: 122
State: STOPPED

So it seems that the container actually stops, although it seems to stop too quickly for the other services to realize that it stopped.

 

[Chris]

Could you please post the container configuration `pct config CTID`, there seems to be quite some delay between the lxc process being terminated and the time the services see it as stopped.

 

[bootsie123]

Sure. Here's the container config:

arch: amd64
cores: 4
hostname: VM-122
memory: 2048
net0: name=eth0,bridge=vmbr0,gw=192.168.1.1,hwaddr=EE:76:9F:A8:79:50,ip=192.168.1.85/24,type=veth
ostype: ubuntu
rootfs: local-zfs:basevol-100-disk-0/subvol-122-disk-0,size=32G
swap: 1024
lxc.cgroup.devices.allow: c 10:200 rwm
lxc.hook.autodev: sh -c "modprobe tun; cd ${LXC_ROOTFS_MOUNT}/dev; mkdir net; mknod net/tun c 10 200; chmod 0666 net/tun"

 

[Chris]

Unfortunately I am not able to reproduce this. Could you check the output of `cat /proc/net/unix | grep /var/lib/lxc/122/command` with the container running and right after it stopped?

 

[bootsie123]

Here's with the container running:

000000003ec2d6b2: 00000002 00000000 00010000 0001 01 1549226 @/var/lib/lxc/122/command
000000004b1b8e2c: 00000003 00000000 00000000 0001 03 1540558 @/var/lib/lxc/122/command

and here's right after I press stop:

000000004b1b8e2c: 00000002 00000000 00000000 0001 03 1540558 @/var/lib/lxc/122/command
00000000bcbcafc5: 00000003 00000000 00000000 0001 03 4069784 @/var/lib/lxc/122/command
000000005180a037: 00000003 00000000 00000000 0001 03 4069787 @/var/lib/lxc/122/command

After it finally stops there is no output

 

[Chris]

Ok so it seems that some unix sockets outlive the lxc init process since you forcefully stop the container (shutdown performs a clean shutdown while stop abruptly stops all processes running in the container). PVE checks those to see if the container is still active and therefore pvestatd and pvedaemon try to get infos via the PID of the lxc init process, which on the other hand is no longer there.
So this is to expect, shutdown is what you should aim for.

 

[bootsie123]

Alright, so that sounds about what should be expected then. I guess it must have to do something with the config of the container since all other containers don't seem to be having that problem. I guess that wraps it up. Thanks for the help!

 

[Peterov]

Sure. Here's the container config:

arch: amd64
cores: 4
hostname: VM-122
memory: 2048
net0: name=eth0,bridge=vmbr0,gw= 192.168.1.1 ,hwaddr=EE:76:9F:A8:79:50,ip=192.168.1.85/24,type=veth
ostype: ubuntu
rootfs: local-zfs:basevol-100-disk-0/subvol-122-disk-0,size=32G
swap: 1024
lxc.cgroup.devices.allow: c 10:200 rwm
lxc.hook.autodev: sh -c "modprobe tun; cd ${LXC_ROOTFS_MOUNT}/dev; mkdir net; mknod net/tun c 10 200; chmod 0666 net/tun"

Thank you !
I have used this website to find my local ip adress

 

[jamesrobert5]

Sure. Here's the container config:

arch: amd64
cores: 4
hostname: VM-122
memory: 2048
net0: name=eth0,bridge=vmbr0,gw=192.168.1.1,hwaddr=EE:76:9F:A8:79:50,ip=192.168.1.85/24,type=veth
ostype: ubuntu
rootfs: local-zfs:basevol-100-disk-0/subvol-122-disk-0,size=32G
swap: 1024
lxc.cgroup.devices.allow: c 10:200 rwm
lxc.hook.autodev: sh -c "modprobe tun; cd ${LXC_ROOTFS_MOUNT}/dev; mkdir net; mknod net/tun c 10 200; chmod 0666 net/tun"

it does help, Thanks.