"unregister_netdevice: waiting for lo to become free" after container shutdown

Proxygen · Nov 30, 2019

After an unsuccessful shutdown, a stop was issued:

Code:

lxc-stop: 101: commands_utils.c: lxc_cmd_sock_rcv_state: 72 Resource temporarily unavailable - Failed to receive message
command 'lxc-stop -n 101 --nokill --timeout 60' failed: exit code 1
TASK OK

Code:

# pveversion --verbose
proxmox-ve: 6.0-2 (running kernel: 5.0.21-4-pve)
pve-manager: 6.0-15 (running version: 6.0-15/52b91481)
pve-kernel-helper: 6.0-12
pve-kernel-5.0: 6.0-11
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.21-4-pve: 5.0.21-9
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-9
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.0-12
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-1
pve-cluster: 6.0-9
pve-container: 3.0-14
pve-docs: 6.0-9
pve-edk2-firmware: 2.20191002-1
pve-firewall: 4.0-8
pve-firmware: 3.0-4
pve-ha-manager: 3.0-5
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 3.13.2-1
qemu-server: 6.1-1
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2

Code:

# cat /proc/101/stack
[<0>] rescuer_thread+0x2c0/0x3a0
[<0>] kthread+0x120/0x140
[<0>] ret_from_fork+0x35/0x40
[<0>] 0xffffffffffffffff

Code:

# pct config 101
arch: amd64
cores: 2
description: _______
hostname: _______
memory: 2048
net0: name=eth0,bridge=vmbr1,firewall=1,gw=10.10.10.1,gw6=2607:____:____::1,hwaddr=____________,ip=10.10.10.101/24,ip6=2607:____:____::101/64,type=veth
onboot: 1
ostype: ubuntu
protection: 1
rootfs: lvmt_containers1-nvme1:vm-101-disk-0,size=31G
swap: 1024
unprivileged: 1

Only solution: reboot. But it causes the PVE Firewall (which I am not using, I have pfsense, so it should be disabled) to hang. 8 minutes and still going.

Alwin · Dec 3, 2019

Did this resolve after the reboot?

Proxygen · Dec 3, 2019

Yes, it always goes away after a reboot. But I can't reboot - I am counting on this system for mission critical tasks with subsecond performance requirements

Alwin · Dec 4, 2019

Gaia said:
Yes, it always goes away after a reboot.

So, it shows up after some time? Could be an hardware issue as well.

Proxygen · Dec 4, 2019

Alwin said:
So, it shows up after some time? Could be an hardware issue as well.

It is not a hardware issue because it is reproducible when you shutdown a container and it fails to shutdown, and a stop order is issued. The issue has occurred before with proxmox and LXC, it is documented in this forum and other LXC discussions elsewhere.

t.lamprecht · Dec 4, 2019

This is fixed with the 5.3 kernel, please update (available on all repositories as of now) and reboot to it.

Proxygen · Dec 5, 2019

thanks, will mark as solved for now.

Proxygen · Jan 13, 2020

Still happening on PVE 5.3.13-1

Abyss · Feb 1, 2020

Same problem on proxmox-ve: 6.1-2 (running kernel: 5.3.10-1-pve)
Seems a historical bug of linux kernel

https://github.com/moby/moby/issues/5618

Proxygen · Feb 9, 2020

still happening on 5.3.10-2-pve

somebody2000 · Jul 17, 2020

yep - still there...

Code:

root@pve1:~# uname -a
Linux pve1 5.4.41-1-pve #1 SMP PVE 5.4.41-1 (Fri, 15 May 2020 15:06:08 +0200) x86_64 GNU/Linux
root@pve1:~#
Message from syslogd@pve1 at Jul 17 11:22:53 ...
kernel:[4417919.318633] unregister_netdevice: waiting for lo to become free. Usage count = 1

Message from syslogd@pve1 at Jul 17 11:23:54 ...
kernel:[4417980.021713] unregister_netdevice: waiting for lo to become free. Usage count = 1

Message from syslogd@pve1 at Jul 17 11:24:04 ...
kernel:[4417990.101528] unregister_netdevice: waiting for lo to become free. Usage count = 1

Proxygen · Jul 17, 2020

Still there for me. Chalked it off to proxmox bug I have to live with. I need a reboot every time it happens.

Stoiko Ivanov · Jul 17, 2020

hmm - we could not deterministically reproduce this here - any specifics common to your containers (and environment):
for example:
* do all your containers/setups have ipv6 addresses?
* do you use the pve-firewall?
** any particular rulesets if yes
* any specific services inside the containers (i.e. which applications runs inside them)?

That would help us in trying to reproduce this.
Thanks!

Proxygen · Jul 17, 2020

Yes to IPv6
No pve-firewall. PF Sense as a guest VM for firewalling.
Is the rulesets question applicable if I am not using pve-firewall?
Yes. Can't disclose them publicly, happy to share privately. But prior experience with this situation (not being able to disclose privately the name of the service running) brings Tom saying I should purchase support. So I will guess this is the end of the road for me, again.

Stoiko Ivanov · Jul 20, 2020

Gaia said:
Yes to IPv6
No pve-firewall. PF Sense as a guest VM for firewalling.

guess that can serve as a starting point - I'll try to reproduce it with a IPv6 setup[
- I assume the node and containers are dual-stack (ipv4+ipv6) and not ipv6-only?
- if yes - are both ipv4 and ipv6 configured on the same networkinterface in the container? (and if no do you have 2 interfaces connected to the same bridge)?

Proxygen · Jul 20, 2020

Yep, they are dual stack.
Here's one of them https://i.imgur.com/UVKr6XS.png

thanks!

ulvida · Aug 10, 2020

Hello Stoiko, hello Gaia. Same problem here, or almost: a similar kernel message:

Code:

root@asado:~#
Message from syslogd@asado at Aug 10 13:15:36 ...
 kernel:[8278276.499346] unregister_netdevice: waiting for lo to become free. Usage count = 1

Message from syslogd@asado at Aug 10 13:15:46 ...
 kernel:[8278286.643117] unregister_netdevice: waiting for lo to become free. Usage count = 1

...

but not exactly in the same situation: it's not in the node after a failed shutdown of a container, but inside some of the running containers.

Answering the questions:
> * do all your containers/setups have ipv6 addresses?
yes, we manage IPv6 on double stack in all hosts and nodes,

> * do you use the pve-firewall?
yes we do.

** any particular rulesets if yes
Nothing very special, fw for IPv54 and IPv6

> * any specific services inside the containers (i.e. which applications runs inside them)?
The reported error is on a debian 9 with AlternC (LAMP hosting panel)

Our configuration:

Code:

root@guri:~# pveversion -v
proxmox-ve: 6.1-2 (running kernel: 5.3.18-3-pve)
pve-manager: 6.1-8 (running version: 6.1-8/806edfe1)
pve-kernel-helper: 6.1-8
pve-kernel-5.3: 6.1-6
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.13-2-pve: 5.3.13-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-17
libpve-guest-common-perl: 3.0-5
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 4.0.1-pve1
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-23
pve-docs: 6.1-6
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.0-10
pve-firmware: 3.0-7
pve-ha-manager: 3.0-9
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-4
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-7
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

Stoiko Ivanov · Aug 11, 2020

ulvida said:
but not exactly in the same situation: it's not in the node after a failed shutdown of a container, but inside some of the running containers.

this sounds a bit odd - at least compared to the other users experiencing this.

In any case your system is outdated - please install the latest updates and see if that improves things
Thanks!

Proxygen · Sep 28, 2020

Still happening on

Code:

proxmox-ve: 6.2-1 (running kernel: 5.4.60-1-pve)
pve-manager: 6.2-11 (running version: 6.2-11/22fb4983)
pve-kernel-5.4: 6.2-6
pve-kernel-helper: 6.2-6
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.60-1-pve: 5.4.60-2
pve-kernel-5.4.55-1-pve: 5.4.55-1
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-5.3.18-3-pve: 5.3.18-3
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve2
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-2
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-1
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-6
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-12
pve-cluster: 6.1-8
pve-container: 3.1-13
pve-docs: 6.2-5
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-2
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-1
pve-qemu-kvm: 5.0.0-13
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-14
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve1

It happens everytime an LXC using IPv6 is shutdown.

mbosma · Mar 4, 2021

Just wanted to note I'm having the same issue on pve 6.3-1:

Code:

proxmox-ve: 6.3-1 (running kernel: 5.4.60-1-pve)
pve-manager: 6.3-3 (running version: 6.3-3/eee5f901)
pve-kernel-5.4: 6.3-3
pve-kernel-helper: 6.3-3
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.60-1-pve: 5.4.60-2
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.0-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.7
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-3
libpve-guest-common-perl: 3.1-4
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-6
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.8-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-5
pve-cluster: 6.2-1
pve-container: 3.3-3
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-8
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-5
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1

"unregister_netdevice: waiting for lo to become free" after container shutdown

Active Member

Proxmox Retired Staff

Active Member

Proxmox Retired Staff

Active Member

Proxmox Staff Member

Active Member

Active Member

Member

Active Member

Member

Active Member

Proxmox Staff Member

Active Member

Proxmox Staff Member

Active Member

Member

Proxmox Staff Member

Active Member

Renowned Member