Debian containers regularly lose internet connection, require reboot to restore

[poisonborz]

All of my Debian Standard containers lose internet connection 1-2 times a day. They can ping local ip-s, but not internet ones. Only reboot solves it.

Journalctl shows just this:

E: Sub-process /lib/systemd/systemd-networkd-wait-online returned an error code (1)
systemd-networkd-wait-online[591]: Timeout occurred while waiting for network connectivity.

I found a number of threads with this problem, but no solution
https://forum.proxmox.com/threads/container-verliert-nach-jeder-nacht-netzwerk-konnektivität.132047/
https://forum.proxmox.com/threads/p...nnection-needs-full-reboot-to-restore.128844/

I've tried to enable networkd.service as suggested here somewhere but it doesn't help.

This does not happen with eg. Alpine containers.

What could be the problem?

 

[Moayad]

Hi,

Can you please post one of the CT configs and the network configuration and the output of `pveversion -v`?

 

[poisonborz]

Sure @Moayad, here it is

Container is Debian 12 Standard 6.2.16-3-pve #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-3 (2023-06-17T05:58Z) x86_64
Note that seemingly only Debian containers need DHCP ip settings, STATIC doesn't work (unlike Alpine containers)

I couldn't see this problem yet with Debian 11.

arch: amd64
cores: 1
features: nesting=1
hostname: NAME
memory: 512
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=<MAC>,ip=dhcp,ip6=dhcp,type=veth
onboot: 1
ostype: debian
rootfs: local-lvm:vm-250-disk-0,size=1G
swap: 512
unprivileged: 1

proxmox-ve: 8.0.1 (running kernel: 6.2.16-3-pve)
pve-manager: 8.0.3 (running version: 8.0.3/bbf3993334bfa916)
pve-kernel-6.2: 8.0.2
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx2
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-3
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.0
libpve-access-control: 8.0.3
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.5
libpve-guest-common-perl: 5.0.3
libpve-http-server-perl: 5.0.3
libpve-rs-perl: 0.8.3
libpve-storage-perl: 8.0.1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 2.99.0-1
proxmox-backup-file-restore: 2.99.0-1
proxmox-kernel-helper: 8.0.2
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.0.5
pve-cluster: 8.0.1
pve-container: 5.0.3
pve-docs: 8.0.3
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.2
pve-firmware: 3.7-1
pve-ha-manager: 4.0.2
pve-i18n: 3.0.4
pve-qemu-kvm: 8.0.2-3
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.6
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.12-pve1

 

[Moayad]

Hello,

Can you also please share the network configuration?

Do the VMs have the same issue, or only the containers?

 

[poisonborz]

Where do I find this config? The setup is the default, with a single physical eth0.
This only seems to affect Debian CT-s, I tried release 11 and 12, I don't have VMs atm.

The problem only appears after 14-20h uptime.

If I manually run dhclient eth0 the network becomes operational again.

There is nothing suspicious in etc/network/interfaces

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

iface eth0 inet6 dhcp

This journalctl log might help.

systemd[1]: Reached target Network is Online.
systemd[1]: Reached target Network.
systemd[1]: Failed to start Raise network interfaces.
systemd[1]: networking.service: Failed with result 'timeout'.
systemd[1]: networking.service: Killing process 95 (isc-timer) with signal SIGKILL.
ifup[62]: ifup: failed to bring up eth0
ifup[62]: Got signal Terminated, terminating...
systemd[1]: networking.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: networking.service: start operation timed out. Terminating.
dhclient[173]: XMT: Solicit on eth0, interval 129410ms.
ifup[173]: XMT: Solicit on eth0, interval 129410ms.
ifup[173]: XMT:  | X-- Request rebind in +5400
ifup[173]: XMT:  | X-- Request renew in  +3600
ifup[173]: XMT:  X-- IA_NA a5:f8:03:dd
ifup[173]: XMT: Forming Solicit, 266190 ms elapsed.
dhclient[173]: XMT: Solicit on eth0, interval 129250ms.
ifup[173]: XMT: Solicit on eth0, interval 129250ms.
ifup[173]: XMT:  | X-- Request rebind in +5400
ifup[173]: XMT:  | X-- Request renew in  +3600
ifup[173]: XMT:  X-- IA_NA a5:f8:03:dd
ifup[173]: XMT: Forming Solicit, 136940 ms elapsed.
dhclient[173]: XMT: Solicit on eth0, interval 67430ms.

 

[Moayad]

Hi,

The network configuration and Syslog from the Proxmox VE side not from the CT.

# Network config:

cat /etc/network/interfaces

# journactl

journalctl --since "2023-09-15 05:30" --until "2023-09-15 09:00" > /tmp/Syslog.txt

You may edit the time/date in the the journalctl above command.