[SOLVED] New Guests (CT/VM) do not have working network anymore.

[Yoshike]

Hi,
My current running VMs/CTs do have working networking.
I spend some time trying to create new CTs and even a few VMss but along the way newly created guests do not have networking anymore e.g. unable to ping anything only the Guest IP itself.
The Proxmox host is able to ping the Guest,
The Guest cannot ping the Proxmox Host.

I've tried settings a Static IPv4,
aswell DHCP.
neither off those things work.
Note that this did work before and is still working for the other CT/VMs.
Its just not working anymore for new created CT/VMs.

My 2 Proxmox nodes are in a cluster.
The other node is working fine (runs a older kernel at the moment).
I've tried rebooting the non working node without any change.

Running Kernel:

root@prox-sara:/var/lib/lxc/101# pveversion -v
proxmox-ve: 5.3-1 (running kernel: 4.15.18-11-pve)
pve-manager: 5.3-9 (running version: 5.3-9/ba817b29)
pve-kernel-4.15: 5.3-2
pve-kernel-4.15.18-11-pve: 4.15.18-33
pve-kernel-4.15.18-10-pve: 4.15.18-32
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-46
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-38
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-3
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-2
openvswitch-switch: 2.7.0-3
proxmox-widget-toolkit: 1.0-22
pve-cluster: 5.0-33
pve-container: 2.0-34
pve-docs: 5.3-2
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-17
pve-firmware: 2.0-6
pve-ha-manager: 2.0-6
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 3.10.1-1
qemu-server: 5.0-46
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1

Here is the brctl on the node with the issues:
The container that do not have working internet has the tag '101'
As you can see its included in the bridge.

bridge name     bridge id               STP enabled     interfaces
vmbr0           8000.009c02a5a8f3       no              bond0
                                                        tap100i0
                                                        veth101i0
                                                        veth104i0
                                                        veth110i0

Here is the host's interface file:

auto lo
iface lo inet loopback

iface ens1 inet manual

iface enp2s0 inet manual

iface eno1 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves eno1 enp2s0 ens1
        bond-miimon 100
        bond-mode balance-rr

auto vmbr0
iface vmbr0 inet static
        address  192.168.1.10
        netmask  255.255.255.0
        gateway  192.168.1.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0

LXC config from the working LXC container:

root@prox-sara:/var/lib/lxc/104# cat config
lxc.arch = amd64
lxc.include = /usr/share/lxc/config/debian.common.conf
lxc.apparmor.profile = generated
lxc.apparmor.raw = deny mount -> /proc/,
lxc.apparmor.raw = deny mount -> /sys/,
lxc.monitor.unshare = 1
lxc.tty.max = 2
lxc.environment = TERM=linux
lxc.uts.name = Observium
lxc.cgroup.memory.limit_in_bytes = 1073741824
lxc.cgroup.memory.memsw.limit_in_bytes = 1610612736
lxc.cgroup.cpu.shares = 1024
lxc.rootfs.path = /var/lib/lxc/104/rootfs
lxc.net.0.type = veth
lxc.net.0.veth.pair = veth104i0
lxc.net.0.hwaddr = D2:F9:BD:CB:4C:B6
lxc.net.0.name = eth0
lxc.cgroup.cpuset.cpus = 3

Non working one:

root@prox-sara:/var/lib/lxc/101# cat config
lxc.arch = amd64
lxc.include = /usr/share/lxc/config/debian.common.conf
lxc.apparmor.profile = generated
lxc.apparmor.raw = deny mount -> /proc/,
lxc.apparmor.raw = deny mount -> /sys/,
lxc.monitor.unshare = 1
lxc.tty.max = 2
lxc.environment = TERM=linux
lxc.uts.name = Plex
lxc.cgroup.memory.limit_in_bytes = 1073741824
lxc.cgroup.memory.memsw.limit_in_bytes = 1610612736
lxc.cgroup.cpu.shares = 1024
lxc.rootfs.path = /var/lib/lxc/101/rootfs
lxc.net.0.type = veth
lxc.net.0.veth.pair = veth101i0
lxc.net.0.hwaddr = 5A:D1:48:45:DF:BC
lxc.net.0.name = eth0
lxc.cgroup.cpuset.cpus = 0-3

No difference really.

There is no Firewall running on the node or cluster.

root@prox-sara:/var/lib/lxc/104# iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination    

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination    

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

I've spend some time running tcpdump on the hosts TAP devices and running dhclient in the non working (newly) created CT/VM

I can see ARP Requests like:

02:35:07.168285 ARP, Request who-has 192.168.10.1 tell 192.168.10.140, length 46
02:49:28.708869 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 2a:8e:f3:42:98:c6 (oui Unknown), length 300

I should note that 192.168.10.0/24 is a working subnet and the HOST is able to function in that subnet it makes no difference if i switch the CT or VM to DHCP which will default to request a IP from the router in the 192.168.1.0/24 subnet.
Or for the heck of it a static IPv4 in the 192.168.1.0/24 subnet.

I can verify that in fact LAN networking is working at the TAP level,
e.g. there is lots of stuff in tcpdump..

Thanks,
Yosi.

 

[Richard]

Problem still open?

For analysis it would be clearer to use static IP (rather than DHCP).

Follow the packets via tcpdump step by step and check where they get lost, e.g. guest-virtual-nic -> host-bridge-port -> host-bridge

As soon as this point is localized have a look to the responsible configuration.

 

[Yoshike]

I've fixed the issue.
By changing the bond-mode from

 bond-mode balance-rr

to:

 bond-mode active-backup

As my switch does not support roundrobin.