[SOLVED] All ports of LXCs are suddenly blocked! VMs work fine.

[PMXUsername]

Everything was working fine initially and all of a sudden I'm facing this rather bizarre issue (only) with LXCs (VMs work fine). As an example, I cannot SSH to the LXC from host but can SSH to it from a different LAN device. I know I can enter the LXC using pct enter <id> but the point is its not just SSH, every other port - I have installed Apache on 8080 and even that cannot be accessed from the host. This applies to ALL the LXCs I have and they were all working fine before.

I even deleted and restored the LXCs from backup and it did not fix. I even created a brand new container from Debian 10 template and even that has exact same issue. So looks like something is messed up with my Proxmox host itself. And ofcourse, I have also restarted the node million times. And I have also tried turning off Firewall in datacenter view and also for each LXC. Any help will be immensely appreciated!

pveversion -v
proxmox-ve: 6.2-2 (running kernel: 5.4.65-1-pve)
pve-manager: 6.2-12 (running version: 6.2-12/b287dd27)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-9
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 0.9.1-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-2
pve-cluster: 6.2-1
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-1
pve-qemu-kvm: 5.1.0-3
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-15
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve2

 

[Moayad]

This issue could be firewall please post config for one LXC Container and output of pve-firewall status

 

[PMXUsername]

pve-firewall status
Status: disabled/running

I executed pve-firewall stop and the status is now Status: disabled/stopped but the issue is not fixed.

LXC Config:

arch: amd64
cores: 1
hostname: Testing
memory: 2048
mp0: /mnt/data,mp=/mnt/data,replicate=0,ro=1
nameserver: 192.168.12.253
net0: name=eth0,bridge=vmbr0,gw=192.168.12.1,hwaddr=BF:3D:36:14:46:B5,ip=192.168.12.221/32,type=veth
onboot: 1
ostype: debian
rootfs: local-lvm:vm-221-disk-0,size=8G
searchdomain: redacted.com
swap: 1024

 

[Moayad]

Could you ping the Testing LXC?

 

[PMXUsername]

Yup!

user@node:~$ ping testing
PING testing.redacted.com (192.168.12.221) 56(84) bytes of data.
64 bytes from testing.redacted.com (192.168.12.221): icmp_seq=1 ttl=63 time=0.324 ms
64 bytes from testing.redacted.com (192.168.12.221): icmp_seq=2 ttl=63 time=0.435 ms
64 bytes from testing.redacted.com (192.168.12.221): icmp_seq=3 ttl=63 time=0.275 ms
64 bytes from testing.redacted.com (192.168.12.221): icmp_seq=4 ttl=63 time=0.320 ms

When I try to SSH, there will be nothing for several seconds and then it shows Operation timed out or some message like that. Same even when I try to curl a web page thats on different port.

 

[PMXUsername]

UPDATE: If I use DHCP instead of Static IP for LXC, everything is working fine! So as a workaround I for now, I'm using DHCP with static IP assignment to that LXC from my router.

Another thing I have noticed is if I ping the LXC which is configured with DHCP, the ping time/latency is much lower than when it had Static IP!

Does this mean routing table is messed up or something on my Proxmox host?

 

[Moayad]

Does this mean routing table is messed up or something on my Proxmox host?

This depends on your network configuration.

Can you please post your network configuration.

 

[PMXUsername]

Nothing special - default vanilla setup.

/etc/network/interfaces

auto lo
iface lo inet loopback
iface eno1 inet manual
auto vmbr0
iface vmbr0 inet static
address 192.168.12.201
netmask 255.255.255.0
gateway 192.168.12.1
bridge_ports eno1
bridge_stp off
bridge_fd 0

 

[PMXUsername]

UPDATE: I fixed the issue! I had configured all my LXCs with /32 as subnet and once I changed them to /24, everything started working again! Thanks!

 

[Moayad]

Please mark the thread as [SOLVED] to help other people who have the same problem Thanks!