[SOLVED] VM cannot access another VM in Proxmox 7.4

[jsabater]

Hey everyone!

I have a Proxmox 7.4 cluster with several nodes. Across them, there are two VMs, live and test, both based on Ubuntu 18.04, both with a private IP address for communication among LXCs and VMs, and with a public IP address to access the Internet. Firewall is open for specific ports only for the private network, default INPUT policy set to DROP. Then I also have a LXC acting as my Ansible Controller, based on Debian 12.

I have a weird situation in which VM live can connect to the VM test, but the VM test cannot connect to the VM live. Both can be connected from the Ansible Controller ansible (and, incidentally, from the LXC with the NGINX acting as reverse proxy). It looks like a routing or firewall problem, but I just cannot find the source of the issue. Even weirder is that VM test can ping VM live. Well, all of them can ping any other guest in the cluster, actually.

NMap says the port appears as filtered. I have disabled the firewall on the VM live but, still, the VM test cannot connect. What can I do to debug this issue? All nodes are dedicated servers at Hetzner using their vSwitch service (maybe the vSwitch went crazy for the MAC address of the VM test, somehow?).

root@test:~# ping -c 3 live.domain.com
PING live.domain.com (192.168.0.245) 56(84) bytes of data.
64 bytes from live.domain.com (192.168.0.245): icmp_seq=1 ttl=64 time=25.3 ms
64 bytes from live.domain.com (192.168.0.245): icmp_seq=2 ttl=64 time=25.3 ms
64 bytes from live.domain.com (192.168.0.245): icmp_seq=3 ttl=64 time=25.2 ms

--- live.domain.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 25.289/25.341/25.381/0.187 ms

root@test:~# nmap -p ssh live.domain.com

Starting Nmap 7.60 ( https://nmap.org ) at 2025-01-23 10:47 UTC
Nmap scan report for live.domain.com (192.168.0.245)
Host is up (0.00048s latency).

PORT   STATE    SERVICE
22/tcp filtered ssh
MAC Address: C6:5F:8D:55:31:3D (Unknown)

Nmap done: 1 IP address (1 host up) scanned in 0.59 seconds

root@test:~# traceroute live.domain.com
traceroute to live.domain.com (192.168.0.245), 30 hops max, 60 byte packets
 1  * * *
[..]
30  * * *

 

[jsabater]

For future reference, after hours of log-checking and dealing with firewall rules, network addresses, and all sort of tests using nmap, ping, traceroute, and more, it turned out that the firewall on the node where the destination VM was had turned crazy. Fortunately, the solution was quick and easy:

systemctl restart pve-firewall.service

But now I'll never know what really happened...