Problem with traffic between VMs on same Host

[poly02]

Hi everyone,

I’m running into a weird networking issue with Proxmox and I’m not sure if I’m hitting a bug or if I misconfigured something. Maybe someone has seen this before.

Setup:

• Proxmox VE 8.4.11 (Enterprise-Repo)
• Multiple nodes in a cluster
• Linux bridges (vmbr0) for VM networking
• Firewall enabled at Datacenter level
• No special per-VM firewall rules, just the default Proxmox chains

The problem:
When two VMs are on the same host, they sometimes can’t talk to each other. For example:

curl -v https://vm2

From VM1 → VM2 will succeed maybe 1–2 times, and then hang/timeout on the 3-4 try.
If I disable the Proxmox firewall, it works fine.

If I migrate VM2 to a different Proxmox host and let the firewall enabled everything works also fine.

So it seems firewall-related, but only when traffic stays inside the same host’s bridge.


iptables output (shortened):

Chain PVEFW-FORWARD (1 references)<br>target prot opt source destination
DROP       all  --  anywhere             anywhere   ctstate INVALID
ACCEPT     all  --  anywhere             anywhere   ctstate RELATED,ESTABLISHED
PVEFW-FWBR-IN  all -- anywhere           anywhere   PHYSDEV match --physdev-in fwln+ --physdev-is-bridged
PVEFW-FWBR-OUT all -- anywhere anywhere PHYSDEV match --physdev-out fwln+ --physdev-is-bridged

The chains PVEFW-FWBR-IN and PVEFW-FWBR-OUT are basically empty.

Disabling this rule fixes the issue:
iptables -D PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP

Looks like some packets between VMs on the same host get marked as INVALID by conntrack and dropped.

I logged the packets marked as INVAILD:

2025-08-20T13:25:20.996227+02:00 artos kernel: [12510.499823] INVALID_DROP: IN=vmbr871 OUT=vmbr871 PHYSIN=tap103i0 PHYSOUT=eno8303 MAC=84:b8:02:5a:24:00:bc:24:11:e3:b3:be:08:00 SRC=10.10.x.x DST=131.15.x.x LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=22716 DF PROTO=TCP SPT=52866 DPT=443 WINDOW=21 RES=0x00 ACK URGP=0

Questions:

Is this a known issue with Proxmox firewall / conntrack on bridged traffic?
Is there a proper way to keep the firewall enabled but avoid dropping these “false positive” INVALID packets?

 

[_gabriel]

vmbr871

is using Masquerading NAT ?
if yes, don't forget additionnal iptables rule
PREROUTING -i fwbr+ ... --zone 1
https://pve.proxmox.com/wiki/Network_Configuration

 

[poly02]

Thanks for checking!
We are not using Masquerading NAT, our setup is a bridged one.
Everything is running in bridge mode, so additional iptables PREROUTING rules for NAT should not needed, or did I missed something?

 

[_gabriel]

post your /etc/network/interfaces

 

[poly02]

post your /etc/network/interfaces

auto lo
iface lo inet loopback

iface eno8303 inet manual

auto vmbr871
iface vmbr871 inet static
    address 131.15.x.x/26
    gateway 131.15.x.x
    bridge-ports eno8303
    bridge-stp off
    bridge-fd 0

 

[_gabriel]

What about this ip ?

SRC=10.10.x.x

 

[poly02]

This is a bit tricky. We are also just end-users of the data center. We use multiple IP ranges on this interface: 131.15.x.x/26 and 10.10.x.x/24, both provided by the data center. No NAT is involved. Outbound traffic always goes through the interface in the 131.15.X.X network to the router, which naturally sends it back immediately.

For testing, I recently added:
ip route add 10.10.X.X/24 dev vmbr871

This prevents traffic from going through the router. However, the behavior remains the same:
curl -v https://10.10.x.x randomly fails every second or third attempt.

I don’t understand why this happens randomly. It’s not a clearly reproducible problem.