L2 Firewall - Traffic disappears across bridge

[abbynormal]

I have what essentially equates to a L2 FW deployment flow, and I'm seeing some issues with the client not receiving traffic when return L2 FW rewrites the MAC with it's VE MAC. The following is my current setup, where all players exist within Proxmox:



I initially had everything on vmbr1, but in my attempts to try and resolve possible issues, I created vmbr3. The only place vmbr3 exists on Proxmox is on the two vNICs above, for the LXC and the FW KVM. So the link between the LXC and FW should be completely L2 segmented.

And to be clear, the breaking point is specifically on this link, between LXC and the FW KVM:



LXC vNIC config:


FW VM vNIC config:


I have taken three simultaneous captures to try and correlate the issue:

1. Capture on FW
2. tcpdump within LXC
3. tcpdump directly in Proxmox host, against vmbr3

I've noticed that traffic is forwarded fine unless FW rewrites source MAC to it's VE MAC for traffic from Proxy to LXC. In the captures we see this traffic on FW, and vmbr3, but never on the client. We also never see the retransmissions from the client after the first connect request:


LXC:



FW VM:



vmbr3:




This flow defiantly has me scratching my head on what the heck is going on.

• From the tcpdump against the vmbr3, it seems like we see traffic in both directions with the full expected MACs
• From the client, we never seen the return traffic when the source MAC is the FW VE interface
• From the FW, we see the initial "connect" request, however we never see any of the follow up retransmissions for the client

Any idea what would cause this behavior? My suspicion is some kind of "gotchas" in the Proxmox vSwitch for such a flow that I'm unaware about.

 

[abbynormal]

SOLVED:
Seems issue was indeed a direct result of the vswitch behavior. Above all references to vbridges were with the standard Linux bridges. As a test, I created two new bridges, each tied to one unused phyiscial NIC. Then I cabled both of these NICs together. Finally, I tied one vbridge to the client vNIC and the other to the FW vNIC.

In doing so, I saw the exact same behavior. IE client-side vbridge capture matched up with above LXC capture, and FW side vbride capture linedup with above capture taken in FW. Seems something within Linux bridge behavior does not play nicely with this L2 flow.

I installed OVS and made an OVS bridge to tie the client and FW together. Worked right away....