Issue with packet flow in hypervisor

[kocifi]

Hello, find out interesting issue with network flow between hypervisor and VMs maybe I have overcomplicated setup but I still thing that should work.
I tried setup BGP routing but I can't get it to work. So I started debuging with tcpdump and I saw in some cases that packet comest to hypervisor interface, than vnet20 and after that ends but it should go to tap interface for VM (in this example virtual switch)

Can anyone help me to debug/fix issue?

Networks are defined in SDN

Ping flow flow:
ping from old_VM1 to test_vm1 (default gw is set through switch2)

From hypervisor1
old_VM1 (ICMP echo request) -> tap201i0 -> tap100i3 -> old_gateway1 -> tap100i9 -> tap118i1 -> gateway1 -> tap118i2 -> ln_vnet101 -> SFP port -> ........
Hypervisor2
........ SFP port -> ln_vnet101 -> tap121i1 -> switch1 -> tap121i3 -> tap110i0 -> test_vm1 -> (ICMP echo reply) tap110i0 -> ln_vnet20 -> SFP port -> ........
Hypervisor3:
........ SFP port -> ln_vnet20 -> ends here (12:02:01.609171 ln_vnet20 P IP 10.7.120.5 > 10.0.12.11: ICMP echo reply, id 113, seq 1, length 64)

I can't see packet arrive to switch2 (even not to tap122i3 on hypervisor)

Setup:

• hypervisor1
  • 2x 10 Gb/s SFP+ (to Mikrotik switches with mlag)
  • SDN - bond0 on SFP+ ports
  • VMs:
    • old_gateway1 (id 100)
      • WAN - bridge to ethernet port on server
      • GW_link - vnet10 (bond0 vlan10)
      • VM_network - vnet5 (bond0 vlan5)
    • gateway1 (id 118)
      • WAN - bridge to ethernet port on server
      • GW_link - vnet10 (bond0 vlan10)
      • SW_Link - vnet101 (bond0 vlan101)
    • old_VM1 (id 201)
      • VM_network - vnet5 (bond0 vlan5)
    • test_vm1 (id 110)
      • VM_network - vnet20 (bond0 vlan20)
• hypervisor2
  • same as hypervisor1
  • VMs:
    • switch1 (id 121)
      • SW_Link - vnet101 (bond0 vlan101)
      • VM_network - vnet20 (bond0 vlan20)
• hypervisor3
  • same as hypervisor1
  • VMs:
    • switch2 (id 122)
      • SW_Link - vnet101 (bond0 vlan101)
      • VM_network - vnet20 (bond0 vlan20)

Between switches is vrrp on 10.7.120.1/28 (in image are wrong IPs - should be sw1 10.7.120.2 and sw2 10.7.120.3)

 

[kocifi]

Anyone have any ideas on what to try?

 

[kocifi]

Update: when everything is on the same hypervisor, the problem disappears.

When I try to ping from the switch to test_vm1 via the BGP link interface, the ping on the hypervisor disappears (from both switches when they are not on the same hypervisor with test_vm1).

 

[kocifi]

Curl from VPN (client IP: 172.31.255.2 and carp vpn server is on GW1)
Switches are manually set to prioritize sending traffic via GW2 (gw2 have route for VPN back to GW1)
vnet were disable for this test and replaced with manual vmbr10 (bond0.10), vmbr11 (bond0.11), vmbr20 (bond0.20) and whole setup was simplified:

GW_Link: 10.0.255.0/27 - vmbr10 - GW1: 10.0.255.1, GW2: 10.0.255.2
SW_Link: 10.0.255.32/27 - vmbr11 - GW1: 10.0.255.62, GW2: 10.0.255.61, SW1: 10.0.255.33, SW2: 10.0.255.34
NewVM: 10.7.120.0/28 - vmbr20 - GW_VIP: 10.7.120.1, SW1: 10.7.120.2, SW2: 10.7.120.3

tcpdump -i any src 10.7.120.5 and port 80 and dst 172.31.255.2

hypervisor1 (gw1, sw1 and test_vm1):

16:55:16.270345 tap110i0 P   IP 10.7.120.5.http > 172.31.255.2.49904: Flags [S.E], seq 4243034865, ack 2488845490, win 65160, options [mss 1460,sackOK,TS val 994311280 ecr 391179508,nop,wscale 7], length 0
16:55:16.270358 tap121i3 Out IP 10.7.120.5.http > 172.31.255.2.49904: Flags [S.E], seq 4243034865, ack 2488845490, win 65160, options [mss 1460,sackOK,TS val 994311280 ecr 391179508,nop,wscale 7], length 0
16:55:16.270503 tap121i0 P   IP 10.7.120.5.http > 172.31.255.2.49904: Flags [S.E], seq 4243034865, ack 2488845490, win 65160, options [mss 1460,sackOK,TS val 994311280 ecr 391179508,nop,wscale 7], length 0
16:55:16.270516 bond0.11 Out IP 10.7.120.5.http > 172.31.255.2.49904: Flags [S.E], seq 4243034865, ack 2488845490, win 65160, options [mss 1460,sackOK,TS val 994311280 ecr 391179508,nop,wscale 7], length 0
16:55:16.270518 bond0 Out IP8 (invalid)
16:55:16.270520 enp3s0f0 Out IP8 (invalid)

hypervisor3 (gw2 and sw2):

16:55:16.270347 eno1  P   IP8 (invalid)
16:55:16.270348 bond0 P   IP8 (invalid)
16:55:16.270349 bond0.11 P   IP 10.7.120.5.http > 172.31.255.2.49904: Flags [S.E], seq 4243034865, ack 2488845490, win 65160, options [mss 1460,sackOK,TS val 994311280 ecr 391179508,nop,wscale 7], length 0

 

[kocifi]

Update:
When I disable firewall with pve-firewall stop it starts working.
Weirdly for all interfaces is firewall disabled.

Update 2:
Knowing exactly what I'm looking for, I also found this thread: https://forum.proxmox.com/threads/please-allow-the-firewall-to-pass-layer2-asymetric-traffic.41723/