[Bug?] OVS Bridge Full Mesh - VMs in bridge cannot communicate with each other

[mnovi]

Hello,

I'm using OVS configured for supporting 3 node cluster with full mesh connection (it's basically copied from wiki and it works perfectly):

auto lo
iface lo inet loopback

iface ens20 inet manual

auto vmbr0
iface vmbr0 inet static
        address  192.168.2.51
        netmask  255.255.240.0
        gateway  192.168.2.1
        bridge_ports ens20
        bridge_stp off
        bridge_fd 0

auto ens18
iface ens18 inet manual
    ovs_type OVSPort
    ovs_bridge vmbr1
    ovs_options other_config:rstp-enable=true other_config:rstp-path-cost=150 other_config:rstp-port-admin-edge=false other_config:rstp-port-auto-edge=false other_config:rstp-port-mcheck=true vlan_mode=native-untagged

auto ens19
iface ens19 inet manual
    ovs_type OVSPort
    ovs_bridge vmbr1
    ovs_options other_config:rstp-enable=true other_config:rstp-path-cost=150 other_config:rstp-port-admin-edge=false other_config:rstp-port-auto-edge=false other_config:rstp-port-mcheck=true vlan_mode=native-untagged

auto vmbr1
iface vmbr1 inet static
    address 10.15.15.50/24
    ovs_type OVSBridge
    ovs_ports ens18 ens19
    up ovs-vsctl set Bridge ${IFACE} rstp_enable=true other_config:rstp-priority=32768 other_config:rstp-forward-delay=4 other_config:rstp-max-age=6
    post-up sleep 10

Along that config I'm also using PVE Firewall and I had to configure rules on each node to accept traffic from the 10.15.15.0/24 network, which I did with ipset named cluster_net and then added the following rule:

IN ACCEPT -i vmbr1 -source +cluster_net -dest +cluster_net

With that configuration in place communication between nodes works. The next step after that is to create VMs which are intended to communicate over full mesh network - those are for running k8s (1x server + 1x agent on each server node currently) and also have additional (virtual) NIC for access to internet (connected to vmbr0). Firewall is enabled at all levels, first at DC level, then at server node level, VM level and (every) NIC level. The problem which appears there is when trying to communicate from one VM with other VMs (doesn't matter if VMs are on the same server node or not) on the cluster_net network. I also added FW rules to each VM:

IN ACCEPT -source +cluster_net -dest +cluster_net

For each VM there are the following network interfaces on the host (in the following list 105 is VMID of one of the VMs):
- fwln105o1
- fwbr105i1
- tap105i1

fwln105o1 is also connected to OVS Bridge:

Bridge vmbr1
    Port ens18
        Interface ens18
    Port vmbr1
        Interface vmbr1
            type: internal
    Port fwln105o1
        Interface fwln105o1
            type: internal
    Port ens19
        Interface ens19
ovs_version: "2.15.0"

The problem is that I cannot get anything over this network to other VMs, but the VMs are able to communicate with server nodes. As soon as I turn off NIC firewall for the VMs connected to cluster_net, they can communicate with each other.

I was "debugging" this with tcpdump, but the packets from one VM simply don't get over to the other. I don't know where in the process they are dropped - I enabled logging on PVE FW and created custom rules to log the packets before they are dropped in each chain. There was no any indication (no helpful logs which would disclose anything related to the problem) from any chain where this happens, so I don't have any idea what can I do to come closer to the root of the problem or how to debug this.

Any help would be greatly appreciated. Thank you in advance!

 

[mnovi]

I'm still trying to resolve this problem. Does anyone have some idea what should I check in order to come closer to the root of the problem?