[SOLVED] Cant get OpnSense working outside of Proxmox

[Quantic Dream]

My configuration:
Proxmox machine has 7 ethernet ports, used as:
1x - Management port
1x - WAN port
x4 - LACP bond to Tp-Link managed switch
x1 - Used as physical Linux bridge port

Spoiler: /etc/network/interfaces

auto eno1
iface eno1 inet static
        address 10.0.0.7/24
#management port

auto enp4s0f0
iface enp4s0f0 inet manual
#LAN bond port

auto enp4s0f1
iface enp4s0f1 inet manual
#LAN bond port

auto enp4s0f2
iface enp4s0f2 inet manual
#LAN bond port

auto enp4s0f3
iface enp4s0f3 inet manual
#LAN bond port

auto enp3s0f0
iface enp3s0f0 inet manual
#WAN

auto enp3s0f1
iface enp3s0f1 inet manual
#LAN

auto bond0
iface bond0 inet manual
        bond-slaves enp4s0f0 enp4s0f1 enp4s0f2 enp4s0f3
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer3+4
#Bond to Switch

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0 enp3s0f1
        bridge-stp off
        bridge-fd 0
#LAN

auto vmbr1
iface vmbr1 inet manual
        bridge-ports enp3s0f0
        bridge-stp off
        bridge-fd 0
#WAN

I have created 2 VMs:
1) OpnSense, uses vmbr1 as WAN, vmbr0 as LAN
2) Ubuntu 20.04, uses vmbr0 as LAN

Both VMs has proxmox firewall disabled on ethernet devices

What works:
1) From laptop, connected to external Tp-Link switch:
a) I can access Proxmox on management port
b) I can access Proxmox on vmbr0 (bonded) if IP is configured
2) From Ubuntu VM:
a) I am receiving IP from OpnSense DHCP, and have access to internet
b) I can ping management port (works only with linux bridge, and not with OVS bridge)

What doesn't work:
1) OpnSense and ubuntu machines, are not visible from machine connected to external switch (to bonded interface)
2) External devices (connected to external switch) are not visible from VM machines

Please advice.

 

[Quantic Dream]

The problem was with bond configuration.
But it was hidden by some magic, that i do not understand ) :
Somehow, all requests targeted to bonded interface, was actually fulfilled by management port.

I have executed iperf3 tests from laptop to bond interface, and noticed activity on management port (switch led).
After disconnecting management port cable, both IPs (management, and bond) become unreachable.
After that i have realized that bond doesn't work.

After some reading about Link Aggregation i have found the main problem fix:
I am using switch TP-Link TL-SG1024DE, which support only Static Link Aggregation (Not LACP which is Dynamic Link Aggregation)
For static link aggregation, bond mode balanced-xor should be used, with some googling i have found that this Tp-Link switch uses Layer 2 hashing.