Packet loss with Firewall enabled in Network device

[systemctl]

Hi everyone
I have multiple kvm vms on node, bridge configured. Today i've faced some issue with packet loss on different guests, after some investigation I've came to that if i turn of Firewall support ping is fine.
Swapping network device model did nothing ( i use virio by default ).
Checked icmp packets from host to tapIDi0 device, indeed some packets did lost

Request timeout for icmp_seq 8
Request timeout for icmp_seq 29

14:13:49.269434 IP  > : ICMP echo request, id 47844, seq 7, length 64
14:13:49.269617 IP  > : ICMP echo reply, id 47844, seq 7, length 64
14:13:51.269025 IP  > : ICMP echo request, id 47844, seq 9, length 64
14:13:51.269095 IP  > : ICMP echo reply, id 47844, seq 9, length 64

14:14:10.351452 IP  > : ICMP echo request, id 47844, seq 28, length 64
14:14:10.351517 IP  > : ICMP echo reply, id 47844, seq 28, length 64
14:14:12.371415 IP  > : ICMP echo request, id 47844, seq 30, length 64
14:14:12.371500 IP  > : ICMP echo reply, id 47844, seq 30, length 64

With firewall enabled i have also this interfaces:
fwbr,fwpr,fwln

And as far as i know fwbr is interface where some kind of filtering are happen. But when i try to check icmp packets there its empty, only ARP request.

Can someone give me a tip how can i check icmp path though the host to find how Firewall option affects packet loss?

Spoiler: pveinfo

# pveversion -v
proxmox-ve: 7.2-1 (running kernel: 5.15.35-3-pve)
pve-manager: 7.2-5 (running version: 7.2-5/12f1e639)
pve-kernel-5.15: 7.2-5
pve-kernel-helper: 7.2-5
pve-kernel-5.15.35-3-pve: 5.15.35-6
ceph-fuse: 14.2.21-1
corosync: 3.1.2-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
libknet1: 1.21-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-2
libpve-storage-perl: 7.2-5
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.12-1
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.3-1
proxmox-backup-file-restore: 2.2.3-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-1
pve-container: 4.2-1
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.4-2
pve-ha-manager: 3.3-4
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-10
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1

 

[systemctl]

I have an update.
Here is ICMP packet number 12 traffic ( Firewall enabled )

15:26:02.529428 eno1 P IP MYPC > VM: ICMP echo request, id 18150, seq 12, length 64
15:26:02.529441 fwpr92518p0 Out IP MYPC > VM: ICMP echo request, id 18150, seq 12, length 64
15:26:02.529444 fwln92518i0 P IP MYPC > VM: ICMP echo request, id 18150, seq 12, length 64
15:26:02.529452 tap92518i0 Out IP MYPC > VM: ICMP echo request, id 18150, seq 12, length 64
15:26:02.529566 tap92518i0 P IP VM > MYPC: ICMP echo reply, id 18150, seq 12, length 64
15:26:02.529582 fwln92518i0 Out IP VM > MYPC: ICMP echo reply, id 18150, seq 12, length 64
15:26:02.529588 fwpr92518p0 P IP VM > MYPC: ICMP echo reply, id 18150, seq 12, length 64
15:26:02.529594 eno1 Out IP VM > MYPC: ICMP echo reply, id 18150, seq 12, length 64

As we can see everything fine.
And here is 13 packet with Request timeout for icmp_seq 13 error

15:26:03.534604 eno1  P   IP MYPC > VM: ICMP echo request, id 18150, seq 13, length 64
15:26:03.534609 fwpr92518p0 Out IP MYPC > VM: ICMP echo request, id 18150, seq 13, length 64

aaaaaand...it's gone.
fwpr92518p0 just ate it.

Guest machine didn't receive packed 13 at all. So he disappeared between fwpr92518p0 and guest.

what should i check next ?

 

[systemctl]

Any tips?
Btw found out that one of the network card channels sends a lot more interrupts than other ones. Like 10x more. So one of the softirq channels always at top. Firmware update did nothing, irqbalance too...