Kernel dropping packets, bug? Or misconfiguration?

[Knogle]

Ahoy friends, i hope you are doing fine.

I am currently experiencing issue with my Proxmox machine.
The system is based on an EPYC 7551P CPU, and Supermicro H11SSL-i motherboard, connected to a swich using a LACP bond.
I am experiencing massive packetloss as i am running OPNsense on the machine, especially inbound loss, around 80000 packet drops per second.

After running dropwatch, the issue seems to be purely software related, or more specific, kernel related. I was able to sort the hardware out entirely, so i won't go into more detail here.
Do you guys have any suggestions? I see the drops on the vmbr0 interface, not on bond0 nor eth0 or eth1.

After doing troubleshooting with dropwatch it seems to happen in several parts of the Kernel procedures.

I am looking forward for your help

dropwatch> start
Enabling monitoring...
Kernel monitoring activated.
Issue Ctrl-C to stop monitoring
13 drops at tun_net_xmit+306 (0xffffffffae2408f6) [software]
257 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
355 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
1 drops at ip6_rcv_core+442 (0xffffffffae575da2) [software]
409 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
40 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
54 drops at br_forward+107 (0xffffffffae5ede67) [software]
24 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
18 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
1 drops at tun_net_xmit+306 (0xffffffffae2408f6) [software]
32 drops at br_forward+107 (0xffffffffae5ede67) [software]
1 drops at ip6_mc_input+1bc (0xffffffffae576e0c) [software]
3 drops at tun_net_xmit+306 (0xffffffffae2408f6) [software]
3 drops at llc_rcv+241 (0xffffffffae463fc1) [software]
1 drops at snap_rcv+bb (0xffffffffae4654bb) [software]
23 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
287 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
18 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
46 drops at llc_rcv+241 (0xffffffffae463fc1) [software]
4 drops at br_stp_rcv+5f (0xffffffffae5f334f) [software]
8 drops at snap_rcv+bb (0xffffffffae4654bb) [software]
6 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
1 drops at tun_net_xmit+306 (0xffffffffae2408f6) [software]
1 drops at br_forward+107 (0xffffffffae5ede67) [software]
13 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
20 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
1 drops at tun_net_xmit+306 (0xffffffffae2408f6) [software]
1 drops at br_stp_rcv+5f (0xffffffffae5f334f) [software]
2 drops at icmp_rcv+205 (0xffffffffae506645) [software]
8 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
6 drops at tun_net_xmit+306 (0xffffffffae2408f6) [software]
487 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
51 drops at br_forward+107 (0xffffffffae5ede67) [software]
231 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
2 drops at ip_rcv_finish_core.constprop.0+1ce (0xffffffffae4ade0e) [software]
79 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
49 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
16 drops at tun_net_xmit+306 (0xffffffffae2408f6) [software]
4 drops at ip_rcv_finish_core.constprop.0+1ce (0xffffffffae4ade0e) [software]
20 drops at br_forward+107 (0xffffffffae5ede67) [software]
18 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
16 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
17 drops at tun_net_xmit+306 (0xffffffffae2408f6) [software]
3 drops at br_stp_rcv+5f (0xffffffffae5f334f) [software]
41 drops at llc_rcv+241 (0xffffffffae463fc1) [software]
7 drops at snap_rcv+bb (0xffffffffae4654bb) [software]
8 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
255 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
40 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
1 drops at ip_rcv_finish_core.constprop.0+1ce (0xffffffffae4ade0e) [software]
12 drops at br_forward+107 (0xffffffffae5ede67) [software]
4 drops at tun_net_xmit+306 (0xffffffffae2408f6) [software]
1 drops at br_forward+107 (0xffffffffae5ede67) [software]
9 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
2 drops at tun_net_xmit+306 (0xffffffffae2408f6) [software]
3 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
3 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
1 drops at tun_net_xmit+306 (0xffffffffae2408f6) [software]
1 drops at ip6_rcv_core+442 (0xffffffffae575da2) [software]
3 drops at ip_rcv_finish_core.constprop.0+1ce (0xffffffffae4ade0e) [software]
1 drops at tun_net_xmit+306 (0xffffffffae2408f6) [software]
3 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
1 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
4 drops at ip_rcv_finish_core.constprop.0+1ce (0xffffffffae4ade0e) [software]
3 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
1 drops at icmp_rcv+205 (0xffffffffae506645) [software]
2 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
10 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
3 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
2 drops at unix_stream_connect+350 (0xffffffffae56a150) [software]
210 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
711 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
52 drops at br_forward+107 (0xffffffffae5ede67) [software]
1 drops at ip6_mc_input+1bc (0xffffffffae576e0c) [software]
4 drops at unix_stream_connect+350 (0xffffffffae56a150) [software]
215 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
1 drops at br_stp_rcv+5f (0xffffffffae5f334f) [software]
113 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
12 drops at tun_net_xmit+306 (0xffffffffae2408f6) [software]
4 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
35 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
26 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
373 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
22 drops at br_forward+107 (0xffffffffae5ede67) [software]
46 drops at llc_rcv+241 (0xffffffffae463fc1) [software]
4 drops at br_stp_rcv+5f (0xffffffffae5f334f) [software]
4 drops at unix_stream_connect+350 (0xffffffffae56a150) [software]
8 drops at snap_rcv+bb (0xffffffffae4654bb) [software]
8 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
5 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
2 drops at tun_net_xmit+306 (0xffffffffae2408f6) [software]
3 drops at llc_rcv+241 (0xffffffffae463fc1) [software]
1 drops at snap_rcv+bb (0xffffffffae4654bb) [software]
7 drops at sk_stream_kill_queues+5d (0xffffffffae3db1ad) [software]
1 drops at unix_dgram_sendmsg+565 (0xffffffffae56ab45) [software]
1 drops at arp_error_report+42 (0xffffffffae501632) [software]
1 drops at skb_queue_purge_reason+f2 (0xffffffffae3d5f42) [software]
8 drops at br_forward+107 (0xffffffffae5ede67) [software]
121 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
178 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
1 drops at sk_stream_kill_queues+5d (0xffffffffae3db1ad) [software]
9 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
3 drops at tun_net_xmit+306 (0xffffffffae2408f6) [software]
4 drops at tun_net_xmit+306 (0xffffffffae2408f6) [software]
517 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
54 drops at br_forward+107 (0xffffffffae5ede67) [software]
1 drops at ip6_mc_input+1bc (0xffffffffae576e0c) [software]
4 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
241 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
168 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
8 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
16 drops at tun_net_xmit+306 (0xffffffffae2408f6) [software]
6 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
28 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
35 drops at br_forward+107 (0xffffffffae5ede67) [software]
370 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
2 drops at icmp_rcv+205 (0xffffffffae506645) [software]
1 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
1 drops at ip6_rcv_core+442 (0xffffffffae575da2) [software]
424 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
283 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
3 drops at br_stp_rcv+5f (0xffffffffae5f334f) [software]
16 drops at tun_net_xmit+306 (0xffffffffae2408f6) [software]
41 drops at llc_rcv+241 (0xffffffffae463fc1) [software]
7 drops at snap_rcv+bb (0xffffffffae4654bb) [software]
504 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
53 drops at br_forward+107 (0xffffffffae5ede67) [software]
1 drops at __init_scratch_end+102ab17b (0xffffffffc0eab17b) [software]
1 drops at ip6_mc_input+1bc (0xffffffffae576e0c) [software]
45 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
1 drops at __init_scratch_end+102ab17b (0xffffffffc0eab17b) [software]
77 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
97 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
13 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
359 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
7 drops at br_forward+107 (0xffffffffae5ede67) [software]
46 drops at llc_rcv+241 (0xffffffffae463fc1) [software]
4 drops at br_stp_rcv+5f (0xffffffffae5f334f) [software]
12 drops at snap_rcv+bb (0xffffffffae4654bb) [software]
6 drops at tun_net_xmit+306 (0xffffffffae2408f6) [software]
25 drops at br_forward+107 (0xffffffffae5ede67) [software]
1 drops at ip6_mc_input+1bc (0xffffffffae576e0c) [software]
1 drops at ip6_rcv_core+442 (0xffffffffae575da2) [software]
1 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
1 drops at ip6_rcv_core+442 (0xffffffffae575da2) [software]
18 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
1 drops at br_stp_rcv+5f (0xffffffffae5f334f) [software]
390 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
280 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
3 drops at llc_rcv+241 (0xffffffffae463fc1) [software]
1 drops at tun_net_xmit+306 (0xffffffffae2408f6) [software]
1 drops at snap_rcv+bb (0xffffffffae4654bb) [software]
89 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
1 drops at icmp_rcv+205 (0xffffffffae506645) [software]
360 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
50 drops at br_forward+107 (0xffffffffae5ede67) [software]
91 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
1 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
516 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
212 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
31 drops at br_forward+107 (0xffffffffae5ede67) [software]
98 drops at tun_net_xmit+306 (0xffffffffae2408f6) [software]
1 drops at ip6_mc_input+1bc (0xffffffffae576e0c) [software]
1 drops at ip6_rcv_core+442 (0xffffffffae575da2) [software]
8 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
1 drops at br_stp_rcv+5f (0xffffffffae5f334f) [software]
592 drops at ip_rcv_core+24a (0xffffffffae4ae43a) [software]
695 drops at __netif_receive_skb_core.constprop.0+145 (0xffffffffae3f4615) [software]
3 drops at br_stp_rcv+5f (0xffffffffae5f334f) [software]
16 drops at tun_net_xmit+306 (0xffffffffae2408f6) [software]
41 drops at llc_rcv+241 (0xffffffffae463fc1) [software]
7 drops at snap_rcv+bb (0xffffffffae4654bb) [software]
194 drops at tun_net_xmit+306 (0xffffffffae2408f6) [software]
Got a stop message

 

[waltar]

Run your env without opnsense (which may first take some effort to do so).

 

[Knogle]

Run your env without opnsense (which may first take some effort to do so).

I have done so, switched to OpenWrt which i had set up already, in case something goes wrong with OPNsense VM. But exactly same issue here.
Also on another machine similar, in a different location, but same network setup.

That's my network configuration.

root@millenium-gbe31:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto eno49
iface eno49 inet manual

auto eno50
iface eno50 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves eno1 eno2
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.10/24
        gateway 192.168.1.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr0.110
iface vmbr0.110 inet manual
#MGMT

auto vmbr0.200
iface vmbr0.200 inet manual
#Virtual

auto vmbr110
iface vmbr110 inet static
        address 172.20.64.10/19
        bridge-ports vmbr0.110
        bridge-stp off
        bridge-fd 0

 

[Knogle]

Problem solved! I had issues for years, due to my nested VLAN configuration in Proxmox.