Dropped packets inside CT

[GastonJ]

Really odd one. I've been receiving alerts for dropped packets for a while and thought nothing much of it really. Inside the CT (just restarted)

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.220 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::e86c:14ff:fe27:e01 prefixlen 64 scopeid 0x20<link>
ether ea:6c:14:27:0e:01 txqueuelen 1000 (Ethernet)
RX packets 320 bytes 70000 (68.3 KiB)
RX errors 0 dropped 38 overruns 0 frame 0
TX packets 261 bytes 116764 (114.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 1714 bytes 90398 (88.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1714 bytes 90398 (88.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 52:54:00:27:25:72 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

I get a lot of dropped packets when I start using it in anger (the CT hosts one of my Oracle database instances)

If I run up Oracle - this is just starting the listener and DB, my dropped packets have increased

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.220 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::e86c:14ff:fe27:e01 prefixlen 64 scopeid 0x20<link>
ether ea:6c:14:27:0e:01 txqueuelen 1000 (Ethernet)
RX packets 2021 bytes 245810 (240.0 KiB)
RX errors 0 dropped 490 overruns 0 frame 0
TX packets 839 bytes 202530 (197.7 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 1862 bytes 112306 (109.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1862 bytes 112306 (109.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 52:54:00:27:25:72 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

and that's before I do anything really.

Inside the CT, the last 4 hours of sar


14:00:03 all 0.41 0.52 0.83 0.01 0.00 98.23
14:10:31 all 0.42 0.48 0.82 0.10 0.00 98.18
14:20:24 all 0.40 0.49 0.81 0.01 0.00 98.28
14:30:03 all 0.34 0.48 0.81 0.01 0.00 98.37
14:40:31 all 0.39 0.48 0.84 0.01 0.00 98.29
14:50:27 all 0.38 0.50 0.80 0.01 0.00 98.31
15:00:03 all 0.40 0.51 0.85 0.01 0.00 98.23
15:10:00 all 0.38 0.55 0.82 0.15 0.00 98.11
15:20:31 all 0.37 0.63 0.86 0.08 0.00 98.07
15:30:01 all 0.40 0.81 0.91 0.01 0.00 97.87
15:40:04 all 0.42 0.55 0.85 0.01 0.00 98.18
15:50:30 all 0.37 0.56 0.86 0.01 0.00 98.20
16:00:01 all 0.40 0.54 0.85 0.01 0.00 98.21
16:10:07 all 0.37 0.48 0.83 0.01 0.00 98.30
16:20:31 all 0.30 0.48 0.79 0.01 0.00 98.43
16:30:01 all 0.29 0.48 0.77 0.02 0.00 98.44
16:40:10 all 5.31 0.48 0.74 0.03 0.00 93.44
16:50:31 all 8.33 0.49 0.86 0.01 0.00 90.29
17:00:01 all 0.66 0.51 0.82 0.01 0.00 97.99
17:10:14 all 0.48 0.45 0.80 0.01 0.00 98.25
17:20:00 all 0.29 0.48 0.77 0.01 0.00 98.45
17:30:01 all 0.27 0.48 0.75 0.01 0.00 98.50
17:40:18 all 0.31 0.47 0.77 0.01 0.00 98.44
17:50:05 all 0.31 0.57 0.85 0.16 0.00 98.12
18:00:02 all 0.37 0.39 0.80 0.20 0.00 98.23
18:10:23 all 0.30 0.40 0.81 0.08 0.00 98.41
18:20:10 all 0.36 0.45 0.86 0.01 0.00 98.33
18:30:02 all 5.52 0.43 0.88 0.01 0.00 93.15
Average: all 3.86 0.53 0.77 0.04 0.00 94.80

CPU utilisation isn't even breaking into a sweat - I assigned 4 cores to the CT, and 8GB RAM.

[oracle@node1 ~]$ mpstat -A
Linux 5.15.74-1-pve (node1) 12/21/22 _x86_64_ (24 CPU)

18:48:55 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
18:48:55 all 2.49 0.25 2.15 0.08 0.00 0.00 0.00 0.13 0.00 94.91
18:48:55 0 2.70 0.52 2.32 0.13 0.00 0.00 0.00 0.17 0.00 94.17
18:48:55 1 3.36 0.48 2.43 0.09 0.00 0.00 0.00 0.05 0.00 93.59
18:48:55 2 2.41 0.00 2.14 0.04 0.00 0.01 0.00 0.03 0.00 95.39
18:48:55 3 1.50 0.00 1.70 0.05 0.00 0.00 0.00 0.27 0.00 96.48


As I write this, more dropped packets - eth0 on teh CT

RX errors 0 dropped 825 overruns 0 frame 0

Start off an Oracle process - that does use tehnetwork and

Now at

RX errors 0 dropped 981 overruns 0 frame 0

The Proxmox node is installed on a HP DL380G7 with 192GB RAM and 2 6 core Xeons and hyperthreading is enabled, all disks are SSD's so I shouldn't really be having issues with CPU utilisation causing network issues. Hardware network details below

enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 3c:4a:92:74:d6:90 txqueuelen 1000 (Ethernet)
RX packets 39272445 bytes 38941459131 (36.2 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6635142 bytes 1530289527 (1.4 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0


vmbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.3 netmask 255.255.255.0 broadcast 0.0.0.0
inet6 fe80::3e4a:92ff:fe74:d690 prefixlen 64 scopeid 0x20<link>
ether 3c:4a:92:74:d6:90 txqueuelen 1000 (Ethernet)
RX packets 3946318 bytes 1583537769 (1.4 GiB)
RX errors 0 dropped 1381046 overruns 0 frame 0
TX packets 150154 bytes 161613135 (154.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0


root@pve2:~# lspci |grep -i eth
03:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
03:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
04:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
04:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

Any ideas welcome.

Thanks

 

[mira]

Dropped packets usually packets were received but not processed [0].

Do you send lots of small packets or could you profit from setting MTU to 9000 (if your switch supports it)?



[0] https://docs.kernel.org/networking/statistics.html

 

[GastonJ]

Thank you, much appreciated, I'll check that out.
It doesn't only happen to CT's. I ran a new VM up. Same result, dropped packets inside that as well. That's a little bit worrying in that I virtualised my firewall (pfsense) a month or so ago into a Proxmox VM.

I'm now looking at the NIC's themselves and the firmware, in case I missed anything. Looking at the NetXtreme II BCM5709 I have seen mention of firmware for Debian https://packages.debian.org/bullseye/firmware-bnx2x but I don't want to install anything that may corrupt my version of Proxmox. That doesn't seem to include anything for a BCM5709 anyway. This one does though https://packages.debian.org/bullseye/firmware-bnx2

Thanks

 

[GastonJ]

Just updated and restarted.

Drops on the bridges not on real NICS. So doesn't look like a hardware problem at least.

Only my pfsense firewall is up and running vmbr1 (enp4s0f0) vmbr2 (enp4s0f1) - so no CT's running. pfsense using VirtIO (paravirtualized)

vmbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.3 netmask 255.255.255.0 broadcast 0.0.0.0
inet6 fe80::3e4a:92ff:fe74:d690 prefixlen 64 scopeid 0x20<link>
ether 3c:4a:92:74:d6:90 txqueuelen 1000 (Ethernet)
RX packets 12287 bytes 4124907 (3.9 MiB)
RX errors 0 dropped 1017 overruns 0 frame 0
TX packets 4989 bytes 4711469 (4.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vmbr1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::3e4a:92ff:fe74:d694 prefixlen 64 scopeid 0x20<link>
ether 3c:4a:92:74:d6:94 txqueuelen 1000 (Ethernet)
RX packets 7 bytes 460 (460.0 B)
RX errors 0 dropped 2 overruns 0 frame 0
TX packets 38 bytes 18881 (18.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vmbr2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::3e4a:92ff:fe74:d696 prefixlen 64 scopeid 0x20<link>
ether 3c:4a:92:74:d6:96 txqueuelen 1000 (Ethernet)
RX packets 4985 bytes 668588 (652.9 KiB)
RX errors 0 dropped 1016 overruns 0 frame 0
TX packets 41 bytes 19785 (19.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0


Nothing dropped on the real NICS

enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 3c:4a:92:74:d6:90 txqueuelen 1000 (Ethernet)
RX packets 26193 bytes 9699997 (9.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8377 bytes 6329470 (6.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp4s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 3c:4a:92:74:d6:94 txqueuelen 1000 (Ethernet)
RX packets 17669 bytes 11796807 (11.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 12687 bytes 3186315 (3.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp4s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 3c:4a:92:74:d6:96 txqueuelen 1000 (Ethernet)
RX packets 29797 bytes 7943427 (7.5 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 20379 bytes 12775514 (12.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

 

[GastonJ]

I am beginning to wonder whether this is "normal" - with each bridge dropping packets that are not destined for a CT/VM on that bridge, but are destined for one of the other bridges? Anyone

Cheers

 

[mira]

It could be similar to this: https://serverfault.com/a/1016113
In that case those were unknown ethernet frames which could not be handled and were dropped.

 

[GastonJ]

I've had a quick look, and I suspect that is the case - the dropped packets don't actually affect anything as such and I only really "noticed" them due to netdata alerting me while I'm testing netdata.

I have a bit more of a life than to sit there sifting through my network traffic for something that isn't materially interfering with my VM/CT's and various servers I have running.

When I have the inclination to do so I'll drop the myriad of daft devices I have on my network, one at a time, until the dropped packets stop growing to identify the culprit and sort it once and for all.

Thank you for your input, it has certainly helped.

Regards,
Pete