[SOLVED] High network utilisation after update

[Paul O'C]

I'm running Proxmox VE 8.3.3 on a Core i3 NUC with dual nics. Everything was going well until I updated the installation on Wednesday and rebooted the nuc. After which the network utilisation is running at about 50M.

The two nic are configured such that one has the bridge attached to it and the other is used to connect to a NAS over iSCSI to provide the disk space.

If start the VMs then the utilisation increases to about 90M and stays there and for some of the VMs the CPU utilisation goes above 100% none of the the VMs are accessible, the console can't connect and if I ping the server or one of the VMs I get high packet lost 50%-80%. If I stop all the VMS except one then the network utilisation remains high and it is still uncontactable.

If I delete the bridge vmbr0 then the network utilisation drops to about 80k, but of course then I wouldn't be able to contact the VMs. Creating the bridge again has the network utilisation increase to the 50M mark again.

Yesterday I noticed there was a new kernel and updated again. The situation is better but it is still not working. Now the utilisation still increases to 50M and if I start a single VM it then increases to about 90M but pings don't seem to be so problematic until I start a second or a a third VM.

The two nic are identified as enp100s0 and enp101s0.

enp100s0 ==> vmbr0
enp101s0 ==> iSCSI separate network (192.168.18.0/24)

both are connected to the same switch but the storage is on it's on VLAN.

If there is anything else you need me to grab then let me know and I'll add it in.

 

[Kingneutron]

Sounds like a network driver issue or possibly bad cable. What happens if you reboot into the older kernel where everything was working?

Post results of ' lspci |grep -i ether ', we need to figure out what NIC you're using

You could try installing iptraf-ng (or wireshark, if you're familiar with it) and monitor what's going across the line when no VMs are running; also install net-tools and do a few ' ifconfig ' repeats to see output for this:

        RX packets 884089  bytes 197019205 (187.8 MiB)
        RX errors 0  dropped 15396  overruns 0  frame 0
        TX packets 636245  bytes 138352227 (131.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Also check dmesg and syslog

Workaround might be to obtain a 1G or 2.5Gbit USB3 NIC and see if that fixes anything

 

[Paul O'C]

Thanks for the reply.

The results from lspci are
64:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
65:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-V (rev 04)

I've also attached the dmesg and ifconfig output. It looks like the bridge is dropping a lot of packets for some reason.

I have tried booting from the 6.8.12-5 and 6.6.12-4 kernels but that didn't make a difference. Also I did find a USB NIC and I tried that but that also didn't seem to make a difference. For the attached files I had associated the bridge with the USB nic which was identified as enx000ec6c00161

i've swapped the bridge back to enp100s0 now and I'm using the USB nic to access the server.

I will give iptraf-ng a go tomorrow and see what that comes up with. I had tried Wireshark but on a separate machine and that didn't pick up anything which suggests whatever is happening isn't leaking outside of the proxmox server. When I have the results from iptraf-ng I'll upload the file as well.

 

[Paul O'C]

I've just carried out a capture with no VMS running and I've attached the logs. I captured both vmbr0 and enp100s0, it all looks normal now(?), although yesterday there were large numbers of drops. I'll spin up a VM and do a capture for that as well include those logs (although right now it all seems weirdly normal so I have no idea what was going wrong)

 

[Kingneutron]

Solar flare, phase of the moon, planet alignment, insufficient goat sacrifice, who knows ¯\_(ツ)_/¯

 

[Paul O'C]

yeah it's been a few days now and it seems stable so I'm going to mark this as fixed. I have no idea how... I guess one of the reboots or the general poking made it load up the right drivers or something.

Thanks for the help.