[SOLVED] Network lag / lost packets and much higher cpu usage since upgrade to PVE 8 / Linux 6.5

A

abma

Active Member
Feb 20, 2021
89
11
28
We've upgraded PVE from 7 to 8 and also the Kernel from 6.2 to 6.5.11-6-pve: since these changes, the CPU-usage of the kvm-process is much higher then before. Before the upgrade the idle process usage was ~2%, and never above ~50% now after the upgrade the idle cpu usage is at ~20% and it goes up to 150%.

When the cpu usage is high, packets get dropped and delayed: ping times are between 0.1ms and 400ms: before it was always at 0.1ms.

The vm is used as router/firewall and runs debian 12 with the default kernel (6.1.0-15-amd64).

Also iperf performance went from ~10GBit to 300Mbit. Enabling multi-queue and adding more vcpus helped a bit to increase bandwith back to ~10Gbit, but the "random" lags are still there. The vm basicly does no disk i/o, it only acts as router.

Does anyone else have this problem?

It seems packet processing is much slower after the upgrade.
 
Last edited:
Code: 
perf_6.5 record -F 99 -p 4072637 -g -- sleep 30
perf_6.5 report -n --stdio
 

Attachments

I've disabled firewall on interface level: this reduced the lag. Thats a bit weird, because firewall was already disabled at datacenter and node level.
 
abma said:
I've disabled firewall on interface level: this reduced the lag. Thats a bit weird, because firewall was already disabled at datacenter and node level.
Click to expand...
The firewall at interface level, create an intermediate bridge (fwbrxx) between the tap interface && the vmbrX bridge. (Even if you don't have firewall enabled at datacenter level,...this part only activate generation of iptables rules)

Personnaly, I don't have seen any regression on my side
 
I had the same issue. The VM with huge vitual CPU resources (2 sockets, 24 cores, 48 VCPUs) sometimes had 60-80% of packet loss. VMs with less CPU resources are not affected. Rolling back to kernel 6.2 seems to solve the problem.
 
You must log in or register to reply here.
Top Bottom