[SOLVED] Network lag / lost packets and much higher cpu usage since upgrade to PVE 8 / Linux 6.5

[abma]

We've upgraded PVE from 7 to 8 and also the Kernel from 6.2 to 6.5.11-6-pve: since these changes, the CPU-usage of the kvm-process is much higher then before. Before the upgrade the idle process usage was ~2%, and never above ~50% now after the upgrade the idle cpu usage is at ~20% and it goes up to 150%.

When the cpu usage is high, packets get dropped and delayed: ping times are between 0.1ms and 400ms: before it was always at 0.1ms.

The vm is used as router/firewall and runs debian 12 with the default kernel (6.1.0-15-amd64).

Also iperf performance went from ~10GBit to 300Mbit. Enabling multi-queue and adding more vcpus helped a bit to increase bandwith back to ~10Gbit, but the "random" lags are still there. The vm basicly does no disk i/o, it only acts as router.

Does anyone else have this problem?

It seems packet processing is much slower after the upgrade.

 

[abma]

perf_6.5 record -F 99 -p 4072637 -g -- sleep 30
perf_6.5 report -n --stdio

 

[abma]

I've disabled firewall on interface level: this reduced the lag. Thats a bit weird, because firewall was already disabled at datacenter and node level.

 

[spirit]

I've disabled firewall on interface level: this reduced the lag. Thats a bit weird, because firewall was already disabled at datacenter and node level.

The firewall at interface level, create an intermediate bridge (fwbrxx) between the tap interface && the vmbrX bridge. (Even if you don't have firewall enabled at datacenter level,...this part only activate generation of iptables rules)

Personnaly, I don't have seen any regression on my side

 

[spirit]

perf record show vhost_net cpu usage (so at kernel level). I don't see relation with extra fwbr bridge.

do you have tried to rollback to 6.2 kernel ?

also, do you use linux bridge or openvswitch ?

 

[nugzarg]

I had the same issue. The VM with huge vitual CPU resources (2 sockets, 24 cores, 48 VCPUs) sometimes had 60-80% of packet loss. VMs with less CPU resources are not affected. Rolling back to kernel 6.2 seems to solve the problem.