High network throughput causes high CPU usage in VM

D

drwtsn32

New Member
Jul 15, 2024
13
0
1
Hello,

I have an Intel N100 mini PC with 4x 2.5Gb NICs. I am observing very high CPU usage in a VM when network throughput is 2+ Gbps. I'm doing throughput tests from a physical machine on my LAN to the VM. Both are connected to 2.5Gb switches.
  • Without rate limiting the throughput test, VM CPU usage goes to 60+%. Speed result fluctuates but averages about 1.8Gbps.
  • If I rate limit to 1Gbps, CPU usage in the VM is only around 2%, and speed result is a solid 1Gbps.
  • If I test against the proxmost host management NIC (also 2.5Gbps), host CPU usage climbs to maybe 6%, and speed result is solid 2.37Gbps.
So, it seems like the host can handle it just fine. Why does VM CPU usage go up so high and cause throughput to drop?

VM is Debian 13.x and its NICs are using VirtIO.

Any ideas?
 
Please share qm config VMIDHERE. How is the traffic generated? What does top -co%CPU inside the VM say? Does it happen with iperf3 between node and VM too?
 
Appreciate your response!

qm config output:

Code: 
agent: 1
boot: order=scsi0;ide2;net0
cores: 4
cpu: host
ide2: none,media=cdrom
memory: 4096
meta: creation-qemu=9.0.0,ctime=1721088552
name: xxx
net0: virtio=00:50:56:00:00:00,bridge=vmbr1
net1: virtio=00:50:56:00:00:01,bridge=vmbr2
net2: virtio=00:50:56:00:00:02,bridge=vmbr3
numa: 0
onboot: 1
ostype: l26
scsi0: local-lvm:vm-100-disk-0,discard=on,iothread=1,size=32G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=b57a4bf8-7f1e-404b-bb14-5ecb78cbd60f
sockets: 1
vmgenid: c2b21659-a006-4736-a09b-d7063f53b375

I was using ntttcp to do bandwidth testing, but iperf3 shows the same issue. Here is iperf3 output (physical PC -> VM):

Code: 
Accepted connection from 10.1.1.11, port 55745
[  5] local 10.1.1.1 port 5201 connected to 10.1.1.11 port 55746
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   282 MBytes  2.36 Gbits/sec                 
[  5]   1.00-2.00   sec   282 MBytes  2.37 Gbits/sec                 
[  5]   2.00-3.00   sec   282 MBytes  2.37 Gbits/sec                 
[  5]   3.00-4.00   sec   282 MBytes  2.37 Gbits/sec                 
[  5]   4.00-5.00   sec   283 MBytes  2.37 Gbits/sec                 
[  5]   5.00-6.00   sec   282 MBytes  2.36 Gbits/sec                 
[  5]   6.00-7.00   sec   282 MBytes  2.36 Gbits/sec                 
[  5]   7.00-8.00   sec   250 MBytes  2.09 Gbits/sec                 
[  5]   8.00-9.00   sec   260 MBytes  2.18 Gbits/sec                 
[  5]   9.00-10.00  sec   273 MBytes  2.29 Gbits/sec                 
[  5]  10.00-11.00  sec   272 MBytes  2.28 Gbits/sec                 
[  5]  11.00-12.00  sec   260 MBytes  2.18 Gbits/sec                 
[  5]  12.00-13.00  sec   250 MBytes  2.10 Gbits/sec                 
[  5]  13.00-14.00  sec   196 MBytes  1.64 Gbits/sec                 
[  5]  14.00-15.00  sec   186 MBytes  1.56 Gbits/sec                 
[  5]  15.00-16.00  sec   223 MBytes  1.87 Gbits/sec                 
[  5]  16.00-17.00  sec   226 MBytes  1.90 Gbits/sec                 
[  5]  17.00-18.00  sec   225 MBytes  1.89 Gbits/sec                 
[  5]  18.00-19.00  sec   222 MBytes  1.86 Gbits/sec                 
[  5]  19.00-20.00  sec   219 MBytes  1.84 Gbits/sec                 
[  5]  20.00-21.00  sec   222 MBytes  1.86 Gbits/sec                 
[  5]  21.00-22.00  sec   244 MBytes  2.05 Gbits/sec                 
[  5]  22.00-23.00  sec   244 MBytes  2.05 Gbits/sec                 
[  5]  23.00-24.00  sec   226 MBytes  1.89 Gbits/sec                 
[  5]  24.00-25.00  sec   257 MBytes  2.15 Gbits/sec                 
[  5]  25.00-26.00  sec   214 MBytes  1.79 Gbits/sec                 
[  5]  26.00-27.00  sec   226 MBytes  1.90 Gbits/sec                 
[  5]  27.00-28.00  sec   223 MBytes  1.87 Gbits/sec                 
[  5]  28.00-29.00  sec   222 MBytes  1.87 Gbits/sec                 
[  5]  29.00-30.00  sec   242 MBytes  2.03 Gbits/sec                 
[  5]  30.00-30.00  sec   256 KBytes   915 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-30.00  sec  7.19 GBytes  2.06 Gbits/sec                  receiver

according to top, the following are the top two CPU % users:

Code: 
iperf3 -s
[ksoftirqd/0]

I do get max bandwidth 2.37Gbps for about 6 seconds before CPU usage starts spiking and bandwidth drops. Maybe that's a clue?

If I do iperf3 from the PVE node to the VM it behaves the same as the test where the traffic comes from the physical PC. Tried iperf3 the other direction (VM -> host) and it also behaved the same.

iperf3 from physical PC to PVE node was solid 2.37Gbps.

Seems to be something with the VM.
 
That looks like a fairly normal guest. I can't see the actual percentages. Can you try another CPU model than host? What is the host CPU model?
 
Last edited:
You must log in or register to reply here.
Top Bottom