Virtio network performance and CPU load on different kernels

[NickUA]

Hi all.

I have few VM on Proxmox VE. They use KVM and Virtio drivers. Debian OS with kernel 2.6.32-5 is running in VM's. They are routers for small ISP network. CPU is Sempron 140(one core).

In this case, I have problem with network performance and latency. During rush hour CPU load is about 100%. Traffic is 40Mbit/s only. Also I tested speed between guests and saw only about 600Mbit/s.

I thin reason of this is kernel version on proxmox host.

pve-manager: 2.2-32 (pve-manager/2.2/3089a616)
running kernel: 2.6.32-16-pve
proxmox-ve-2.6.32: 2.2-83
pve-kernel-2.6.32-16-pve: 2.6.32-82
pve-kernel-2.6.32-17-pve: 2.6.32-83
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-34
qemu-server: 2.0-72
pve-firmware: 1.0-21
libpve-common-perl: 1.0-41
libpve-access-control: 1.0-25
libpve-storage-perl: 2.0-36
vncterm: 1.0-3
vzctl: 4.0-1pve2
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.3-10
ksm-control-daemon: 1.1-1

Who has the test results on different kernels?

 

[dietmar]

There is a know virtio bug in 2.6.32 kernels. Please try to use a 3.X kernel inside the Guest.

 

[michu]

There is a know virtio bug in 2.6.32 kernels. Please try to use a 3.X kernel inside the Guest.

Hi,

I'm very confused about that bug. I did a little test today:

1) VM with Ubuntu 10.04.4 LTS (2.6.32-45), 2 cores.
2) VM with Ubuntu 12.04.2 LTS (3.2.0-37), 2 cores.
3) Physical machine with 10.04.4 LTS (2.6.32-45) - standard PC with Realtek 1Gb/s NIC.

Proxmox has tg3 driver NIC.

All connected with 1 Gb/s NIC thru 100/1000 switch.

What I got:

1->3 got average 429 Mbps,
2->3 got average 473 Mbps,

I'm not sure if results are O.K (maybe general network performance is low). But it seems there is no big difference.
All tested with newest stable Proxmox with qemu 1.3, on production environment so results could be not 100% accurate.

Not sure.... but maybe Ubuntu has it fixed with: http://patchwork.ozlabs.org/patch/182282/

Regards,
michu

 

[spirit]

Hi,

I'm very confused about that bug. I did a little test today:

1) VM with Ubuntu 10.04.4 LTS (2.6.32-45), 2 cores.
2) VM with Ubuntu 12.04.2 LTS (3.2.0-37), 2 cores.
3) Physical machine with 10.04.4 LTS (2.6.32-45) - standard PC with Realtek 1Gb/s NIC.

Proxmox has tg3 driver NIC.

All connected with 1 Gb/s NIC thru 100/1000 switch.

What I got:

1->3 got average 429 Mbps,
2->3 got average 473 Mbps,

I'm not sure if results are O.K (maybe general network performance is low). But it seems there is no big difference.
All tested with newest stable Proxmox with qemu 1.3, on production environment so results could be not 100% accurate.

Not sure.... but maybe Ubuntu has it fixed with: http://patchwork.ozlabs.org/patch/182282/

Regards,
michu

Maybe are you cpu limited ? (do you see vhost-net process on your host using 100% cpu ?).
do the test with guest kernel 3.2.

 

[NickUA]

There is a know virtio bug in 2.6.32 kernels. Please try to use a 3.X kernel inside the Guest.

I did test with 3.2 kernel on guest's, but the result is the same.​

 

[michu]

I did test with 3.2 kernel on guest's, but the result is the same.​

Process takes 12% of cpu, but host machine has 8 cores (16 logical cpu's with HT).
Machines with 3.2.0 kernel takes about 3-5%.

Regards,
michu

 

[NickUA]

Do you speak about host or guest kernel? What kernel version do you use on host?

 

[michu]

Sorry, I didn't notice that you use AMD Sempron 140 with 1 core. Looks for rather old cpu. I rememeber when in past I worked with Proxmox 1.x on old Intel Server with 2.0Ghz CPU with 2 cores. I used to have 50-80% of cpu used with 5 VMs.... So maybe there are some issues with VT support in older cpu's rather than in QEMU/KVM ???

Regards,
michu

- - - Updated - - -

I speak about kvm process on host machine. On host I have newest stable Proxmox 2.2 with 2.6.32-17 and qemu 1.3.

 

[michu]

Do you speak about host or guest kernel? What kernel version do you use on host?

I speak about kvm process on host machine. I use newest Proxmox 2.2 with 2.6.32-17.
I see that you use rather old AMD Sempron 140. I remember when I was working on old Intel Server with 2.00 Ghz, 2 core cpu. I used to have cpu usage in about 50%-70% for 5 VMs. So maybe this is issue with VT support in older cpu rather than KVM/QUEMU problem ???

Regards,
michu