Network throughput limits for local VM <> VM communication

[kamil]

Hi Everyone,
I'm currently looking at the network performance between two KVM guests
running on the same host. The host system is applied with two quad core
Xeons each 3GHz and 32G memory.
I use a bridge to pass packets between those guests and the
virtio NIC model. I am wondering why the throughput only goes up to about
950Mbps.

KVM VM101
​

>> br0 | HOST KVM | eth0
​

KVM VM102
​

performance between two guests (iperf server is running on other guest
VM101 <-> VM102):
[ 3] 0.0-60.0 sec 6.07 GBytes 868 Mbits/sec
It is the speed of network interface hardware
Speed is the same VM101 <-> Host 860 Mbits/sec

results within the same guest (iperf server is running on the same
system: VM101 <-> VM101):
[ 3] 0.0-10.4 sec 9.54 GBytes 7.90 Gbits/sec


XenServer 5.6
performance between two guests VM101 <-> VM102 aprox 4 GBits/s

This is for KVM features?

Host Proxmox 1.5 with update
Server HP ML330G6
2x Broadcom Corporation NetXtreme BCM5715 Gigabit Ethernet
driver: tg3 version: 3.102

VM101 CentOS 5.5 with virtio
VM102 CentOS 5.5 with virtio

 

[dietmar]

I use a bridge to pass packets between those guests and the

Is the bridge connected to any physical network card?

 

[kamil]

Advance thanks for the quick reply. The bridge is attached to eth0. eth1 is currently do not use.

 

[dietmar]

The bridge is attached to eth0. eth1 is currently do not use.

A bridge send packets to all attached interfaces, so I guess it is as fast as the slowest attached device.

 

[kamil]

Well I understand it, but then the question is how to make internal connections so that two or more VMs on one host communicate with each other faster than 900 Mbit / s.

 

[dietmar]

Well I understand it, but then the question is how to make internal connections so that two or more VMs on one host communicate with each other faster than 900 Mbit / s.

Use a bridge without any physical device attached?

 

[kamil]

Hi,
Thank you for your interest to solve this problem.

# network interface settings
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address  192.168.1.58
        netmask  255.255.255.0
        gateway  192.168.1.254
        post-up echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp

iface eth1 inet manual

auto vmbr0
iface vmbr0 inet static
        address  10.0.1.1
        netmask  255.255.255.0
        bridge_ports none
        bridge_stp off
        bridge_fd 0

Yes I am connected. VMs connected to vmbr0 without physical linked devices eth0, eth1

Host 10.0.1.1 virtio
VM1 10.0.1.2 virtio
VM2 10.0.1.3 virtio

Throughput VM1 <> VM2 900 Mbit/s

What connection speeds reach VM1 <> Vm2 for you in one host?

 

[kamil]

I've observed that the CPU has to test a large workload. 85% - 90%.

​

 

[dietmar]

I've observed that the CPU has to test a large workload. 85% - 90%

So this explains the throughput limit. KVM is known to have high IO CPU overhead.