Poor network performance on guest

D

danb35

Renowned Member
Oct 31, 2015
84
6
73
tl;dr: I'm seeing poor network performance on a CentOS 6.7 guest over 10GbE. Bare metal on the same hardware would see 6+ Gb/sec using iperf; now the average is closer to 1 Gb/sec.

Before installing Proxmox, I ran my home network on a Centos 6.7-based firewall/server/router (SME 9.0, www.contribs.org). Hardware was a Supermicro X9SCL-F motherboard, i3-3240 CPU, 8 GB RAM. One NIC was used for the WAN; the other for the LAN. Recently I decided I wanted more network bandwidth between that box and my FreeNAS server, so I installed Chelsio S310E-CR NICs in both, and upgraded to a Dell 5524 switch. That worked pretty well--though I never saw 10 Gb/sec throughput, I'd pretty consistently see 6 Gb/sec. Life was good.

Then I decided to virtualize. I bumped the RAM to 16 GB, installed PVE 4.0, and put my SME 9.0 installation on it as a KVM guest. The Chelsio NIC was assigned to a vmbr interface, which was attached to the SME 9.0 guest. Although everything worked, network performance wasn't so good--iperf to my FreeNAS server was now generally showing 800-900 Mb/sec, and internet bandwidth was showing 10-12 Mb/sec download, compared to the 50 Mb/sec I was seeing before (and I'm paying for).

For a variety of reasons (unrelated to network performance), I concluded that my PVE host was under-resourced, so I moved it to one node of a Dell C6100 server with 2x Xeon X5650 and 48 GB of RAM. I also replaced the NIC with a Chelsio T420-SO-CR, a two-port SFP+ 10G NIC. This resolved the other issues, and network performance is improved, but still not what I'm expecting.

My host is now running PVE 4.1-2, and is current on all updates. The onboard NICs are eth3 and eth4, and the two ports on the Chelsio NIC are eth5 and eth6 (eth0-eth2 were assigned when on the old host machine). vmbr0 is tied to eth3, and is the management interface for the host. vmbr1 is tied to eth4, which is presented to the SME guest as net0, a virtio device--it's connected to the cable modem. vmbr2 is tied to eth5, and presented to the SME guest as net1, also a virtio device--it's the LAN interface.

I'm not currently doing anything with the second port of the T420, but I have a second one of those cards that I'm planning to put in the FreeNAS server so I can directly connect the PVE host to it.
 
can you post output of
Code: 
cat /etc/network/interfaces

Also, what does your load-avarage and Iowait say when your running iperf to that VM and only reach 1Gbit/s ?
 
Code: 
root@pve:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage part of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

iface eth2 inet manual

auto vmbr0
iface vmbr0 inet static
    address  192.168.1.3
   netmask  255.255.255.0
   gateway  192.168.1.1
   bridge_ports eth3
   bridge_stp off
   bridge_fd 0

auto vmbr1
iface vmbr1 inet manual
   bridge_ports eth4
   bridge_stp off
   bridge_fd 0

auto vmbr2
iface vmbr2 inet manual
   bridge_ports eth5
   bridge_stp off
   bridge_fd 0

root@pve:~#

Load average on the host is normally about 2.5. While running a 2-minute-long iperf from the VM to the FreeNAS box, it rose to about 3, with a peak at about 3.25. Iowait was generally under 1%, though I saw 6-7% for a second or two. That run resulted in 1.77 Gb/sec, which is an improvement over what I was seeing on the old hardware, but still well under what I was seeing on bare metal.
 
Your IO-wait & load averages look okay (unless you are using a non-HT dual-core CPU :p)

danb35 said:
I'm seeing poor network performance on a CentOS 6.7 guest over 10GbE. Bare metal on the same hardware would see 6+ Gb/sec using iperf; now the average is closer to 1 Gb/sec.
Click to expand...
danb35 said:
That run resulted in 1.77 Gb/sec
Click to expand...
a 77% increase is "strange" especially since you did not indicate that you changed anything. To me that makes me wanna look into all sorts of resource saturation along your VM <-> FreeNas chain.



unrelated, but based on your cat /etc/network/interfaces output:
Q: Any chance you used jumbo frames (MTU >14xx) on your "bare metal" test-case ?
 
mir said:
What nic hardware emulation do you use in your guest?
Click to expand...

I'm afraid I don't understand the question. The guest appears to be using the virtio_net driver, if that addresses your question. If not, how could I better answer it?
 
Q-wulf said:
Your IO-wait & load averages look okay (unless you are using a non-HT dual-core CPU :p)
Click to expand...
Nope, 2x six-core CPUs, both with HT.

a 77% increase is "strange" especially since you did not indicate that you changed anything. To me that makes me wanna look into all sorts of resource saturation along your VM <-> FreeNas chain.
Click to expand...
The results have been pretty variable, and have improved since I moved the PVE host to "new" (old, but more powerful, and new to me) hardware, but are still significantly less than before. The chain from FreeNAS to VM is pretty short: FreeNAS <-> switch <-> PVE host <-> VM. There are two SFP+ ports on the switch--the FreeNAS box is plugged into one; the PVE host into the other. I'll see if I can't whack my other T420 into the FreeNAS box so I can run a direct connection from it to the PVE host.

unrelated, but based on your cat /etc/network/interfaces output:
Q: Any chance you used jumbo frames (MTU >14xx) on your "bare metal" test-case ?
Click to expand...
I won't say it's impossible, but I don't remember doing it, and I'm pretty sure I'd remember.
 
danb35 said:
I'm afraid I don't understand the question. The guest appears to be using the virtio_net driver, if that addresses your question. If not, how could I better answer it?
Click to expand...
I was referring to the hardware chosen in proxmox when adding the network card to the guest. If the guest uses virtio_net I guess this means you have chosen virtio as hardware emulation in proxmox.
 
you can check by looking at your FreeNas's network config. Would have large mtu set.

danb35 said:
a 77% increase is "strange" especially since you did not indicate that you changed anything. To me that makes me wanna look into all sorts of resource saturation along your VM <-> FreeNas chain.
Click to expand...
[...]
The chain from FreeNAS to VM is pretty short: FreeNAS <-> switch <-> PVE host <-> VM. There are two SFP+ ports on the switch--the FreeNAS box is plugged into one; the PVE host into the other[...]
Click to expand...

What i meant by this is e.g high load averages, IO-waits, saturated network on any component along the chain.
Based on your example that would be "FreeNas, Switch, PVE Node and VM".

ps.: I just tested the following on our Work lab:
Centos7(vNic, virtio) <-> PVE(OVS_Bridge) <-> TOR-Switch(1) <-> Aggregate-Switch <-> TOR-Switch(2) <-> PVE(OVS_Bridge) <-> FreeNas (vNic, virtio)
Click to expand...
Iperf showed consistent 40 Gbit/s (minus overhead). This is where we have it rate-limited at. Multiple 40G links Bonded into a single OVS_bridge. Jumbo Frames. Latest pveversion, Vm's with latest updates.

So, it does not seem to be a general issue with Proxmox. (could be native linux bridge related issues, but I do not have much experience with those in 4.x. We use openvswitch instead.)

pps: you checked the network connectors that you plugged into the new physical node, for proper seating, right ?
 
Last edited:
I just installed the second T420 in the FreeNAS box. The first port on it is configured as before, at 192.168.1.10. The second port is configured as 192.168.2.2. On the PVE node, I configured eth6 (the second port of the T420 installed there) as 192.168.2.1 and rebooted. The guest VM is at 192.168.1.1. The 1.x ports are connected to the switch; the 2.x ports are connected directly to each other with a SFP+ patch cable. Here are the iperf results:
Code: 
[root@freenas2] ~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 39898
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  10.9 GBytes  9.38 Gbits/sec
[  5] local 192.168.1.10 port 5001 connected with 192.168.1.1 port 43302
[  5]  0.0-10.0 sec  2.42 GBytes  2.08 Gbits/sec
[  4] local 192.168.1.10 port 5001 connected with 192.168.1.1 port 43307
[  4]  0.0-10.0 sec  2.12 GBytes  1.82 Gbits/sec
[  5] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 39926
[  5]  0.0-10.0 sec  10.9 GBytes  9.33 Gbits/sec

I'd think that would have to pretty well rule out hardware.

Q-wulf said:
you can check by looking at your FreeNas's network config. Would have large mtu set.
Click to expand...
No, large MTU is not, and has not been, set on the FreeNAS box.

Q-wulf said:
pps: you checked the network connectors that you plugged into the new physical node, for proper seating, right ?
Click to expand...
Yes, they're well seated, and latch nicely into place.
 
to verify (hard to keep track in full text form :p):
  • You have 2 vmbr
    • One contains the nic via switch (vmbr<>)
    • One contains teh nic via direct attach (vmbr<>)
  • FreeNas IP's are
    • 192.168.1.10 via switch
    • 192.168.2.2 via direct attach
  • Proxmox-Node IP's are
    • 192.168.1.3 via switch
    • 192.168.2.1 via direct attach
  • Vm IP's are
    • 192.168.1.1 via switch
    • ? via Direct attach


what i'd still do:
  • Assign 2 ip's to the VM
    • IP1 connected to the vmbrX that is connected to the switch
    • IP2 connected to the vmbrY that is connected to the FreeNas via direct attach ?
      • Then run Iperf on those 2 VM IP's ? so we get a side by side comparison.

​
also, Can you post a updated cat /etc/network/interfaces ? (makes it a lot easier to understand whats going on)
 
Q-wulf said:
  • You have 21 vmbr
    • One contains the nic via switch (vmbr<>)
    • One contains teh nic via direct attach (vmbr<>)
  • FreeNas IP's are
    • 192.168.1.10 via switch
    • 192.168.2.2 via direct attach
  • Proxmox-Node IP's are
    • 192.168.1.3 via switch
    • 192.168.2.1 via direct attach
  • Vm IP's are
    • 192.168.1.1 via switch
    • ? via Direct attach
Click to expand...
Correct as modified--the second port of the T420 (eth6) isn't on a vmbr; I simply configured eth6 as 192.168.2.1. I actually have three vmbr interfaces configured, but two of them aren't involved here--one (vmbr0) is the management interface for the PVE host, and the other (vmbr1) is the WAN interface that connects to the cable modem.

Q-wulf said:
Can you post a updated cat /etc/network/interfaces ?
Click to expand...
Code: 
root@pve:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage part of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

iface eth2 inet manual

iface eth3 inet manual

iface eth4 inet manual

iface eth5 inet manual

auto eth6
iface eth6 inet static
   address  192.168.2.1
   netmask  255.255.255.0

auto vmbr0
iface vmbr0 inet static
   address  192.168.1.3
   netmask  255.255.255.0
   gateway  192.168.1.1
   bridge_ports eth3
   bridge_stp off
   bridge_fd 0

auto vmbr1
iface vmbr1 inet manual
   bridge_ports eth4
   bridge_stp off
   bridge_fd 0

auto vmbr2
iface vmbr2 inet manual
   bridge_ports eth5
   bridge_stp off
   bridge_fd 0

root@pve:~#
 
Let me make sure I understand your suggested course of action, since I try to minimize reboots of my host:
  • Unconfigure eth6 (second port of the T420, direct-connected to the T420 in the FreeNAS box)
  • Create vmbr3, bridged to eth6
  • Configure vmbr3 with an IP of 192.168.2.x
  • (reboot host)
  • Add a network device to the VM, bridged to vmbr3
  • Within the VM, configure the new network device with an IP of 192.168.2.x
  • Test using iperf from the PVE node, and from the VM
Testing PVE node <-> FreeNAS with this configuration would involve the Linux Bridge subsystem, which was not involved in my previous test. The VM <-> FreeNAS test would involve that, as well as whatever other layers are used to present the device to the guest OS. If the issue persists in this configuration, that would exclude the switch and cabling to/from the switch as possible problems (which seem unlikely--it's the same ports on the switch, and the same cables, that were used previously when I hit 6 Gb/sec, but anything's possible). Is this right?
 
My suggested course of action is this:
  • create 2 vmbr
    • vmbrX for the eth that runs via the switch
    • vmbrY for the eth that runs via the DAC
  • Assign Proxmox-node 2 IPs
    • 1 IP running via VmbrX
    • 1 IP running via VmbrY
  • Assign VM 2 IPs
    • 1 IP running via VmbrX
    • 1 IP running via VmbrY
  • run Iperf on the following connections and post results:
    • VM <-> FreeNas via vmbrX and vmbrY
    • Proxmox-Node <-> FreeNas via vmbrX and vmbrY
  • post the then updated cat /etc/network/interfaces again
    • state which vmbr is via switch and which is via DAC.
    • state which VM-IP is via swtch and which is via DAC.
Click to expand...
danb35 said:
[..]Is this right?[..]
Click to expand...
Yes, mostly.
We will be able to tell if it is the switch or the linux native vmbr used.

Once we know that, we can advise you on the course to mitigate the issue.
 
OK, here are the results. IPs are:
  • VM via DAC: 192.168.2.3
  • VM via switch: 192.168.1.1
  • PVE node via DAC: 192.168.2.1
  • PVE node via switch: 192.168.1.33
  • FreeNAS via DAC: 192.168.2.2
  • FreeNAS via switch: 192.168.1.10
vmbrs are:
  • vmbr2 is via switch
  • vmbr3 is via DAC
iperf results are:
Code: 
[root@freenas2] ~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 192.168.1.10 port 5001 connected with 192.168.1.1 port 35958
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-60.0 sec  5.75 GBytes   822 Mbits/sec
[  5] local 192.168.2.2 port 5001 connected with 192.168.2.3 port 32996
[  5]  0.0-60.0 sec  42.8 GBytes  6.13 Gbits/sec
[  4] local 192.168.1.10 port 5001 connected with 192.168.1.1 port 35960
[  4]  0.0-60.0 sec  7.51 GBytes  1.08 Gbits/sec
[  5] local 192.168.2.2 port 5001 connected with 192.168.2.3 port 32998
[  5]  0.0-60.0 sec  44.0 GBytes  6.29 Gbits/sec
[  4] local 192.168.2.2 port 5001 connected with 192.168.2.1 port 49938
[  4]  0.0-60.0 sec  65.6 GBytes  9.40 Gbits/sec
[  5] local 192.168.1.10 port 5001 connected with 192.168.1.33 port 38842
[  5]  0.0-10.1 sec  5.33 GBytes  4.54 Gbits/sec
[  4] local 192.168.1.10 port 5001 connected with 192.168.1.33 port 38844
[  4]  0.0-10.1 sec  5.06 GBytes  4.31 Gbits/sec
Average 900 Mb/sec from VM, via switch, to FreeNAS
Average 6.2 Gb/sec from VM, via DAC, to FreeNAS
Average 4.4 Gb/sec from PVE node, via switch, to FreeNAS
9.4 Gb/sec from PVE node, via DAC, to FreeNAS

After waiting 10-15 minutes after booting the VM, the "via switch" speed increased to 1.6 Gb/sec; the DAC speed went down slightly to 5.9 Gb/sec. After waiting another 5 minutes or so and trying again, the "via switch" speed dropped back to 832 Mb/sec.

Here's /etc/network/interfaces:
Code: 
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage part of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

iface eth2 inet manual

iface eth3 inet manual

iface eth4 inet manual

iface eth5 inet manual

auto eth6
iface eth6 inet manual

auto vmbr0
iface vmbr0 inet static
    address  192.168.1.3
    netmask  255.255.255.0
    gateway  192.168.1.1
    bridge_ports eth3
    bridge_stp off
    bridge_fd 0

auto vmbr1
iface vmbr1 inet manual
    bridge_ports eth4
    bridge_stp off
    bridge_fd 0

auto vmbr2
iface vmbr2 inet static
    address  192.168.1.33
    netmask  255.255.255.0
    bridge_ports eth5
    bridge_stp off
    bridge_fd 0

auto vmbr3
iface vmbr3 inet static
    address  192.168.2.1
    netmask  255.255.255.0
    bridge_ports eth6
    bridge_stp off
    bridge_fd 0
 
The obvious conclusion would appear to be that there's something wrong with the switch and/or the cables from the NICs to the switch. But there's still a huge disparity between the PVE node performance and the VM performance. I've tried a little cable-swapping without effect, but haven't tried all possible configurations with three cables and three connections.
 
danb35 said:
The obvious conclusion would appear to be that there's something wrong with the switch and/or the cables from the NICs to the switch. But there's still a huge disparity between the PVE node performance and the VM performance. I've tried a little cable-swapping without effect, but haven't tried all possible configurations with three cables and three connections.
Click to expand...

If this were universally low Iperfs when running via a linux native vmbro, i'd have said, test it with openvswitch, but .. that does not seem to be the issue based on your benchmarks.



Indeed, these tests suggest there is an issue when running via the physical switch. Could be cabling, could be overload, could be heat, could be lots of stuff.


There however also seems to be an issue with the VM somewhere. Proxmox and VM should get near equal speeds when running via the vmbr in perfect conditions.



Out of curiosity,
when you do the following iperfs:

Code: 
Vm <-> FreeNas via DAC vmbr3
Node <-> via DAC vmbr3

What do Load-avarage and IO-wait say in comparison ?? ON Proxmox-Node and preferable on your FreeNas aswell?

edit: What switch are you using ? Make + model
edit: the VM is Centos 7, right ?
 
Last edited:
The VM is CentOS 6.7, the switch is a Dell PowerConnect 5524. The vmbr2 port on the PVE node, and port 1 on the FreeNAS box, are connected to the SFP+ ports on the switch using 2M twinax patch cables. The vmbr3 port on the PVE node, and port 2 on the FreeNAS box, are connected to each other using the same kind of cable. The switch reports an operating temperature of 46 C, and there's not a great deal of other load on it.

When doing node to FreeNAS via DAC, load on the node ranges from 2.5-3.0, and IOWait is under 1%. CPU usage is hovering around 10%. Load on the FreeNAS server increases from a baseline of 0.8 to about 2.0. Speed is 9.4 Gb/sec. Going via the switch, CPU usage is 8-9%, IOWait is still under 1%, load average is 3.2-3.5. Load average on the FreeNAS server is as high as 3 during this run. Speed is 4.6 Gb/sec.

When doing VM to FreeNAS via DAC, load on the node climbs a bit higher, to about 3.5. CPU usage is a bit higher as well, 12-14%. Load on the FreeNAS server is a bit lower, 1.4-1.6; load on the VM goes from a baseline of 0 to 1. IOWait remains under 1%. Speed is 5.8 Gb/sec. Going via the switch, CPU usage on the node is more variable, but hits peaks of about 12%. Load average on the node is around 3.2. Load average on the FreeNAS box is 1.25, and on the VM about 0.6. Speed on that run is 1.4 Gb/sec.

Just for the sake of comparison, I ran iperf from the VM to the node on the "via switch" interface and on the "via DAC" interface. The "via switch" interface showed 8.4 Gb/sec, and the "via DAC" interface showed 9.4 Gb/sec.

So, there certainly appears to be something going on with the switch (and/or cabling to/from), but there also seems to be something going on between PVE and the VM--the PVE node was about 50% faster than the VM via DAC, and over 3x faster via the switch.

On the switch side of things, AIUI, my cabling isn't officially supported. I could look for a set of Dell-branded optics for the switch, Chelsio-branded optics for the NICs, and a couple of fiber patch cables. That should eliminate any potential incompatibility on that side of things, though at a bit of expense. The switch itself is still under warranty, though I might have a hard time getting it replaced without a hard failure to show.
 
You must log in or register to reply here.
Top Bottom