Poor virtio network performance, pfSense guest. Proxmox 3.4

W

whitewater

Member
Nov 26, 2012
107
0
16
france
Hello,

i have a curious network bandwith problem.

Here my install :
2 nodes proxmox v3.4, identical, with :
motherboard Tyan S5393, 2x intel gigabit.
1x intel 10 gigabits, DRBD dedicaced.
1x Intel PRO/1000 CT Desktop, Intel 82574L chipset.
1x Realtek 100 mbps.

a 3rd node for HA quorum.

I have a pfsense v2.2.4. Fresh install
4 network virtio card : LAN (intel motherboard), LAN2 (not used here), WAN1 and WAN2.

for iperf test :
192.168.1.20 : synology rs2212+
192.168.1.12 : proxmox02 (node 2)
192.138.1.253 : LAN virtio network card

"Hardware Checksum Offloading" disabled on pfsense.
All pluged on a cisco sg200 switch.

pfSense Guest run on proxmox02

* test with synology and host proxmox02 :
192.168.1.12 : iperf -s
192.168.1.20 : iperf -c

result : 941 mbits

192.168.1.20 : iperf -s
192.168.1.12 : iperf -c

result : 941 mbits too

* test with synology and LAN virtio network card (proxmox02 host) :
192.168.1.253 : iperf -s
192.168.1.20 : iperf -c

result : 148 mbits

192.168.1.20 : iperf -s
192.168.1.253 : iperf -c

result : 220 mbits

* test with proxmox02 host and and LAN virtio network card (not same physical card).
192.168.1.253 : iperf -s
192.168.1.12 : iperf -c

result : 150 mbits

192.168.1.12 : iperf -s
192.168.1.253 : iperf -c

result : 340 mbits

So, pfSense bandwith seam to keep around 150 mbits.

Same result with :
- change guest type network card to intel e1000.
- Run Guest on proxmox01 host.
- turn off checksum offload of physical ethernet card (ethtool -K eth0 tx off).

With Windows 7 pro guest, i have a bandwith around 500 - 600 Mbits.

Code: 
root@proxmox02:~# pveversion -V
proxmox-ve-2.6.32: 3.4-165 (running kernel: 3.10.0-13-pve)
pve-manager: 3.4-11 (running version: 3.4-11/6502936f)
pve-kernel-2.6.32-40-pve: 2.6.32-160
pve-kernel-2.6.32-32-pve: 2.6.32-136
pve-kernel-3.10.0-13-pve: 3.10.0-38
pve-kernel-3.10.0-8-pve: 3.10.0-30
pve-kernel-3.10.0-5-pve: 3.10.0-19
pve-kernel-3.10.0-11-pve: 3.10.0-36
pve-kernel-2.6.32-42-pve: 2.6.32-165
pve-kernel-2.6.32-37-pve: 2.6.32-150
pve-kernel-2.6.32-34-pve: 2.6.32-140
pve-kernel-2.6.32-31-pve: 2.6.32-132
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-3
pve-cluster: 3.0-19
qemu-server: 3.4-6
pve-firmware: 1.1-4
libpve-common-perl: 3.0-24
libpve-access-control: 3.0-16
libpve-storage-perl: 3.0-33
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.2-11
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1
root@proxmox02:~# pveversion -V
proxmox-ve-2.6.32: 3.4-165 (running kernel: 3.10.0-13-pve)
pve-manager: 3.4-11 (running version: 3.4-11/6502936f)
pve-kernel-2.6.32-40-pve: 2.6.32-160
pve-kernel-2.6.32-32-pve: 2.6.32-136
pve-kernel-3.10.0-13-pve: 3.10.0-38
pve-kernel-3.10.0-8-pve: 3.10.0-30
pve-kernel-3.10.0-5-pve: 3.10.0-19
pve-kernel-3.10.0-11-pve: 3.10.0-36
pve-kernel-2.6.32-42-pve: 2.6.32-165
pve-kernel-2.6.32-37-pve: 2.6.32-150
pve-kernel-2.6.32-34-pve: 2.6.32-140
pve-kernel-2.6.32-31-pve: 2.6.32-132
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-3
pve-cluster: 3.0-19
qemu-server: 3.4-6
pve-firmware: 1.1-4
libpve-common-perl: 3.0-24
libpve-access-control: 3.0-16
libpve-storage-perl: 3.0-33
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.2-11
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1

I not at same location.
Maybie test with intel 1000 CT desktop when i could ?

An idea ? Thank you :p
 
Just a shoot in the dark: cpu resource is enough in the VM? What about shut down pfsense, install i.e. Debian 8 64 on the same vmbr and repeat the iperf tests?
 
Hello mmenaz,
CPU ressource is enough for that.
Motherboard have 2x Intel Xeon CPU E5310 @ 1,60 Ghz :)

I have intalled debian 8.2 with virtio :
good iperf test, 941 Mbits

Another test, I have changed LAN pfSsense bridged to intel gigabit CT desktop card :
same result.

On another location, with 2 proxmox 3.4, pfsense 2.1.5, intel e1000 as virtio netword card :
iperf around 235 Mbits.
 
I was referring to VM cpu resources, maybe you have assigned only one core and traffic inspection of pfSense saturates it's capability? AFAIR, there is a way to disable in pfSense the firewall part, try that also. Or try directly the OS pfSense is based on. Just to know if is a pfSense issue, an FreeBSD issue, or a config one.
Also please show your VM config (# qm config <VMID>)
 
Sorry for the confusion about CPU ressource.
Here is :

root@proxmox02:~# qm config 106
bootdisk: virtio0
cores: 2
ide2: none,media=cdrom
memory: 2048
name: pfSense-224
net0: virtio=9A:D9:F7:FA:31:EB,bridge=vmbr0
net1: virtio=A6:18:AC:BC:BA:1C,bridge=vmbr1,link_down=1
net2: virtio=86:92:70:F8:B2:04,bridge=vmbr4,tag=835
net3: virtio=BE:59:1E:FA:D1:62,bridge=vmbr2,link_down=1
net4: virtio=7A:4E:E5:BB:17:1E,bridge=vmbr0,link_down=1
numa: 0
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=b4cdc0a2-7ab2-44c4-a0ca-7b12162dedac
sockets: 1
tablet: 0
virtio0: drbdr0:vm-106-disk-1,cache=writethrough,size=50G
Click to expand...

I have just do test with 2 CPU socket, 4 core and CPU host type. No another VM running on this host.
Same result, 143 Mbits
root@proxmox02:~# qm config 106
bootdisk: virtio0
cores: 4
cpu: host
ide2: none,media=cdrom
memory: 2048
name: pfSense-224
net0: virtio=9A:D9:F7:FA:31:EB,bridge=vmbr0
net1: virtio=A6:18:AC:BC:BA:1C,bridge=vmbr1,link_down=1
net2: virtio=86:92:70:F8:B2:04,bridge=vmbr4,tag=835
net3: virtio=BE:59:1E:FA:D1:62,bridge=vmbr2,link_down=1
net4: virtio=7A:4E:E5:BB:17:1E,bridge=vmbr0,link_down=1
numa: 0
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=b4cdc0a2-7ab2-44c4-a0ca-7b12162dedac
sockets: 2
tablet: 0
virtio0: drbdr0:vm-106-disk-1,cache=writethrough,size=50G
Click to expand...
 
Last edited:
Assuming latest pfSense based on FreeBSD 10.2 kernel, I would ask if you could make a simple test. I've been playing around with boosting performance and stability of my FreeBSD VM's and one thing I noticed is that the system time keeper is not very great by default with FreeBSD 10.

Specifically, in FreeBSD 10, it finds the following possible time counter choices (as per dmesg boot info):

Code: 
Timecounter "HPET" frequency 100000000 Hz quality 950
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
Timecounter "i8254" frequency 1193182 Hz quality 0
Timecounters tick every 1.000 msec

As the HPET counter has the highest quality, it gets chosen. You can verify which time counter your kernel is using with
Code: 
sysctl kern.timecounter.hardware
from the command line.

The problem is that it seems to me HPET is not well emulated as FreeBSD wants to use it. One of the major symptoms you will see are lines like these in your system messages log (and dmesg outout):

Code: 
kernel: calcru: runtime went backwards from 236 usec to 230 usec for pid 0 (kernel)

(by now you're wondering what this has to do with your NIC throughput, but hang on...)

So I dug around the internet and found two pieces of advice to fix this. One was to set
Code: 
kern.hz=100
but that turns out to be invalid advice for anything newer than FreeBSD 9.0 since it doesn't use a constant ticker by default. The other advice was to choose a different timecounter device. As expected, there were conflicting recommendations. One said to force it to choose HPET, and others said to disable HPET. Since I was already having HPET chosen by default, I went the other way:

Code: 
sysctl kern.timecounter.hardware="ACPI-fast"

Now, here is where I think it may be relevant to your situation... the moment I ran that command on one of my large VMs (16GB RAM, 3x 1TB virtual disks for ZFS) the interactive response and ZFS I/O speed improved. At the time ZFS was running a scrub and estimated 3 hours remaining. It finished in under 1.5.

I no longer feel stutters when editing files or running large analysis jobs on the database. I haven't had a calcru warning since then whereas I normally see them every 2-3 days.

So, my request to you is if you could please try switching the timecounter (you do not need to reboot to do it) and see if you get any improvement on your NIC throughput.

As I haven't booted pfSense on my PVE (I run it on bare metal), you can verify your timecounter choices with this:

Code: 
sysctl kern.timecounter.choice

And select the highest quality timer other than HPET.

For what its worth, on my FreeBSD VM configs, I have added

Code: 
qm set 103 -args "-no-hpet"

So it picks the next best clock automatically.
 
Hello vkhera,
thank you for this post very instructive :D

I have shutdown my VM, add the command line :
qm set 106 -args "-no-hpet"

And start, but... same result. Around 143 Mbits.

Here result of command line sysctl kern.timecounter.choice :
kern.timecounter.choice: TSC(1000) i8254(0) ACPI-safe(850) dummy(-1000000)
Click to expand...

i will test with ACPI tomorrow to compare.
 
I have seen similar performance, i put it down to the freebsd virtio drivers being quite new and not as fast as the linux/windows drivers yet...
 
sigxcpu said:
FreeBSD 10 has ~30Gbit local virito performance (host - guest) and 500Mbit (limited by Wi-Fi connection) outside the host with my laptop.
Click to expand...

I see same numbers with wired-only connections. VM-VM on same PVE host gets ~20GB/s with both VM's being FreeBSD 10 and virtio network.

I do not see significant reduction in throughput between another host on the LAN and the PVE server vs. the VM running on same PVE server.

My FreeBSD's are configured with bridged network in PVE.
 
mir said:
Strange. On my virtual pfsense timecounter hardware is chosen to be TSC-low
Click to expand...

My FreeBSD 9 guests will select TSC-low and keep great time. For some reason I cannot figure, FreeBSD 10 will not even detect the TSC timer.

Are you running older pfSense? If you are running 2.2, I'd be interested to see the output of "dmesg|grep counter"
 
From index:
2.2.4-RELEASE (amd64)
built on Sat Jul 25 19:57:37 CDT 2015
FreeBSD 10.1-RELEASE-p15
Click to expand...
dmesg|grep counter
Timecounter "HPET" frequency 100000000 Hz quality 950
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
Timecounter "i8254" frequency 1193182 Hz quality 0
Timecounters tick every 1.000 msec
Timecounter "TSC-low" frequency 1400069495 Hz quality 1000
Click to expand...
pve hardware for pfsense:
CPU: 1 socket 1 core Opteron_G5
OS Type: Other
Disc: scsi
Disc controller: scsi virtio
 
I running last pfsense version at this moment, v2.2.4.

I have remove the line -args "-no-hpet" and sysctl kern.timecounter.choice commande give TSC.

dmesg|grep counter give :
Timecounter "HPET" frequency 100000000 Hz quality 950
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
Timecounter "i8254" frequency 1193182 Hz quality 0
Timecounters tick every 1.000 msec
Timecounter "TSC" frequency 1595932206 Hz quality 1000
Click to expand...

i have shutdown vm pfsense and add the line -args "-no-hpet" again. dmesg|grep counter give :
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
Timecounter "i8254" frequency 1193182 Hz quality 0
Timecounters tick every 1.000 msec
Timecounter "TSC" frequency 1595933826 Hz quality 1000
Click to expand...

ACPI-safe mode is worse (normal :p)

Core CPU type for pfsense VM is Host. Not a problem ?
 
TSC works quit well here so there is no point in adding extra args to the startup command line.
Regarding using CPU type 'Host': This is only a good choice if all your proxmox hosts in the cluster are completely identical. I my case they are so here choosing host should cause no problem. From old habit I favor choosing the CPU family.
 
For a FreeBSD guest using virtio drivers I always disable checksum offloading. Otherwise I am seeing poor performance or in certain situations packets may get corrupt and odd things happen.

ifconfig virtio0 -rxcsum -txcsum

I also like to disable segment offloading:

ifconfig virtio0 -tso -lro -vlanhwtso

For PfSense there should be a setting in the GUI to "Disable Hardware Checksum Offloading" which basically does the same thing. Networking -> System -> Advanced or in some similar place.

Edit: Sorry, just noticed you've tried the above without any results. Can't delete the post :(
 
Last edited:
I have tested with a fresh install again.
A little better : around 200 Mbits instead of 150 Mbits :))

First install was with a restore backup, but traffic shaping reseted.

At another location, fresh install, it's good :
i have around 600 - 700 Mbits.

I will have to be on move next weeks.
I will have a little time to check, but i will notice you when i should do it.
 
mir said:
From index:

dmesg|grep counter

pve hardware for pfsense:
CPU: 1 socket 1 core Opteron_G5
OS Type: Other
Disc: scsi
Disc controller: scsi virtio
Click to expand...

Interesting... It must be the CPU type selection that changes the clock availability. I agree if TSC clock is available that's the best. I don't get any issues in VMs when that clock is selected. The HPET clock for sure causes degradation in interactive performance. I shall experiment with other CPU types and see what happens.

In FreeBSD 9 kernel it does detect TSC clock with the same configs I use now.
 
Does you test traffic go through vmbr0?
Mine does, and with pfsense 2.2.4-RELEASE (amd64) I've (pfsense lan on vmbr0 with IP 192.168.1.253, Proxmox vmbr0 with 192.168.1.9)
Code: 
root@proxmox:~# iperf -s 
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.1.9 port 5001 connected with 192.168.1.253 port 32402
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  1.39 GBytes  1.19 Gbits/sec

and
[2.2.4-RELEASE][admin@pfSense.asgard.local]/root: iperf -c 192.168.1.9                                                                                                                       
------------------------------------------------------------                                                                                                                                 
Client connecting to 192.168.1.9, TCP port 5001                                                                                                                                              
TCP window size: 65.0 KByte (default)                                                                                                                                                        
------------------------------------------------------------                                                                                                                                 
[  3] local 192.168.1.253 port 32402 connected with 192.168.1.9 port 5001                                                                                                                    
[ ID] Interval       Transfer     Bandwidth                                                                                                                                                  
[  3]  0.0-10.0 sec  1.39 GBytes  1.20 Gbits/sec

are you really sure that you disabled checksum offloading in pfSense?
Have you any special setting in /etc/network/interfaces?
Is the tested traffic going through your "net2" inside vlan (tag=835)?
my pfSense config is really like your one:
Code: 
boot: c                                                                                                                                                                                      
bootdisk: virtio0                                                                                                                                                                            
cores: 2                                                                                                                                                                                     
description: vmbr0 192.168.1.253
ide2: none,media=cdrom                                                                                                                                                                       
memory: 512                                                                                                                                                                                  
name: pfsense                                                                                                                                                                                
net0: virtio=96:F5:F2:95:7A:3D,bridge=vmbr0                                                                                                                                                  
net1: virtio=3E:53:3B:1D:AB:CD,bridge=vmbr4                                                                                                                                                  
net2: virtio=06:0E:E3:E5:58:70,bridge=vmbr5                                                                                                                                                  
net3: virtio=96:2E:BB:F6:2E:F1,bridge=vmbr6                                                                                                                                                  
numa: 0                                                                                                                                                                                      
onboot: 1                                                                                                                                                                                    
ostype: other
smbios1: uuid=9fe41a13-39d6-47e8-86bb-bb72d1f509e6
sockets: 1
startup: order=1
tablet: 0
virtio0: local:108/vm-108-disk-1.qcow2,size=8G
My vmbr0 is on eth1 that is a r8169 driven interface.
I'm on Proxmox 4, maybe qemu is more efficient here?
I've no more ideas
 
Really strange:
2 socket 1 core
dmesg | grep counter
Timecounter "HPET" frequency 100000000 Hz quality 950
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
Timecounter "i8254" frequency 1193182 Hz quality 0
Timecounters tick every 1.000 msec
Timecounter "TSC-low" frequency 1400069608 Hz quality 800

1 socket 2 core
dmesg |grep counter
Timecounter "HPET" frequency 100000000 Hz quality 950
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
Timecounter "i8254" frequency 1193182 Hz quality 0
Timecounters tick every 1.000 msec
Timecounter "TSC-low" frequency 1400072622 Hz quality 1000
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back