PVE3.1 + FreeBSD8.3 / pfsense - KVM Hardware Virtualization and VirtIO not possible

N

Nachtfalke

Member
Apr 7, 2013
46
0
6
Hi,

I am running Proxmox VE 3.1 on a Fujitsu Siemens Primergy TX120 S6 Server with Intel Xeon E1220v2 CPU.
I have virtualisation support enabled on the Fujitsu BIOS.

I have two VMs running, one is Windows Server 2008R2, and I have KVM Hardware Virtualization enabled on Proxmox. This is working. I installed the VirtIO drivers for HDD and NICs and all worked.

The second VM is running pfsense 2.1 AMD64 (FreeBSD 8.3). If I enable KVM Hardware Virtualization on Proxmox pfsense stops booting after the menu. I tried with disabling ACPI, Single User mode, and so on. It does not boot. It is the same with FreeBSD 8.3 bootonly ISO. If I disable KVM Hardware Virtualization and change the CPU type from "host" to "Sandy Bridge" or "KVM64" it works. pfsense and FreeBSD 8.3 is booting.

So I left KVM disabled and followed the VirtIO tutorial on the pfsense docs (https://doc.pfsense.org/index.php/VirtIO_Driver_Support). It unfortunately just worked for HDD and for the ballooning driver. If I change the NIC to VirtIO I do not get a connection. With the emulated E1000 it works.


So for me it looks like I need to do some other configurations on pfsense 2.1/FreeBSD 8.3 or on Proxmox 3.1 to make it work properly with KVM virtualization. I am not sure if this is a Proxmox problem because it works with a Windows guest.

I would really appreciate any help and tips because pfsense is very slow in my VM.


Thank you for your help in advance!
 
Re: PVE3.1 + FreeBSD8.3 / pfsense - KVM Hardware Virtualization and VirtIO not possib

Nachtfalke said:
Hi,

I am running Proxmox VE 3.1 on a Fujitsu Siemens Primergy TX120 S6 Server with Intel Xeon E1220v2 CPU.
I have virtualisation support enabled on the Fujitsu BIOS.

I have two VMs running, one is Windows Server 2008R2, and I have KVM Hardware Virtualization enabled on Proxmox. This is working. I installed the VirtIO drivers for HDD and NICs and all worked.

The second VM is running pfsense 2.1 AMD64 (FreeBSD 8.3). If I enable KVM Hardware Virtualization on Proxmox pfsense stops booting after the menu. I tried with disabling ACPI, Single User mode, and so on. It does not boot. It is the same with FreeBSD 8.3 bootonly ISO. If I disable KVM Hardware Virtualization and change the CPU type from "host" to "Sandy Bridge" or "KVM64" it works. pfsense and FreeBSD 8.3 is booting.

So I left KVM disabled and followed the VirtIO tutorial on the pfsense docs (https://doc.pfsense.org/index.php/VirtIO_Driver_Support). It unfortunately just worked for HDD and for the ballooning driver. If I change the NIC to VirtIO I do not get a connection. With the emulated E1000 it works.


So for me it looks like I need to do some other configurations on pfsense 2.1/FreeBSD 8.3 or on Proxmox 3.1 to make it work properly with KVM virtualization. I am not sure if this is a Proxmox problem because it works with a Windows guest.

I would really appreciate any help and tips because pfsense is very slow in my VM.


Thank you for your help in advance!
Click to expand...

Change the CPU emulation to "core2duo". I'm not sure if it's the new Intel CPUs causing problems in BSD or proxmox but it fixes the problem. I still use the E1000 drivers rather than virtio. Even a lowly i3 can handle 100mbit of NAT translations through the E1000 drivers.
 
Re: PVE3.1 + FreeBSD8.3 / pfsense - KVM Hardware Virtualization and VirtIO not possib

Hi,

thank you for your feedback. This - of course - helps to boot the system but it does not solve the problem. I still cannot enable KVM Hardware Virtualization within Proxmox 3.1 and boot FreeBSD 8.3 / pfsense.
I read something here in the forum about a BIOS file which could be replaced to fix some problems but I am not sure if this is related to my problem.

Nevertheless, thank you for your help. Perhaps someone else can help.
 
Re: PVE3.1 + FreeBSD8.3 / pfsense - KVM Hardware Virtualization and VirtIO not possib

There is a distinction: the server must have (check bios setting) hw virtualization enabled.
Then in a "per VM basis" you can disable hw virtualization (under "Options") if it, for that specific vm, gives you some problem.
So for a good working Proxmox server you have to be sure that hw virtualization of your server is working fine, and then if you really need, you can disable for a specific VM if it solves some problem (and you can accept the consequent slowliness).
 
Re: PVE3.1 + FreeBSD8.3 / pfsense - KVM Hardware Virtualization and VirtIO not possib

Hi,

thank you for your response. I of course enabled Intel VT-d and all the other Virtualization options. Further the BIOS shows me that Virtualizations is supported for that specific CPU. To make sure I shutdown every VM and I completely shutdown the server after I made changes in the BIOS.

But as I said in my first post, KVM Hardware Virtualiszation is working for the VM with Windows Server 2008R2 but it is not working for the VM with FreeBSD 8.3 / pfsense which is running on the same server hardware.

I can boot FreeBSD when I disable KVM Hardware Virtualization for this specific VM but this makes the VM really slow so this is not an option.
 
Re: PVE3.1 + FreeBSD8.3 / pfsense - KVM Hardware Virtualization and VirtIO not possib

Thank you for that link.

Unfortunately the problem is not the HDD. I installed pfsense with IDE drives and after that I switched to VirtIO and I changed config files on pfsense. This is how it is documented on the pfsense wiki page. After that I can boot with HDD and VirtIO mode but still only with KVM Hardware Virtualization disabled. :(
 
Re: PVE3.1 + FreeBSD8.3 / pfsense - KVM Hardware Virtualization and VirtIO not possib

I tried different CPU types but it still does nott boot if KVM Harware Virtualization is enabled. Further I do not recognize any performance improvements when changing the CPU types.
So a little question which is not directly related to the topic:

Which emulated CPU should I used best if "host" ist not working because of the missing ability to use "KVM Hardware Virtualization" ?


I added some pfsense outputs:
[2.1-RELEASE][admin@pfsense3.hpa]/root(1): kldstat
Id Refs Address Size Name
1 20 0xffffffff80100000 156a4e8 kernel
2 5 0xffffffff8166b000 4ce8 virtio.ko
3 1 0xffffffff81670000 5ca0 virtio_pci.ko
4 1 0xffffffff81676000 af78 if_vtnet.ko
5 1 0xffffffff81681000 2f70 virtio_balloon.ko
6 1 0xffffffff81684000 5030 virtio_blk.ko
7 1 0xffffffff81812000 133e50 zfs.ko
8 1 0xffffffff81946000 1fcd opensolaris.ko
Click to expand...

[2.1-RELEASE][admin@pfsense3.hpa]/root(4): ls -l /boot/kernel/*virtio*
-r-xr-xr-x 1 root wheel 18144 Sep 12 00:51 /boot/kernel/virtio.ko
-r-xr-xr-x 1 root wheel 11280 Sep 12 00:51 /boot/kernel/virtio_balloon.ko
-r-xr-xr-x 1 root wheel 18544 Sep 12 00:51 /boot/kernel/virtio_blk.ko
-r-xr-xr-x 1 root wheel 21752 Sep 12 00:51 /boot/kernel/virtio_pci.ko
-r-xr-xr-x 1 root wheel 34648 Sep 12 00:51 /boot/kernel/virtio_scsi.ko
Click to expand...

[2.1-RELEASE][admin@pfsense3.hpa]/root(5): cat /boot/loader.conf.local
virtio_load="YES"
virtio_pci_load="YES"
if_vtnet_load="YES"
virtio_balloon_load="YES"
virtio_blk_load="YES"
Click to expand...

[2.1-RELEASE][admin@pfsense3.hpa]/root(6): cat /etc/fstab
# Device Mountpoint FStype Options Dump Pass#
/dev/vtbd0s1a / ufs rw 1 1
/dev/vtbd0s1b none swap sw 0 0
Click to expand...

root@ProxmoxVE2:~# pveversion -v
proxmox-ve-2.6.32: 3.1-114 (running kernel: 2.6.32-26-pve)
pve-manager: 3.1-21 (running version: 3.1-21/93bf03d4)
pve-kernel-2.6.32-20-pve: 2.6.32-100
pve-kernel-2.6.32-19-pve: 2.6.32-96
pve-kernel-2.6.32-24-pve: 2.6.32-111
pve-kernel-2.6.32-22-pve: 2.6.32-107
pve-kernel-2.6.32-26-pve: 2.6.32-114
pve-kernel-2.6.32-18-pve: 2.6.32-88
pve-kernel-2.6.32-23-pve: 2.6.32-109
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.0-2
pve-cluster: 3.0-8
qemu-server: 3.1-8
pve-firmware: 1.0-23
libpve-common-perl: 3.0-8
libpve-access-control: 3.0-7
libpve-storage-perl: 3.0-17
pve-libspice-server1: 0.12.4-2
vncterm: 1.1-4
vzctl: 4.0-1pve4
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.4-17
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.1-1
Click to expand...

root@ProxmoxVE2:~# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 58
Stepping: 9
CPU MHz: 3092.913
BogoMIPS: 6185.82
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 8192K
NUMA node0 CPU(s): 0-3
Click to expand...

And here some specifications from my mainboard and CPU:

Motherboard chipset is:
Intel® C202
http://www.intel.de/content/www/de/de/chipsets/server-chipsets/server-chipset-c202.html
Which should support "Intel® Directed-I/O-Virtualisierungstechnik (Intel® VT-d)"

And the CPU is:
Intel Xeon E1220v2
http://ark.intel.com/products/52269/

--- Update ---
Still no luck after upgrading the BIOS of the Fujitsu Server to latest BIOS. Checked the BIOS setting again and still all virtualization options enabled.
 
Last edited:
Re: PVE3.1 + FreeBSD8.3 / pfsense - KVM Hardware Virtualization and VirtIO not possib

Nachtfalke said:
I tried different CPU types but it still does nott boot if KVM Harware Virtualization is enabled. Further I do not recognize any performance improvements when changing the CPU types.
So a little question which is not directly related to the topic:

Which emulated CPU should I used best if "host" ist not working because of the missing ability to use "KVM Hardware Virtualization" ?


I added some pfsense outputs:












And here some specifications from my mainboard and CPU:

Motherboard chipset is:
Intel® C202
http://www.intel.de/content/www/de/de/chipsets/server-chipsets/server-chipset-c202.html
Which should support "Intel® Directed-I/O-Virtualisierungstechnik (Intel® VT-d)"

And the CPU is:
Intel Xeon E1220v2
http://ark.intel.com/products/52269/

--- Update ---
Still no luck after upgrading the BIOS of the Fujitsu Server to latest BIOS. Checked the BIOS setting again and still all virtualization options enabled.
Click to expand...


Have tried this with the Pfsense amd64 for years. It stopped working where the CPU was Haswell and newer...

Try using the Pfsense i386 instead....

/Michael
 
Re: PVE3.1 + FreeBSD8.3 / pfsense - KVM Hardware Virtualization and VirtIO not possib

Hi Michael,

thank you very much for your feedback. I tried with the i386 pfsense 2.1 version and KVM Hardware Virtualization enabled and CPU type set to "host" and it worked without problems. The CD booted.
So it looks like something in the AMD64 build of FreeBSD 8.3 / pfsense 2.1 and the virtio drivers isn't working correctly, right!?

Did you post that somewhere on the pfsense forum?
I posted a thread here on pfsense forum some days ago and I will post a link there to this thread. Perhaps some new drivers can fix that.
http://forum.pfsense.org/index.php/topic,68717.0.html

Thank you very much!
 
Re: PVE3.1 + FreeBSD8.3 / pfsense - KVM Hardware Virtualization and VirtIO not possib

It has nothing to do with the virtio driver. It is a more deeply rooted problem inside KVM in its emulation of CPU's over Intel chipsets in combination with *BSD amd64.
 
Re: PVE3.1 + FreeBSD8.3 / pfsense - KVM Hardware Virtualization and VirtIO not possib

mir said:
It has nothing to do with the virtio driver. It is a more deeply rooted problem inside KVM in its emulation of CPU's over Intel chipsets in combination with *BSD amd64.
Click to expand...

Mir is right...

Yes I have made a previous post some time ago...

On all Sandy Bridge CPU and older there is not problems at all.

/Michael
 
Re: PVE3.1 + FreeBSD8.3 / pfsense - KVM Hardware Virtualization and VirtIO not possib

Hi,

so to make it clearer for me who has to fix this bug / problem? Is this qemu related?

Thank you for your response!
 
Re: PVE3.1 + FreeBSD8.3 / pfsense - KVM Hardware Virtualization and VirtIO not possib

Two solutions to the problem:
1) A bug in the RHEL kernel used in Proxmox. Requires intervention from Redhat and/or Linux kernel developers.
2) A bug in qemu/kvm. Requires intervention from qemu/kvm developers. Since the version of qemu/kvm in current Proxmox is rather dated chances are this bug is fixed in newer qemu/kvm (qemu/kvm in Proxmox is 1.4 and newest stable is 1.6). Replacing 1.4 with 1.6 is not trivial so this might not happen until Proxmox 3.2 or 4.0 if that is next major release.
 
Re: PVE3.1 + FreeBSD8.3 / pfsense - KVM Hardware Virtualization and VirtIO not possib

mir said:
Two solutions to the problem:
1) A bug in the RHEL kernel used in Proxmox. Requires intervention from Redhat and/or Linux kernel developers.
2) A bug in qemu/kvm. Requires intervention from qemu/kvm developers. Since the version of qemu/kvm in current Proxmox is rather dated chances are this bug is fixed in newer qemu/kvm (qemu/kvm in Proxmox is 1.4 and newest stable is 1.6). Replacing 1.4 with 1.6 is not trivial so this might not happen until Proxmox 3.2 or 4.0 if that is next major release.
Click to expand...

Thank you very much!

This really helps me. So I will probably first go with pfsense 2.1 i386 and hopefully in the near fute any part of the software fixes this bug and I can upgrade to AMD64.
Did you test on any other hardware if it is working with qemu 1.6?

Thank you very much to the community for this conversation!
I really appreciate it.
 
Re: PVE3.1 + FreeBSD8.3 / pfsense - KVM Hardware Virtualization and VirtIO not possib

I have not tried qemu-1.6 myself. To be able to do that you can try one of the following solutions:
- Fedora testing: http://fedoraproject.org/en/get-prerelease
- Debian unstable
- Ubuntu server 13.10 has qemu 1.5
- Slackware latest
- Arch latest

So you see all enterprise distributions like RHEL, Debian Wheezy, Suse, CentOS, Scientific Linux, and Proxmox share the same version of qemu which is 1.4
 
Re: PVE3.1 + FreeBSD8.3 / pfsense - KVM Hardware Virtualization and VirtIO not possib

mir said:
2) A bug in qemu/kvm. Requires intervention from qemu/kvm developers. Since the version of qemu/kvm in current Proxmox is rather dated chances are this bug is fixed in newer qemu/kvm
Click to expand...

What patch do you talk about (maybe we can backport)?
 
Re: PVE3.1 + FreeBSD8.3 / pfsense - KVM Hardware Virtualization and VirtIO not possib

dietmar said:
What patch do you talk about (maybe we can backport)?
Click to expand...
I do not know of any specific patch. I just assumes a newer version of qemu will support the latest chipsets from Intel. It should also be noted that current enterprise servers first recently starts shipping with Haswell and Ivy bridge so maybe Redhat will backport support for these chipsets very soon.

p. 32 http://rhsummit.files.wordpress.com/2013/06/sarathy_t_1040_kvm_hypervisor_roadmap_and_overview.pdf
States Haswell and Ivy bridge are new to 6.4 so maybe is still has bugs?

Latest kernel for RHEL 6.4 is 2.6.32-358.23.2.el6. Is this the one in Proxmox?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom