PCI / PCIe passthrough

tin · Sep 1, 2011

What's the current status of PCI passthrough?
What do I need to do to make it happen? I ran "qm set 123 -hostpci x:x:x *" which seems to have worked, but lspci on the VM doesn't show the card.

Longer version:
I've got a PCIe FXS/FXO card which I want to pass through to a virtual machine acting as a VoIP server. My preference is to run Trixbox for a number of reasons, which means (without lots of stuffing around) I need to use KVM. I actually already have a KVM running Trixbox from prior testing (without a card).
If KVM isn't going to work with the PCI card, can I use an OpenVZ container? I gather I'd need to load the module for the card in the host kernel to do this - anyone know if the dahdi modules are easily set up on a Proxmox server? I don't want to create a maintenance nightmare for myself by having to manually install (or worse - compile) modules every time I do an update.

Edit: I should probably mention this is on a Dell Poweredge R515 which uses 2x Opteron 4130.

tin · Sep 3, 2011

OK, I think I have it working now... For both my own reference and anyone else who comes across this, here's what I've done.

First, I enabled "DMA Virtualisation" in the BIOS. This seems to be what Dell have labelled IOMMU.
Next, as documented in this page, I ran (though with the corrected PCI ids, etc):

Code:

echo "8086 10b9" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind

Now, here's where things got confusing. The device still didn't show when I booted up the VM (which had been down during the above). I then ran the "device_add pci-assign,host=01:00.0,id=mydevice" command in the "monitor" window, which completed without error, but the device still didn't show until I rebooted.

So partial success. I doubt this will survive a reboot at this point. And I suspect it won't even survive a full stop/start of the VM... But at least it's passing the device through correctly now.

Edit: Further info - when I tried to reboot the VM, the KVM process locked up and couldn't be killed (even with -9). I had to restart the entire server. Not the greatest situation...

tin · Sep 10, 2011

I just attempted to reboot the VM again, only this time after running "device_del mydevice".... Same result.

The kernel on the host is spewing out the following (CPU involved changes though), if it means anything to anyone:

Code:

Sep 10 11:37:10     kernel         BUG: soft lockup - CPU#7 stuck for 61s! [kvm:10118]
Sep 10 11:37:10     kernel         Modules linked in: tun kvm_amd kvm vzethdev vznetdev simfs vzrst vzcpt vzdquota vzmon vzdev xt_tcpudp ipt_REDIRECT iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 xt_length xt_hl xt_tcpmss xt_TCPMSS iptable_mangle iptable_filter xt_multiport xt_limit xt_dscp ipt_REJECT ip_tables x_tables vzevent ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi bridge stp dcdbas snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw psmouse pcspkr k10temp edac_core edac_mce_amd i2c_piix4 i2c_core evdev shpchp pci_hotplug power_meter button processor ext3 jbd mbcache usbhid hid dm_mirror dm_region_hash dm_log dm_snapshot ses enclosure ata_generic pata_atiixp ehci_hcd ohci_hcd ahci usbcore nls_base libata bnx2 megaraid_sas r8169 mii thermal fan thermal_sys [last unloaded: scsi_wait_scan]
Sep 10 11:37:10     kernel         CPU 7:
Sep 10 11:37:10     kernel         Modules linked in: tun kvm_amd kvm vzethdev vznetdev simfs vzrst vzcpt vzdquota vzmon vzdev xt_tcpudp ipt_REDIRECT iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 xt_length xt_hl xt_tcpmss xt_TCPMSS iptable_mangle iptable_filter xt_multiport xt_limit xt_dscp ipt_REJECT ip_tables x_tables vzevent ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi bridge stp dcdbas snd_pcm snd_timer snd soundcore snd_page_alloc serio_raw psmouse pcspkr k10temp edac_core edac_mce_amd i2c_piix4 i2c_core evdev shpchp pci_hotplug power_meter button processor ext3 jbd mbcache usbhid hid dm_mirror dm_region_hash dm_log dm_snapshot ses enclosure ata_generic pata_atiixp ehci_hcd ohci_hcd ahci usbcore nls_base libata bnx2 megaraid_sas r8169 mii thermal fan thermal_sys [last unloaded: scsi_wait_scan]
Sep 10 11:37:10     kernel         Pid: 10118, comm: kvm Tainted: G W 2.6.32-4-pve #1 feoktistov PowerEdge R515
Sep 10 11:37:10     kernel         RIP: 0010:[<ffffffff81315a48>] [<ffffffff81315a48>] _spin_lock+0x15/0x1b
Sep 10 11:37:10     kernel         RSP: 0018:ffff8803231ebb70 EFLAGS: 00000297
Sep 10 11:37:10     kernel         RAX: 0000000000007735 RBX: ffff8804138ac800 RCX: 000000000000160b
Sep 10 11:37:10     kernel         RDX: 0000000000007734 RSI: 000000000000000b RDI: ffff8804138ac800
Sep 10 11:37:10     kernel         RBP: ffffffff8101172e R08: 0000000000000800 R09: 000000000000000a
Sep 10 11:37:10     kernel         R10: 0000000000000001 R11: 0000000000000096 R12: 00000000000016d1
Sep 10 11:37:10     kernel         R13: ffffffffa036167d R14: 0542d2462b4000ac R15: ffffffff81321200
Sep 10 11:37:10     kernel         FS: 0000000040ecf950(0063) GS:ffff880229460000(0000) knlGS:0000000000000000
Sep 10 11:37:10     kernel         CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 10 11:37:10     kernel         CR2: 0000000000000000 CR3: 00000003471db000 CR4: 00000000000006e0
Sep 10 11:37:10     kernel         DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Sep 10 11:37:10     kernel         DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Sep 10 11:37:10     kernel         Call Trace:
Sep 10 11:37:10     kernel         [<ffffffffa0379ae6>] ? kvm_pic_set_irq+0x20/0xc6 [kvm]
Sep 10 11:37:10     kernel         [<ffffffffa036167d>] ? kvm_set_irq+0x5e/0x100 [kvm]
Sep 10 11:37:10     kernel         [<ffffffffa03616e5>] ? kvm_set_irq+0xc6/0x100 [kvm]
Sep 10 11:37:10     kernel         [<ffffffff81036195>] ? gup_pud_range+0x15e/0x19b
Sep 10 11:37:10     kernel         [<ffffffffa035dea3>] ? kvm_assigned_dev_ack_irq+0x2b/0x5c [kvm]
Sep 10 11:37:10     kernel         [<ffffffffa0361601>] ? kvm_notify_acked_irq+0x87/0xa5 [kvm]
Sep 10 11:37:10     kernel         [<ffffffffa0379fd0>] ? kvm_pic_reset+0x6e/0xbc [kvm]
Sep 10 11:37:10     kernel         [<ffffffffa037a101>] ? picdev_write+0xe3/0x31e [kvm]
Sep 10 11:37:10     kernel         [<ffffffffa035c1f5>] ? kvm_io_bus_write+0x37/0x50 [kvm]
Sep 10 11:37:10     kernel         [<ffffffffa03685ac>] ? kvm_emulate_pio+0x131/0x155 [kvm]
Sep 10 11:37:10     kernel         [<ffffffffa03a866e>] ? skip_emulated_instruction+0x4d/0xbb [kvm_amd]
Sep 10 11:37:10     kernel         [<ffffffffa036b12b>] ? kvm_arch_vcpu_ioctl_run+0x7ed/0xa44 [kvm]
Sep 10 11:37:10     kernel         [<ffffffff810e7abb>] ? virt_to_head_page+0x9/0x2a
Sep 10 11:37:10     kernel         [<ffffffffa035d9d1>] ? kvm_vcpu_ioctl+0xf1/0x4e6 [kvm]
Sep 10 11:37:10     kernel         [<ffffffff8105d692>] ? __dequeue_signal+0x10c/0x133
Sep 10 11:37:10     kernel         [<ffffffff8105c275>] ? recalc_sigpending+0x12/0x3a
Sep 10 11:37:10     kernel         [<ffffffff8105edcc>] ? dequeue_signal+0x9c/0x10d
Sep 10 11:37:10     kernel         [<ffffffff810fd532>] ? vfs_ioctl+0x21/0x6c
Sep 10 11:37:10     kernel         [<ffffffff810fda80>] ? do_vfs_ioctl+0x48d/0x4cb
Sep 10 11:37:10     kernel         [<ffffffff810fdafb>] ? sys_ioctl+0x3d/0x5c
Sep 10 11:37:10     kernel         [<ffffffff81010c12>] ? system_call_fastpath+0x16/0x1b

dietmar · Sep 10, 2011

What kernel do you use?

Note: you should use the latest kernel from the pvetest repository

tin · Sep 10, 2011

I'm using 2.6.32-4-pve
Do you know whether the passthrough works significantly better in the newer kernel?

tom · Sep 10, 2011

tin said:
I'm using 2.6.32-4-pve
Do you know whether the passthrough works significantly better in the newer kernel?

no, but in any case we never fix old kernels so in the case of any problems you should test with the latest kernel (pvetest).

tin · Sep 12, 2011

Just tried the 2.6.32-6-pve that pvetest seems to have put on - didn't even want to boot up properly. Got a kernel error of some sort (couldn't see - ran off the page and froze up). I'm suspecting the "netjet" ISDN module may be involved - this doesn't load on the -4 kernel, but does on the -6. The device it's saying it finds is actually the PSTN line card I'm trying to pass through.

I'm planning to try blacklisting that module to see if -6 works, and also will give the 2.6.35 kernel a go. Failing that, I think I'll have to build a physical machine

c0mputerking · Oct 12, 2011

I am also trying to get pci-x pass though to work, did you ever manage to get things going??

tin · Oct 12, 2011

c0mputerking said:
I am also trying to get pci-x pass though to work, did you ever manage to get things going??

It's currently working fine as far as I can tell, aside from still causing that CPU lockup issue on the host when shutting down the guest.
However, this seems to only be happening some of the time. Haven't had the time or guts to fully work out what triggers it, but I THINK it might only be happening when the guest has be shut down once already... Rerunning the pci_stub echo stuff after shutting down but before restarting the guest may help, but I haven't tried yet.

One thing that improved things dramatically for me - use the VM's config file to specify the PCI card rather than using the console... Add a line like this one:

args: -device pci-assign,host=03:00.0,id=pstncard

p3x-749 · Oct 12, 2011

tin said:
It's currently working fine as far as I can tell, aside from still causing that CPU lockup issue on the host when shutting down the guest.
However, this seems to only be happening some of the time. Haven't had the time or guts to fully work out what triggers it, but I THINK it might only be happening when the guest has be shut down once already... Rerunning the pci_stub echo stuff after shutting down but before restarting the guest may help, but I haven't tried yet.

...that's very interesting.
What kernel are you running as of today?

One thing that improved things dramatically for me - use the VM's config file to specify the PCI card rather than using the console... Add a line like this one:

Hmm, ok..is this all there is to it, that makes the pastrough config stick upon reboot?
What about the unbind/bind of the pci/pci-stub drivers, you mentioned in your second post? Is this still needed? (I am assuming this won't survive a reboot either)

I am currently stuck with an ESXi installation on one of my severs, because there I can use vmdirectpath with the PCIe-cards.
....would love to do the same with Proxmox/KVM and get rid of the nasty Windoze managemnet VM!!

TIA,

p3x-749

tin · Oct 12, 2011

I'm running 2.6.32-4-pve. I tried 2.6.32-6-pve, but it doesn't seem to have the IOMMU support required for the passthrough.

I'm pretty sure the PCI Stub stuff I mentioned is required. I made a script and stuck it at /etc/rc2.d/S19dahdi-card-to-pci-stub
Having it in there makes it boot up and start the VM fine on it's own.

p3x-749 · Oct 12, 2011

...thanks for the fast response and the info, tin.

dietmar · Oct 12, 2011

tin said:
I'm running 2.6.32-4-pve. I tried 2.6.32-6-pve, but it doesn't seem to have the IOMMU support required for the passthrough.

What option is missing?

c0mputerking · Oct 12, 2011

I cannot seem to make this work as stated earlier i an using a dell power edge T110 II with visualization enabled in the BIOS. Also running Proxmox 2.0 with this kernel installed. Linux sun 2.6.32-6-pve #1 SMP Mon Sep 26 10:35:47 CEST 2011 x86_64 GNU/Linux

The device on the host i would like to pass through is this lspci
04:05.0 Multimedia video controller: Bluecherry BC-H16480A 16 port H.264 video and audio encoder / decoder
lspci -n
04:05.0 0400: 1bb3:5310

I have tried adding the following lines to my /etc/pve/nodes/sun/qemu-server/104.conf, however in line 4 i was not sure what to use for id so just made it up. lines 1,2, and 3 all give errors I have included the error for line 3 below, and proxmox will not start the VM. Using line 4 the machine starts but as far as i can tell the card is not detected and does not seem to work.

#args: -pcidevice host=04:05.0 -cpu hosta
#args: -pcidevice host=04:05.0
#args: -device pci-assign,host=04:05.0,id=bluecherry
#hostpci 04:05.0

start failed: command '/usr/bin/kvm -id 104 -chardev socket,id=monitor,path=/var/run/qemu-server/104.mon,server,nowait -mon chardev=monitor,mode=readline -vnc unix:/var/run/qemu-server/104.vnc,x509,password -pidfile /var/run/qemu-server/104.pid -daemonize -readconfig /usr/share/qemu-server/pve-usb.cfg -device usb-tablet,bus=ehci.0,port=6 -name bluccherry2 -smp sockets=1,cores=1 -nodefaults -boot menu=on,order=cdn -vga cirrus -tdf -k en-us -drive file=/var/lib/vz/iso/turnkey-core-11.2-lucid-x86.iso,if=none,id=drive-ide2,media=cdrom,aio=native -device ide-drive,bus=ide.1,unit=0,drive=drive-ide2,id=device-ide2 -drive file=/dev/vg1/vm-104-disk-1,if=none,id=drive-ide0,aio=native,boot=on -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0,id=device-ide0 -m 512 -netdev type=tap,id=net0,ifname=tap104i0,script=/var/lib/qemu-server/pve-bridge -device rtl8139,romfile=,mac=22:BD:58:6A:16:79,netdev=net0,bus=pci.0,addr=0x12 -cpuunits 1000 -device pci-assign,host=04:05.0,id=bluecherry' failed: exit code 1 (500)

I did manage to get the pci stub stuff to complete without error i think! but still the card is not appearing in the KVM
echo "1bb3 5310" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:04:05.0 > /sys/bus/pci/devices/0000\:04\:05.0/driver/unbind
echo 0000:04:05.0 > /sys/bus/pci/drivers/pci-stub/bind

dietmar · Oct 13, 2011

c0mputerking said:
echo "1bb3 5310" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:04:05.0 > /sys/bus/pci/devices/0000\:04\:05.0/driver/unbind
echo 0000:04:05.0 > /sys/bus/pci/drivers/pci-stub/bind

Well, this is what option 'hostpci 04:05' do for you.

tin · Oct 13, 2011

dietmar said:
What option is missing?

Well, to be honest, I'm not sure. I just know when I boot the newer kernel, I get no IOMMU (or AMD-Vi) info in the kernel output, and the VM using it won't start. I'm on an AMD system if that helps - maybe it's there for Intel but not AMD.

I haven't had time to look at the kernel configs to try and work out what's missing.

Edit:

dietmar said:
Well, this is what option 'hostpci 04:05' do for you.

Does it? That could save some mucking around

c0mputerking · Oct 13, 2011

Ok that is good to know, and is a pain to type in too haha i am glad that proxmox/kvm does it for me. However the card still does not show in the VM with a lspci and does not seem to work? Anyway to check that it is being properly forwarded?

dietmar · Oct 13, 2011

tin said:
I haven't had time to look at the kernel configs to try and work out what's missing.

Well, the kernel does not set CONFIG_DMAR_DEFAULT_ON, so you need to tur that on manually:

http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM said:
NOTE: If you still get an error "No IOMMU found." Check dmesg for errors suggesting your BIOS is broken. Another possible reason: CONFIG_DMAR_DEFAULT_ON is not set. In that case, pass "intel_iommu=on" as kernel parameter to enable it. AMD uses different kernel parameter than Intel, on AMD you need to pass "iommu=pt iommu=1".

Does that help?

c0mputerking · Oct 13, 2011

This is a dmesg output is my system pci pass through enabled or not? what does it mean CONFIG_DMAR_DEFAULT_ON do i need it to get pass through working? Also my KVM boots with this line hostpci 04:05.0 should it be 04:05 is the last 0 not needed?

dmesg | grep -e DMAR -e IOMMU
ACPI: DMAR 00000000bf7e8000 000B0 (v01 INTEL SNB 00000001 INTL 00000001)
DMAR: Host address width 36
DMAR: DRHD base: 0x000000fed91000 flags: 0x1
IOMMU fed91000: ver 1:0 cap c9008020660262 ecap f010da
DMAR: RMRR base: 0x000000bf65a000 end: 0x000000bf65bfff
DMAR: No ATSR found

tin · Nov 6, 2011

I've got "amd_iommu=on iommu=pt iommu=1", but now getting a message telling me I need to "allow_unsafe_assigned_interrupts=1" in the kvm module options... Does anyone know exactly what that option does? The vague info I found suggests it's to stop malicious acts on guests reaching the host, but is that all? Is long term (as in days and weeks, not years) stability of the host effected at all assuming no malicious code is present?

I might test what happens with the 2.6.35 kernel later today, along with trying that kvm option. All depends if my spare time coincides with no one using the server

PCI / PCIe passthrough

Renowned Member

Renowned Member

Renowned Member

Proxmox Staff Member

Renowned Member

Proxmox Staff Member

Renowned Member

Active Member

Renowned Member

Active Member

Renowned Member

Active Member

Proxmox Staff Member

Active Member

Proxmox Staff Member

Renowned Member

Active Member

Proxmox Staff Member

Active Member

Renowned Member