After Upgrade (RC -> final) -> PCI Passthrough Problems

Hi Dietmar,

yes it works for me!
Thank's a lot!

First i upgraded again to "qemu-server_2.0-33_amd64.deb" by "aptitude update" & "aptitude full-upgrade".
Then, i installed the new Package "pve-qemu-kvm_1.0-9_amd64.deb".
And now, it works as before the upgrade to final, without the additional argument "prefer_msi=on".


pve-manager: 2.0-54 (pve-manager/2.0/4b59ea39)
running kernel: 2.6.32-10-pve
proxmox-ve-2.6.32: 2.0-63
pve-kernel-2.6.32-10-pve: 2.6.32-63
pve-kernel-2.6.32-7-pve: 2.6.32-60
lvm2: 2.02.88-2pve2
clvm: 2.02.88-2pve2
corosync-pve: 1.4.1-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.8-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.7-2
pve-cluster: 1.0-26
qemu-server: 2.0-33
pve-firmware: 1.0-15
libpve-common-perl: 1.0-23
libpve-access-control: 1.0-17
libpve-storage-perl: 2.0-16
vncterm: 1.0-2
vzctl: 3.0.30-2pve2
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.0-9
ksm-control-daemon: 1.1-1
Click to expand...


Alex
 
trendco said:
Hi Dietmar,

yes it works for me!
Thank's a lot!

First i upgraded again to "qemu-server_2.0-33_amd64.deb" by "aptitude update" & "aptitude full-upgrade".
Then, i installed the new Package "pve-qemu-kvm_1.0-9_amd64.deb".
And now, it works as before the upgrade to final, without the additional argument "prefer_msi=on".
Alex
Click to expand...

Hello,
today I also update by "aptitude update" & "aptitude full-upgrade".
=> PCI Passthrough doesn't work - as after the last updates, since > pve-qemu-kvm_1.0-7
I get:
Code: 
dmesg

[   18.325389] SAA716x FF 0000:00:10.0: PCI INT A disabled
[   18.325396] SAA716x FF: probe of 0000:00:10.0 failed with error -1

Also Ditmars fix "pve-qemu-kvm_1.0-9_amd64.deb" doesn't work for me

the only solution for my PCI passthrough is to downgrade to version pve-qemu-kvm_1.0-5
Code: 
root@fcprox01:~# pveversion -v
pve-manager: 2.1-1 (pve-manager/2.1/f9b0f63a)
running kernel: 2.6.32-11-pve
proxmox-ve-2.6.32: 2.0-66
pve-kernel-2.6.32-10-pve: 2.6.32-63
pve-kernel-2.6.32-11-pve: 2.6.32-66
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.3-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.8-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.7-2
pve-cluster: 1.0-26
qemu-server: 2.0-39
pve-firmware: 1.0-15
libpve-common-perl: 1.0-27
libpve-access-control: 1.0-21
libpve-storage-perl: 2.0-18
vncterm: 1.0-2
vzctl: 3.0.30-2pve5
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.0-5     [B]<<==[/B]
ksm-control-daemon: 1.1-1
root@fcprox01:~#

maxprox
 
maxprox,

Maybe it has been updated since your last post, but I just updated and it works for me. I use "hostpci0: XX:XX.X" in the .conf file (no msi or intx options). However, I found that passing through my PCIe device works, while PCI does not. Maybe it's due to:
- The VT-d specification states that all conventional PCI devices behind a PCIe-to-PCI bridge have to be assigned to the same domain.
- PCIe devices do not have this restriction.
Click to expand...
(from http://wiki.xensource.com/xenwiki/VTdHowTo)
This may be specific to my motherboard, which probably uses a PCIe-to-PCI bridge.

Also, on the Proxmox wiki page about pci passthrough (http://pve.proxmox.com/wiki/Pci_passthrough), I had to use different options, as what's provided there resulted in a kernel panic for me. Here is what works for my AMD Opteron 4130:
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=on,pt"
Anyone else experience the same? If so, maybe we should update the wiki.
 
jram said:
maxprox,

Maybe it has been updated since your last post, but I just updated and it works for me. I use "hostpci0: XX:XX.X" in the .conf file (no msi or intx options). However, I found that passing through my PCIe device works, while PCI does not. Maybe it's due to:
(from http://wiki.xensource.com/xenwiki/VTdHowTo)
This may be specific to my motherboard, which probably uses a PCIe-to-PCI bridge.

Also, on the Proxmox wiki page about pci passthrough (http://pve.proxmox.com/wiki/Pci_passthrough), I had to use different options, as what's provided there resulted in a kernel panic for me. Here is what works for my AMD Opteron 4130:
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=on,pt"
Anyone else experience the same? If so, maybe we should update the wiki.
Click to expand...

Hello jram,

thank you for your answer. Unfortunately it does not work for me.
I do a new full-upgrade with exact the same result I wrote above.
I work with a supermicro X8SIL-F board - Intel 3420 Chipset with a Xeon X3450 QC CPU
My problem child is a Technotrend S2-6400 full-HD PCIe x1 TV-card.
The only solution for me is
1:
downgrade to pve-qemu-kvm 1.0-5
2:
Kernel option in the /etc/default/grub file =>
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
3:
cat /etc/pve/nodes/fcprox01/qemu-server/109.conf =>
Code: 
hostpci0: 03:00.0
4:
a new file with only one row:
Code: 
:~# cat /etc/modprobe.d/kvm_iommu_map_guest.conf 
options kvm allow_unsafe_assigned_interrupts=1

what is the ",pt" in your kernel options for?
EDIT: I see it in the wiki, it's for AMD systems

I will have a look with the next version pve-qemu-kvm 1.0-10

regards,
maxprox
 
Last edited:
jram said:
maxprox,

Maybe it has been updated since your last post, but I just updated and it works for me. I use "hostpci0: XX:XX.X" in the .conf file (no msi or intx options). However, I found that passing through my PCIe device works, while PCI does not. Maybe it's due to:
(from http://wiki.xensource.com/xenwiki/VTdHowTo)
This may be specific to my motherboard, which probably uses a PCIe-to-PCI bridge.
Click to expand...

Hi,

Can you c/p your pveversion -v? I have been trying for days now to get this working but I can't. Maybe I can find a difference with your setup.

Also, on the Proxmox wiki page about pci passthrough (http://pve.proxmox.com/wiki/Pci_passthrough), I had to use different options, as what's provided there resulted in a kernel panic for me. Here is what works for my AMD Opteron 4130:
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=on,pt"
Anyone else experience the same? If so, maybe we should update the wiki.
Click to expand...

This is correct, I also get the kernel panic if I use what is on the wiki. I already found out that amd_iommu is what you need, I didn't use the on with the iommu option but that doesn't seem to make a difference.

I know this should work because I previously had esXi on the same hardware and there I could easily pass-through my PCIe DVB-S2 card to my virtual machine. Now I'm trying to do the same in Proxmox but it fails no matter what I do. Currently I'm at the state where I reverted everything back to a "clean" install, other then the boot options. My motherboard is from ASRock with an 890FX chipset

Code: 
root@barabas:~# pveversion -v
pve-manager: 2.1-1 (pve-manager/2.1/f9b0f63a)
running kernel: 2.6.32-11-pve
proxmox-ve-2.6.32: 2.0-66
pve-kernel-2.6.32-11-pve: 2.6.32-66
pve-kernel-2.6.35-2-pve: 2.6.35-13
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.3-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.8-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.7-2
pve-cluster: 1.0-26
qemu-server: 2.0-39
pve-firmware: 1.0-15
libpve-common-perl: 1.0-27
libpve-access-control: 1.0-21
libpve-storage-perl: 2.0-18
vncterm: 1.0-2
vzctl: 3.0.30-2pve5
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.0-9
ksm-control-daemon: 1.1-1

The VM does not start from the web interface. This is what I get when I try to start the VM from the commandline:

Code: 
root@barabas:~# /usr/bin/kvm -id 103 -chardev socket,id=monitor,path=/var/run/qemu-server/103.mon,server,nowait -mon chardev=monitor,mode=readline -vnc unix:/var/run/qemu-server/103.vnc,x509,password -pidfile /var/run/qemu-server/103.pid -daemonize -usbdevice tablet -device pci-assign,host=06:00.0,id=hostpci0 -name Streamer -smp sockets=2,cores=2 -nodefaults -boot menu=on -vga cirrus -k fr-be -drive if=none,id=drive-ide2,media=cdrom,aio=native -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -drive file=/var/lib/vz/images/103/vm-103-disk-1.qcow2,if=none,id=drive-ide0,aio=native -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100 -m 4096 -netdev type=tap,id=net0,ifname=tap103i0,script=/var/lib/qemu-server/pve-bridge -device e1000,mac=62:F9:C4:84:6B:6F,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 -netdev type=tap,id=net1,ifname=tap103i1,script=/var/lib/qemu-server/pve-bridge -device e1000,mac=CA:E8:4F:A6:59:C3,netdev=net1,bus=pci.0,addr=0x13,id=net1,bootindex=301 -cpuunits 1000
Failed to assign device "hostpci0" : Invalid argument
kvm: -device pci-assign,host=06:00.0,id=hostpci0: Device 'pci-assign' could not be initialized

Googling on "invalid argument" didn't bring me much luck. The following appears in my dmesg:

Code: 
pci 0000:06:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
pci 0000:06:00.0: restoring config space at offset 0x1 (was 0x100400, writing 0x100003)
assign device 6:0.0 failed
pci 0000:06:00.0: PCI INT A disabled

So it seems assigning the PCI card doesn't seem to work. Hints I have found are:

A lot of devices (including my USB controllers) are sharing the same IRQ:

Code: 
root@barabas:~# lspci -vv | grep IRQ
        Interrupt: pin A routed to IRQ 56
        Interrupt: pin A routed to IRQ 19
        Interrupt: pin A routed to IRQ 18
        Interrupt: pin B routed to IRQ 17
        Interrupt: pin A routed to IRQ 18
        Interrupt: pin B routed to IRQ 17
        Interrupt: pin C routed to IRQ 18
        Interrupt: pin A routed to IRQ 18
        Interrupt: pin B routed to IRQ 17
        Interrupt: pin A routed to IRQ 60
        Interrupt: pin B routed to IRQ 59
        Interrupt: pin A routed to IRQ 57
        Interrupt: pin A routed to IRQ 58
        Interrupt: pin A routed to IRQ 20
        Interrupt: pin B routed to IRQ 21
        Interrupt: pin A routed to IRQ 17

From dmesg :
Code: 
AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40
AMD-Vi: Initialized for Passthrough Mode
AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40

I also found this :
Code: 
Your BIOS doesn't leave a aperture memory hole
Please enable the IOMMU option in the BIOS setup
This costs you 64 MB of RAM
Mapping aperture over 65536 KB of RAM @ 20000000

Which I found strange. It is enabled in the BIOS and I know it works since I haven't changed it since esXi.
 
If I try pve-qemu-kvm_1.0-5_amd64.deb like max suggests I get this error:
Code: 
Failed to assign device "hostpci0" : Invalid argument
**
ERROR:qom/object.c:375:object_delete: assertion failed: (obj->ref == 0)

EDIT: BTW, I boot the normal .32 kernel, the .35 was also one I installed during my efforts, but don't use anymore
 
Last edited:
Mathy said:
If I try pve-qemu-kvm_1.0-5_amd64.deb like max suggests I get this error:
Code: 
Failed to assign device "hostpci0" : Invalid argument
**
ERROR:qom/object.c:375:object_delete: assertion failed: (obj->ref == 0)

EDIT: BTW, I boot the normal .32 kernel, the .35 was also one I installed during my efforts, but don't use anymore
Click to expand...

Hello,

Your ASRock Mainboard BIOS has the Option AMD IOMMU (like Intel: Vt-d) on?
Then try all my four steps at the same time with AMD and your Hardware Options - "It's worth a try"

maxprox
 
Hey maxprox,

The BIOS setting is on. All your 4 steps are the same, I have the hostpci0 statement, the boot line adaptations and the modprobe file in place. I tried with the 1.0-5, it didn't work. Only thing I can still think of is that I pulled it of a mirror server since I can't find it on the official and perhaps it's a different version. Can you perhaps upload your .deb somewhere so I can try that?

Thx,
Mathy
 
I'm a fan too, that's why I want it to work ;-) If I don't get a reply here from the dev's maybe I'll try an OpenVZ container and just passing through the device node, it's just that I have this VM already set up completely, and I don't want to the conversion work if it maybe is as simple as some config parameter.
 
Mathy,
Here is my pveversion -v:
Code: 
pve-manager: 2.1-1 (pve-manager/2.1/f9b0f63a)
running kernel: 2.6.32-11-pve
proxmox-ve-2.6.32: 2.0-66
pve-kernel-2.6.32-11-pve: 2.6.32-66
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.3-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.8-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.7-2
pve-cluster: 1.0-26
qemu-server: 2.0-39
pve-firmware: 1.0-15
libpve-common-perl: 1.0-27
libpve-access-control: 1.0-21
libpve-storage-perl: 2.0-18
vncterm: 1.0-2
vzctl: 3.0.30-2pve5
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.0-9
ksm-control-daemon: 1.1-1
Just for testing, have you tried passthrough on the machine with a different PCIe device, like an extra NIC? Or, if you have extra servers laying around, maybe try that TV tuner card in a different proxmox server (with a different type of motherboard/processor). Could be worth seeing if you can get anything to pass through, to narrow the problem down to the tv tuner card.
 
I did try it before with a PCI card and failed but apparently the PCI card was faulty. Haven't tried anything else but that is a good tip, I'll see if I can find something.

Our versions are exactly the same BTW.
 
You must log in or register to reply here.
Top Bottom