[SOLVED] PCIe Passthrough error

[EarlBacid]

Hi there,

I am trying to implement proxmox on a PCEngines APU Board. Proxmox itself is running fine, but I also want to passthrough a PCIe Wifi Card (AR928X) to a VM running pfsense.

I think I did everything mentioned in the Pci_passthrough Wiki, but when I start the virtual machine, I get this error message: "TASK ERROR: Cannot open iommu_group: No such file or directory" and the virtual machine stops.

root@proxmoxapu:~# pveversion -v
proxmox-ve-2.6.32: 3.3-138 (running kernel: 3.10.0-4-pve)pve-manager: 3.3-2 (running version: 3.3-2/995e687e)
pve-kernel-3.10.0-4-pve: 3.10.0-17
pve-kernel-2.6.32-33-pve: 2.6.32-138
pve-kernel-2.6.32-29-pve: 2.6.32-126
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-1
pve-cluster: 3.0-15
qemu-server: 3.1-35
pve-firmware: 1.1-3
libpve-common-perl: 3.0-19
libpve-access-control: 3.0-15
libpve-storage-perl: 3.0-23
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.1-9
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1

root@proxmoxapu:/etc/pve/qemu-server# cat 100.confbootdisk: ide0
cores: 2
cpu: qemu64
ide0: local:100/vm-100-disk-1.raw,format=raw,backup=no,size=8G
ide2: local:iso/pfSense-LiveCD-2.1.5-RELEASE-amd64-20140825-0744.iso,media=cdrom
memory: 768
name: frapf01
net0: rtl8139=A6:46:03:54:D7:76,bridge=vmbr1
net1: rtl8139=DA:81:33:2A:B9:65,bridge=vmbr2
ostype: other
smbios1: uuid=e2e7b8f1-57df-4042-b9da-2dc131bbd8ef
sockets: 1
machine: q35
hostpci0: 04:00.0,pcie=1,driver=vfio

Does anyone has any idea if I am doing something wrong or what I could try to get this working?
I really love the idea of running a hypervisor on my router hardware with a virtual router and AccessPoint (pfSense) as well as some other virtual machines providing services to my local network

 

[EarlBacid]

some additional information:

if I remove the hostpci line from the vm config file, I get the following error

[COLOR=#000000][FONT=tahoma]Could not access KVM kernel module: No such file or directory[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]failed to initialize KVM: No such file or directory[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]TASK ERROR: start failed: command '/usr/bin/kvm -id 100 -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/100.vnc,x509,password -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=e2e7b8f1-57df-4042-b9da-2dc131bbd8ef' -name frapf01 -smp 'sockets=1,cores=2' -nodefaults -boot 'menu=on' -vga cirrus -cpu qemu64,+x2apic -k en-us -m 768 -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:ece5833bd93f' -drive 'file=/var/lib/vz/template/iso/pfSense-LiveCD-2.1.5-RELEASE-amd64-20140825-0744.iso,if=none,id=drive-ide2,media=cdrom,aio=native' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/var/lib/vz/images/100/vm-100-disk-1.raw,if=none,id=drive-ide0,format=raw,aio=native,cache=none' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'rtl8139,mac=A6:46:03:54:D7:76,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -netdev 'type=tap,id=net1,ifname=tap100i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'rtl8139,mac=DA:81:33:2A:B9:65,netdev=net1,bus=pci.0,addr=0x13,id=net1,bootindex=301' -machine 'type=q35'' failed: exit code 1[/FONT][/COLOR]

this happens as long as I am using the 3.10.0-4-pve kernel. If I reboot my host using the 2.6.32-33-pve kernel I can start my virtual maschine without any problems.
Do I have to change anything else if I want to use the 3.10 Kernel?

 

[golduck]

are iommu active? try to delete the unsafe interrupt file on modprobe.d folder
use pc-q35-2.0 instead q35 on machine
the cpu and chipset support vt-d?

 

[EarlBacid]

Hi golduck,

thanks for your ideas.

I already activated iommu by changing /etc/defaults/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on"

is there a way to check if the kernel is really booted with this option?

The CPU is a AMD G-T40E which supports VT (AMD-V) reagrding to the specs.

If I delete /etc/modprobe.de/iommu_unsafe_interrupts.conf I get the error message "unable to read tail (got 0 bytes) when I try to start the virtual machine.

using pc-q35-2.0 instead q35 on the machine does not changy anything

 

[golduck]

try to post the result of the command "dmesg | grep AMD-Vi"

for use the last kernel purge the old kernel with apt-get purge , then update-grub for update the boot menu

 

[EarlBacid]

with amd_iommu=on I dont get any result.

root@proxmoxapu:~# dmesg | grep AMD-Vi
root@proxmoxapu:~#

but with intel_iommu=on I get something:

root@proxmoxapu:~# dmesg | grep -e IOMMU
[    0.000000] Intel-IOMMU: enabled
root@proxmoxapu:~#

is it possible that this AMD CPU only supports Intels VT?
unfortunatelly even with intel iommu enabled, I still get the "Could not access KVM kernel module: No such file or directory" error message when I want to start my vm with 3.10 Kernel on the host system.

 

[golduck]

try to search kvm in dmesg

 

[EarlBacid]

root@proxmoxapu:~# dmesg | grep -e kvm
[   18.706588] kvm: Unknown parameter `allow_unsafe_assigned_interrupts'

 

[mijanek]

I don't think that you CPU and/or Mainboard supports AMD-Vi (IO Virtualisation commonly named IOMMU) So far I found, its only avaiable in Opterons and some Phenoms or A -Series.. This CPU seems tonly to support basic virtualisation.

P.S. Intel-IOMMU: enabled Doesn't mean nothing in your environment, as you don't have any loaded driver for it and it can't work with AMD.

 

[EarlBacid]

mijanek: thanks for that input!
Until now I wasn't aware that there is a difference between "AMD-V" (which is supportet by my CPU) and "AMD-Vi" which is not. So I can stop trying.

 

[mir]

It is not enough that your CPU supports AMD-Vi your motherboard has to support it too.