PCI Passthrough Freeze Host

[Oliver R]

Hi!

I am currently using a Custom Little Server with H97N-WIFI Motherboard and an i5-4460. I got an Intel Wifi 7260 Mini Pcie and an Atheros AR9462 Mini Pcie.

The Problem is that I can't pass the Atheros Card through my KVM Machine with pfsense installed because it freezes the host after starting the VM. The Intel Card is working flawless with the passthrough but the Intel one doesn't support the Access Point option in pfsense.

In Proxmox and even when I dd my pfsense vm on an usb stick, the Atheros Card was recognised and works perfect. I don't understand why the Card crashes Proxmox while passing it through.

I used the docs from Proxmox itself to enable the passthrough and it works only for the Intel one.

Does someone got this kind of a problem? Did I forget something?

Spoiler: VM Config

agent: 1
bios: ovmf
bootdisk: scsi0
cores: 2
cpu: host
efidisk0: local:200/vm-200-disk-2.qcow2,size=128K
ide2: local-ISO:iso/pfSense-CE-2.4.0-BETA-amd64-20170607-0927.iso,media=cdrom
memory: 2048
name: pfsense
net0: virtio=40:8D:5C:B5:A1:45,bridge=vmbr0
net1: virtio=40:8D:5C:B5:A1:46,bridge=vmbr1
numa: 1
ostype: other
scsi0: local:200/vm-200-disk-1.qcow2,cache=writethrough,discard=on,size=20G
scsihw: virtio-scsi-pci
smbios1: uuid=c03b51c5-284b-42c3-872d-99200ce71498
sockets: 1
machine: q35
hostpci0: 03:00.0,pcie=1

Spoiler: Pci List [lspci -nnk]

00:00.0 Host bridge [0600]: Intel Corporation 4th Gen Core Processor DRAM Controller [8086:0c00] (rev 06)
Subsystem: Gigabyte Technology Co., Ltd 4th Gen Core Processor DRAM Controller [1458:5000]
Kernel driver in use: hsw_uncore
00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller [8086:0412] (rev 06)
Subsystem: Gigabyte Technology Co., Ltd Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller [1458:d000]
Kernel driver in use: i915
Kernel modules: i915
00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] (rev 06)
Subsystem: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:2010]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
00:14.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB xHCI Controller [8086:8cb1]
Subsystem: Gigabyte Technology Co., Ltd 9 Series Chipset Family USB xHCI Controller [1458:5007]
Kernel driver in use: xhci_hcd
00:16.0 Communication controller [0780]: Intel Corporation 9 Series Chipset Family ME Interface #1 [8086:8cba]
Subsystem: Gigabyte Technology Co., Ltd 9 Series Chipset Family ME Interface [1458:1c3a]
Kernel driver in use: mei_me
Kernel modules: mei_me
00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection I217-V [8086:153b]
Subsystem: Gigabyte Technology Co., Ltd Ethernet Connection I217-V [1458:e000]
Kernel driver in use: e1000e
Kernel modules: e1000e
00:1a.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 [8086:8cad]
Subsystem: Gigabyte Technology Co., Ltd 9 Series Chipset Family USB EHCI Controller [1458:5006]
Kernel driver in use: ehci-pci
00:1b.0 Audio device [0403]: Intel Corporation 9 Series Chipset Family HD Audio Controller [8086:8ca0]
Subsystem: Gigabyte Technology Co., Ltd 9 Series Chipset Family HD Audio Controller [1458:a182]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
00:1c.0 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 [8086:8c90] (rev d0)
Kernel driver in use: pcieport
Kernel modules: shpchp
00:1c.3 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 [8086:8c96] (rev d0)
Kernel driver in use: pcieport
Kernel modules: shpchp
00:1c.4 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 5 [8086:8c98] (rev d0)
Kernel driver in use: pcieport
Kernel modules: shpchp
00:1d.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 [8086:8ca6]
Subsystem: Gigabyte Technology Co., Ltd 9 Series Chipset Family USB EHCI Controller [1458:5006]
Kernel driver in use: ehci-pci
00:1f.0 ISA bridge [0601]: Intel Corporation 9 Series Chipset Family H97 Controller [8086:8cc6]
Subsystem: Gigabyte Technology Co., Ltd 9 Series Chipset Family H97 Controller [1458:5001]
Kernel driver in use: lpc_ich
Kernel modules: lpc_ich
00:1f.2 SATA controller [0106]: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] [8086:8c82]
Subsystem: Gigabyte Technology Co., Ltd 9 Series Chipset Family SATA Controller [AHCI Mode] [1458:b005]
Kernel driver in use: ahci
Kernel modules: ahci
00:1f.3 SMBus [0c05]: Intel Corporation 9 Series Chipset Family SMBus Controller [8086:8ca2]
Subsystem: Gigabyte Technology Co., Ltd 9 Series Chipset Family SMBus Controller [1458:5001]
Kernel driver in use: i801_smbus
Kernel modules: i2c_i801
02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 0c)
Subsystem: Gigabyte Technology Co., Ltd Onboard Ethernet [1458:e000]
Kernel driver in use: r8169
Kernel modules: r8169
03:00.0 Network controller [0280]: Qualcomm Atheros AR9462 Wireless Network Adapter [168c:0034] (rev 01)
Subsystem: Lite-On Communications Inc AR9462 Wireless Network Adapter [11ad:6611]
Kernel driver in use: vfio-pci
Kernel modules: ath9k

Spoiler: Proxmox Version (Using PVE Test Repo) [pveversion -v]

proxmox-ve: 5.0-21 (running kernel: 4.10.17-3-pve)
pve-manager: 5.0-32 (running version: 5.0-32/2560e073)
pve-kernel-4.10.17-2-pve: 4.10.17-20
pve-kernel-4.10.17-3-pve: 4.10.17-21
libpve-http-server-perl: 2.0-6
lvm2: 2.02.168-pve3
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-12
qemu-server: 5.0-15
pve-firmware: 2.0-2
libpve-common-perl: 5.0-18
libpve-guest-common-perl: 2.0-11
libpve-access-control: 5.0-6
libpve-storage-perl: 5.0-15
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-2
pve-docs: 5.0-9
pve-qemu-kvm: 2.9.1-1
pve-container: 2.0-16
pve-firewall: 3.0-3
pve-ha-manager: 2.0-2
ksm-control-daemon: 1.2-2
glusterfs-client: 3.8.8-1
lxc-pve: 2.1.0-1
lxcfs: 2.0.7-pve4
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.6.5.11-pve17~bpo90
ceph: 12.2.0-pve1

Spoiler: Grub Config

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
GRUB_CMDLINE_LINUX=""

# Disable os-prober, it might add menu entries for each guest
GRUB_DISABLE_OS_PROBER=true

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

Spoiler: VFIO Conf in mobprobe.d

options vfio-pci ids=168c:0034

 

[dcsapak]

how do your iommu groups look like?

find /sys/kernel/iommu_groups -type l | sort -t '/' -n -k 5

 

[Oliver R]

Hi dcsapak!

Here is the Output. It looks like the entry in the docs.

/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:02.0
/sys/kernel/iommu_groups/2/devices/0000:00:03.0
/sys/kernel/iommu_groups/3/devices/0000:00:14.0
/sys/kernel/iommu_groups/4/devices/0000:00:16.0
/sys/kernel/iommu_groups/5/devices/0000:00:19.0
/sys/kernel/iommu_groups/6/devices/0000:00:1a.0
/sys/kernel/iommu_groups/7/devices/0000:00:1b.0
/sys/kernel/iommu_groups/8/devices/0000:00:1c.0
/sys/kernel/iommu_groups/9/devices/0000:00:1c.3
/sys/kernel/iommu_groups/10/devices/0000:00:1c.4
/sys/kernel/iommu_groups/11/devices/0000:00:1d.0
/sys/kernel/iommu_groups/12/devices/0000:00:1f.0
/sys/kernel/iommu_groups/12/devices/0000:00:1f.2
/sys/kernel/iommu_groups/12/devices/0000:00:1f.3
/sys/kernel/iommu_groups/13/devices/0000:02:00.0
/sys/kernel/iommu_groups/14/devices/0000:03:00.0

 

[Oliver R]

It seams like I am not the only one who is lost

 

[dcsapak]

It seams like I am not the only one who is lost

yeah sorry, no clue ... maybe the Atheros card is not suitable for passthrough?
at least with vga passthrough it really depends on the card if it works smoothly or not, i guess it is the same with other pcie hardware