I am trying to follow the wiki to get pci passthrough working on my server. I currently have a dell R510 with an H310 in it flashed to stock LSI firmware I am trying to pass through to FreeNAS. The hardware I have should be able to do it but I am getting errors.
Here is what I have so far:
/etc/default/grub
dmesg | grep -e DMAR -e IOMMU (BEFORE reboot)
dmesg | grep ecap
I changes those as well as changing modules and rebooted. Here is what I have after reboot.
find /sys/kernel/iommu_groups/ -type l
I then edited .etc/pve/qemu-server/100.conf as per guide (i have tried both the pcie and pci methods. Current config:
When I try to start the machine it fails with the following dialogue (in both pcie and pci methods)
After trying to boot machine output of dmesg | grep -e DMAR -e IOMMU changes as follows:
I have changed the unsafe_interrupts with no change in result (even though it SHOULD be supported)
Has anyone managed to get this to work on a Dell R510/R610/R710?
Here is what I have so far:
/etc/default/grub
Code:
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
GRUB_CMDLINE_LINUX=""
# Disable os-prober, it might add menu entries for each guest
GRUB_DISABLE_OS_PROBER=true
# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"
# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console
# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480
# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true
# Disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY="true"
# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"dmesg | grep -e DMAR -e IOMMU (BEFORE reboot)
Code:
root@proxmox:~# dmesg | grep -e DMAR -e IOMMU
[ 0.000000] ACPI: DMAR 0x00000000BF3B3668 0001D0 (v01 DELL PE_SC3 00000001 DELL 00000001)
[ 0.073593] DMAR-IR: This system BIOS has enabled interrupt remapping
[ 0.979277] DMAR: Host address width 40
[ 0.979280] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[ 0.979297] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c90780106f0462 ecap f020fe
[ 0.979299] DMAR: RMRR base: 0x000000bf4c8000 end: 0x000000bf4dffff
[ 0.979300] DMAR: RMRR base: 0x000000bf4b1000 end: 0x000000bf4bffff
[ 0.979301] DMAR: RMRR base: 0x000000bf4a1000 end: 0x000000bf4a1fff
[ 0.979302] DMAR: RMRR base: 0x000000bf4a3000 end: 0x000000bf4a3fff
[ 0.979303] DMAR: RMRR base: 0x000000bf4a5000 end: 0x000000bf4a5fff
[ 0.979303] DMAR: RMRR base: 0x000000bf4a7000 end: 0x000000bf4a7fff
[ 0.979304] DMAR: RMRR base: 0x000000bf4a9000 end: 0x000000bf4a9fff
[ 0.979305] DMAR: RMRR base: 0x000000bf4c0000 end: 0x000000bf4c0fff
[ 0.979306] DMAR: RMRR base: 0x000000bf4c2000 end: 0x000000bf4c2fff
[ 0.979307] DMAR: ATSR flags: 0x0dmesg | grep ecap
Code:
root@proxmox:~# dmesg | grep ecap
[ 0.979297] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c90780106f0462 ecap f020feI changes those as well as changing modules and rebooted. Here is what I have after reboot.
find /sys/kernel/iommu_groups/ -type l
Code:
00:00.0 Host bridge: Intel Corporation 5500 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)
00:09.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 13)
00:0a.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 10 (rev 13)
00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 13)
00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)
00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)
00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
00:1a.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.3 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10 Family) 4 port SATA IDE Controller #1
00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10 Family) 2 port SATA IDE Controller #2
01:00.0 Ethernet controller: Broadcom Limited NetXtreme II BCM5716 Gigabit Ethernet (rev 20)
01:00.1 Ethernet controller: Broadcom Limited NetXtreme II BCM5716 Gigabit Ethernet (rev 20)
04:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
05:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
06:03.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)
fe:00.0 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers (rev 02)
fe:00.1 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture System Address Decoder (rev 02)
fe:02.0 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 0 (rev 02)
fe:02.1 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 0 (rev 02)
fe:02.2 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 0 (rev 02)
fe:02.3 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 1 (rev 02)
fe:02.4 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 1 (rev 02)
fe:02.5 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 1 (rev 02)
fe:03.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Registers (rev 02)
fe:03.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Target Address Decoder (rev 02)
fe:03.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller RAS Registers (rev 02)
fe:03.4 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Test Registers (rev 02)
fe:04.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Control (rev 02)
fe:04.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Address (rev 02)
fe:04.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Rank (rev 02)
fe:04.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control (rev 02)
fe:05.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Control (rev 02)
fe:05.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Address (rev 02)
fe:05.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Rank (rev 02)
fe:05.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control (rev 02)
fe:06.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Control (rev 02)
fe:06.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Address (rev 02)
fe:06.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Rank (rev 02)
fe:06.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control (rev 02)
ff:00.0 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers (rev 02)
ff:00.1 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture System Address Decoder (rev 02)
ff:02.0 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 0 (rev 02)
ff:02.1 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 0 (rev 02)
ff:02.2 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 0 (rev 02)
ff:02.3 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 1 (rev 02)
ff:02.4 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 1 (rev 02)
ff:02.5 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 1 (rev 02)
ff:03.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Registers (rev 02)
ff:03.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Target Address Decoder (rev 02)
ff:03.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller RAS Registers (rev 02)
ff:03.4 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Test Registers (rev 02)
ff:04.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Control (rev 02)
ff:04.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Address (rev 02)
ff:04.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Rank (rev 02)
ff:04.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control (rev 02)
ff:05.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Control (rev 02)
ff:05.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Address (rev 02)
ff:05.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Rank (rev 02)
ff:05.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control (rev 02)
ff:06.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Control (rev 02)
ff:06.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Address (rev 02)
ff:06.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Rank (rev 02)
ff:06.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control (rev 02)I then edited .etc/pve/qemu-server/100.conf as per guide (i have tried both the pcie and pci methods. Current config:
Code:
bootdisk: sata0
cores: 4
cpu: Westmere
hostpci0: 05:00.0,pcie=1
ide2: none,media=cdrom
memory: 49152
name: FreeNAS
net0: e1000=6E:AD:BB:8A:CA:63,bridge=vmbr0
numa: 0
ostype: other
sata0: local-lvm:vm-100-disk-1,size=16G
scsihw: virtio-scsi-pci
smbios1: uuid=f8b9cab9-1d7a-4b78-af97-6b45fd95ed88
sockets: 1
machine: q35When I try to start the machine it fails with the following dialogue (in both pcie and pci methods)
Code:
kvm: -device vfio-pci,host=05:00.0,id=hostpci0,bus=pci.0,addr=0x10: vfio error: 0000:05:00.0: failed to setup container for group 13: failed to set iommu for container: Operation not permitted
start failed: command '/usr/bin/kvm -id 100 -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=f8b9cab9-1d7a-4b78-af97-6b45fd95ed88' -name FreeNAS -smp '4,sockets=1,cores=4,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga std -vnc unix:/var/run/qemu-server/100.vnc,x509,password -cpu 'Westmere,+kvm_pv_unhalt,+kvm_pv_eoi,enforce,vendor=GenuineIntel' -m 49152 -k en-us -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'vfio-pci,host=05:00.0,id=hostpci0,bus=pci.0,addr=0x10' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:34faf2b941c0' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'ahci,id=ahci0,multifunction=on,bus=pci.0,addr=0x7' -drive 'file=/dev/pve/vm-100-disk-1,if=none,id=drive-sata0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'ide-drive,bus=ahci0.0,drive=drive-sata0,id=sata0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=6E:AD:BB:8A:CA:63,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'' failed: exit code 1After trying to boot machine output of dmesg | grep -e DMAR -e IOMMU changes as follows:
Code:
root@proxmox:~# dmesg | grep -e DMAR -e IOMMU
[ 0.000000] ACPI: DMAR 0x00000000BF3B3668 0001D0 (v01 DELL PE_SC3 00000001 DELL 00000001)
[ 0.000000] DMAR: IOMMU enabled
[ 0.074191] DMAR-IR: This system BIOS has enabled interrupt remapping
[ 0.984153] DMAR: Host address width 40
[ 0.984155] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[ 0.984174] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c90780106f0462 ecap f020fe
[ 0.984175] DMAR: RMRR base: 0x000000bf4c8000 end: 0x000000bf4dffff
[ 0.984177] DMAR: RMRR base: 0x000000bf4b1000 end: 0x000000bf4bffff
[ 0.984178] DMAR: RMRR base: 0x000000bf4a1000 end: 0x000000bf4a1fff
[ 0.984179] DMAR: RMRR base: 0x000000bf4a3000 end: 0x000000bf4a3fff
[ 0.984180] DMAR: RMRR base: 0x000000bf4a5000 end: 0x000000bf4a5fff
[ 0.984180] DMAR: RMRR base: 0x000000bf4a7000 end: 0x000000bf4a7fff
[ 0.984181] DMAR: RMRR base: 0x000000bf4a9000 end: 0x000000bf4a9fff
[ 0.984182] DMAR: RMRR base: 0x000000bf4c0000 end: 0x000000bf4c0fff
[ 0.984183] DMAR: RMRR base: 0x000000bf4c2000 end: 0x000000bf4c2fff
[ 0.984184] DMAR: ATSR flags: 0x0
[ 0.987576] DMAR: dmar0: Using Queued invalidation
[ 0.987602] DMAR: Setting RMRR:
[ 0.987746] DMAR: Setting identity map for device 0000:00:1d.7 [0xbf4c2000 - 0xbf4c2fff]
[ 0.987885] DMAR: Setting identity map for device 0000:00:1a.7 [0xbf4c0000 - 0xbf4c0fff]
[ 0.988027] DMAR: Setting identity map for device 0000:00:1d.2 [0xbf4a9000 - 0xbf4a9fff]
[ 0.988196] DMAR: Setting identity map for device 0000:00:1d.1 [0xbf4a7000 - 0xbf4a7fff]
[ 0.988387] DMAR: Setting identity map for device 0000:00:1d.0 [0xbf4a5000 - 0xbf4a5fff]
[ 0.988550] DMAR: Setting identity map for device 0000:00:1a.1 [0xbf4a3000 - 0xbf4a3fff]
[ 0.988704] DMAR: Setting identity map for device 0000:00:1a.0 [0xbf4a1000 - 0xbf4a1fff]
[ 0.988722] DMAR: Setting identity map for device 0000:00:1a.0 [0xbf4b1000 - 0xbf4bffff]
[ 0.988724] DMAR: Setting identity map for device 0000:00:1a.1 [0xbf4b1000 - 0xbf4bffff]
[ 0.988726] DMAR: Setting identity map for device 0000:00:1d.0 [0xbf4b1000 - 0xbf4bffff]
[ 0.988727] DMAR: Setting identity map for device 0000:00:1d.1 [0xbf4b1000 - 0xbf4bffff]
[ 0.988729] DMAR: Setting identity map for device 0000:00:1d.2 [0xbf4b1000 - 0xbf4bffff]
[ 0.988731] DMAR: Setting identity map for device 0000:00:1a.7 [0xbf4c8000 - 0xbf4dffff]
[ 0.988733] DMAR: Setting identity map for device 0000:00:1d.7 [0xbf4c8000 - 0xbf4dffff]
[ 0.988736] DMAR: Prepare 0-16MiB unity mapping for LPC
[ 0.988891] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[ 0.989066] DMAR: Intel(R) Virtualization Technology for Directed I/O
[ 81.733525] vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[ 118.411870] vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platformI have changed the unsafe_interrupts with no change in result (even though it SHOULD be supported)
Has anyone managed to get this to work on a Dell R510/R610/R710?