PCIe HBA Passthrough not working

J

jackydany

Member
Jul 20, 2016
26
1
23
44
Hi,

i have a dell T620 with a H710 Raid controller flashed to IT mode.
On Baremetal Freenas it works, SAS disks are available in freenas.

As i need freenas as VM in proxmox (please, no discussion :D ) i want to passthrough the HBA to the VM.

But there is still something not working. I cant guess what it is so i am hoping for your support.

Code: 
cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Code: 
dmesg | grep -e DMAR -e IOMMU
[    0.025428] ACPI: DMAR 0x00000000CD3346F4 000138 (v01 DELL   PE_SC3   00000001 DELL 00000001)
[    0.299292] DMAR: IOMMU enabled
[    0.539448] DMAR: Host address width 46
[    0.539449] DMAR: DRHD base: 0x000000d5000000 flags: 0x0
[    0.539455] DMAR: dmar0: reg_base_addr d5000000 ver 1:0 cap d2078c106f0462 ecap f020fe
[    0.539456] DMAR: DRHD base: 0x000000df900000 flags: 0x1
[    0.539459] DMAR: dmar1: reg_base_addr df900000 ver 1:0 cap d2078c106f0462 ecap f020fe
[    0.539460] DMAR: RMRR base: 0x000000cf458000 end: 0x000000cf46ffff
[    0.539461] DMAR: RMRR base: 0x000000cf450000 end: 0x000000cf450fff
[    0.539462] DMAR: RMRR base: 0x000000cf452000 end: 0x000000cf452fff
[    0.539463] DMAR: ATSR flags: 0x0
[    0.539465] DMAR-IR: IOAPIC id 2 under DRHD base  0xd5000000 IOMMU 0
[    0.539466] DMAR-IR: IOAPIC id 0 under DRHD base  0xdf900000 IOMMU 1
[    0.539467] DMAR-IR: IOAPIC id 1 under DRHD base  0xdf900000 IOMMU 1
[    0.539468] DMAR-IR: HPET id 0 under DRHD base 0xdf900000
[    0.539469] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.539469] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.539970] DMAR-IR: Enabled IRQ remapping in xapic mode
[    1.853879] DMAR: dmar0: Using Queued invalidation
[    1.853886] DMAR: dmar1: Using Queued invalidation
[    1.885283] DMAR: Intel(R) Virtualization Technology for Directed I/O

Code: 
find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/17/devices/0000:07:00.0
/sys/kernel/iommu_groups/45/devices/0000:7f:11.0
/sys/kernel/iommu_groups/35/devices/0000:41:00.0
/sys/kernel/iommu_groups/35/devices/0000:41:00.1
/sys/kernel/iommu_groups/7/devices/0000:00:1a.0
/sys/kernel/iommu_groups/25/devices/0000:3f:0d.1
/sys/kernel/iommu_groups/25/devices/0000:3f:0d.6
/sys/kernel/iommu_groups/25/devices/0000:3f:0d.2
/sys/kernel/iommu_groups/25/devices/0000:3f:0d.0
/sys/kernel/iommu_groups/15/devices/0000:06:00.0
/sys/kernel/iommu_groups/43/devices/0000:7f:0f.6
/sys/kernel/iommu_groups/43/devices/0000:7f:0f.4
/sys/kernel/iommu_groups/43/devices/0000:7f:0f.2
/sys/kernel/iommu_groups/43/devices/0000:7f:0f.0
/sys/kernel/iommu_groups/43/devices/0000:7f:0f.5
/sys/kernel/iommu_groups/43/devices/0000:7f:0f.3
/sys/kernel/iommu_groups/43/devices/0000:7f:0f.1
/sys/kernel/iommu_groups/33/devices/0000:40:03.0
/sys/kernel/iommu_groups/5/devices/0000:00:11.0
/sys/kernel/iommu_groups/23/devices/0000:3f:0b.0
/sys/kernel/iommu_groups/23/devices/0000:3f:0b.3
/sys/kernel/iommu_groups/13/devices/0000:00:1f.2
/sys/kernel/iommu_groups/13/devices/0000:00:1f.0
/sys/kernel/iommu_groups/41/devices/0000:7f:0d.0
/sys/kernel/iommu_groups/41/devices/0000:7f:0d.1
/sys/kernel/iommu_groups/41/devices/0000:7f:0d.6
/sys/kernel/iommu_groups/41/devices/0000:7f:0d.2
/sys/kernel/iommu_groups/31/devices/0000:40:01.0
/sys/kernel/iommu_groups/3/devices/0000:00:03.0
/sys/kernel/iommu_groups/21/devices/0000:3f:09.0
/sys/kernel/iommu_groups/11/devices/0000:00:1d.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/38/devices/0000:7f:0a.0
/sys/kernel/iommu_groups/38/devices/0000:7f:0a.3
/sys/kernel/iommu_groups/38/devices/0000:7f:0a.1
/sys/kernel/iommu_groups/38/devices/0000:7f:0a.2
/sys/kernel/iommu_groups/28/devices/0000:3f:10.0
/sys/kernel/iommu_groups/28/devices/0000:3f:10.7
/sys/kernel/iommu_groups/28/devices/0000:3f:10.5
/sys/kernel/iommu_groups/28/devices/0000:3f:10.3
/sys/kernel/iommu_groups/28/devices/0000:3f:10.1
/sys/kernel/iommu_groups/28/devices/0000:3f:10.6
/sys/kernel/iommu_groups/28/devices/0000:3f:10.4
/sys/kernel/iommu_groups/28/devices/0000:3f:10.2
/sys/kernel/iommu_groups/18/devices/0000:09:00.0
/sys/kernel/iommu_groups/18/devices/0000:08:00.0
/sys/kernel/iommu_groups/18/devices/0000:0a:00.0
/sys/kernel/iommu_groups/46/devices/0000:7f:13.4
/sys/kernel/iommu_groups/46/devices/0000:7f:13.0
/sys/kernel/iommu_groups/46/devices/0000:7f:13.5
/sys/kernel/iommu_groups/46/devices/0000:7f:13.1
/sys/kernel/iommu_groups/46/devices/0000:7f:13.6
/sys/kernel/iommu_groups/36/devices/0000:7f:08.0
/sys/kernel/iommu_groups/8/devices/0000:00:1c.0
/sys/kernel/iommu_groups/26/devices/0000:3f:0e.1
/sys/kernel/iommu_groups/26/devices/0000:3f:0e.0
/sys/kernel/iommu_groups/16/devices/0000:06:00.1
/sys/kernel/iommu_groups/44/devices/0000:7f:10.2
/sys/kernel/iommu_groups/44/devices/0000:7f:10.0
/sys/kernel/iommu_groups/44/devices/0000:7f:10.7
/sys/kernel/iommu_groups/44/devices/0000:7f:10.5
/sys/kernel/iommu_groups/44/devices/0000:7f:10.3
/sys/kernel/iommu_groups/44/devices/0000:7f:10.1
/sys/kernel/iommu_groups/44/devices/0000:7f:10.6
/sys/kernel/iommu_groups/44/devices/0000:7f:10.4
/sys/kernel/iommu_groups/34/devices/0000:40:05.0
/sys/kernel/iommu_groups/34/devices/0000:40:05.2
/sys/kernel/iommu_groups/6/devices/0000:00:16.0
/sys/kernel/iommu_groups/6/devices/0000:00:16.1
/sys/kernel/iommu_groups/24/devices/0000:3f:0c.2
/sys/kernel/iommu_groups/24/devices/0000:3f:0c.0
/sys/kernel/iommu_groups/24/devices/0000:3f:0c.7
/sys/kernel/iommu_groups/24/devices/0000:3f:0c.1
/sys/kernel/iommu_groups/24/devices/0000:3f:0c.6
/sys/kernel/iommu_groups/14/devices/0000:02:00.0
/sys/kernel/iommu_groups/42/devices/0000:7f:0e.1
/sys/kernel/iommu_groups/42/devices/0000:7f:0e.0
/sys/kernel/iommu_groups/32/devices/0000:40:02.0
/sys/kernel/iommu_groups/4/devices/0000:00:05.2
/sys/kernel/iommu_groups/4/devices/0000:00:05.0
/sys/kernel/iommu_groups/22/devices/0000:3f:0a.3
/sys/kernel/iommu_groups/22/devices/0000:3f:0a.1
/sys/kernel/iommu_groups/22/devices/0000:3f:0a.2
/sys/kernel/iommu_groups/22/devices/0000:3f:0a.0
/sys/kernel/iommu_groups/12/devices/0000:00:1e.0
/sys/kernel/iommu_groups/40/devices/0000:7f:0c.6
/sys/kernel/iommu_groups/40/devices/0000:7f:0c.2
/sys/kernel/iommu_groups/40/devices/0000:7f:0c.0
/sys/kernel/iommu_groups/40/devices/0000:7f:0c.7
/sys/kernel/iommu_groups/40/devices/0000:7f:0c.1
/sys/kernel/iommu_groups/30/devices/0000:3f:13.0
/sys/kernel/iommu_groups/30/devices/0000:3f:13.5
/sys/kernel/iommu_groups/30/devices/0000:3f:13.1
/sys/kernel/iommu_groups/30/devices/0000:3f:13.6
/sys/kernel/iommu_groups/30/devices/0000:3f:13.4
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/20/devices/0000:3f:08.0
/sys/kernel/iommu_groups/10/devices/0000:00:1c.7
/sys/kernel/iommu_groups/39/devices/0000:7f:0b.3
/sys/kernel/iommu_groups/39/devices/0000:7f:0b.0
/sys/kernel/iommu_groups/29/devices/0000:3f:11.0
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/19/devices/0000:08:01.0
/sys/kernel/iommu_groups/37/devices/0000:7f:09.0
/sys/kernel/iommu_groups/9/devices/0000:00:1c.4
/sys/kernel/iommu_groups/27/devices/0000:3f:0f.6
/sys/kernel/iommu_groups/27/devices/0000:3f:0f.4
/sys/kernel/iommu_groups/27/devices/0000:3f:0f.2
/sys/kernel/iommu_groups/27/devices/0000:3f:0f.0
/sys/kernel/iommu_groups/27/devices/0000:3f:0f.5
/sys/kernel/iommu_groups/27/devices/0000:3f:0f.3
/sys/kernel/iommu_groups/27/devices/0000:3f:0f.1

Code: 
kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
 
Code: 
lspci -nn
00:00.0 Host bridge [0600]: Intel Corporation Xeon E5/Core i7 DMI2 [8086:3c00] (rev 07)
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a [8086:3c02] (rev 07)
00:02.0 PCI bridge [0604]: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a [8086:3c04] (rev 07)
00:03.0 PCI bridge [0604]: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode [8086:3c08] (rev 07)
00:05.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management [8086:3c28] (rev 07)
00:05.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors [8086:3c2a] (rev 07)
00:11.0 PCI bridge [0604]: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port [8086:1d3e] (rev 05)
00:16.0 Communication controller [0780]: Intel Corporation C600/X79 series chipset MEI Controller #1 [8086:1d3a] (rev 05)
00:16.1 Communication controller [0780]: Intel Corporation C600/X79 series chipset MEI Controller #2 [8086:1d3b] (rev 05)
00:1a.0 USB controller [0c03]: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 [8086:1d2d] (rev 05)
00:1c.0 PCI bridge [0604]: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 [8086:1d10] (rev b5)
00:1c.4 PCI bridge [0604]: Intel Corporation C600/X79 series chipset PCI Express Root Port 5 [8086:1d18] (rev b5)
00:1c.7 PCI bridge [0604]: Intel Corporation C600/X79 series chipset PCI Express Root Port 8 [8086:1d1e] (rev b5)
00:1d.0 USB controller [0c03]: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 [8086:1d26] (rev 05)
00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev a5)
00:1f.0 ISA bridge [0601]: Intel Corporation C600/X79 series chipset LPC Controller [8086:1d41] (rev 05)
00:1f.2 SATA controller [0106]: Intel Corporation C600/X79 series chipset 6-Port SATA AHCI Controller [8086:1d02] (rev 05)
02:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 [1000:0087] (rev 05)
06:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
06:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
07:00.0 PCI bridge [0604]: Renesas Technology Corp. SH7757 PCIe Switch [PS] [1912:0013]
08:00.0 PCI bridge [0604]: Renesas Technology Corp. SH7757 PCIe Switch [PS] [1912:0013]
08:01.0 PCI bridge [0604]: Renesas Technology Corp. SH7757 PCIe Switch [PS] [1912:0013]
09:00.0 PCI bridge [0604]: Renesas Technology Corp. SH7757 PCIe-PCI Bridge [PPB] [1912:0012]
0a:00.0 VGA compatible controller [0300]: Matrox Electronics Systems Ltd. G200eR2 [102b:0534]
3f:08.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 QPI Link 0 [8086:3c80] (rev 07)
3f:09.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 QPI Link 1 [8086:3c90] (rev 07)
3f:0a.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Power Control Unit 0 [8086:3cc0] (rev 07)
3f:0a.1 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Power Control Unit 1 [8086:3cc1] (rev 07)
3f:0a.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Power Control Unit 2 [8086:3cc2] (rev 07)
3f:0a.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Power Control Unit 3 [8086:3cd0] (rev 07)
3f:0b.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Interrupt Control Registers [8086:3ce0] (rev 07)
3f:0b.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Semaphore and Scratchpad Configuration Registers [8086:3ce3] (rev 07)
3f:0c.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
3f:0c.1 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
3f:0c.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
3f:0c.6 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 0 [8086:3cf4] (rev 07)
3f:0c.7 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 System Address Decoder [8086:3cf6] (rev 07)
3f:0d.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
3f:0d.1 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
3f:0d.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
3f:0d.6 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 1 [8086:3cf5] (rev 07)
3f:0e.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Processor Home Agent [8086:3ca0] (rev 07)
3f:0e.1 Performance counters [1101]: Intel Corporation Xeon E5/Core i7 Processor Home Agent Performance Monitoring [8086:3c46] (rev 07)
3f:0f.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers [8086:3ca8] (rev 07)
3f:0f.1 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller RAS Registers [8086:3c71] (rev 07)
3f:0f.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 0 [8086:3caa] (rev 07)
3f:0f.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 1 [8086:3cab] (rev 07)
3f:0f.4 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 2 [8086:3cac] (rev 07)
3f:0f.5 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 3 [8086:3cad] (rev 07)
3f:0f.6 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 4 [8086:3cae] (rev 07)
3f:10.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0 [8086:3cb0] (rev 07)
3f:10.1 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 1 [8086:3cb1] (rev 07)
3f:10.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 0 [8086:3cb2] (rev 07)
3f:10.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 1 [8086:3cb3] (rev 07)
3f:10.4 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 2 [8086:3cb4] (rev 07)
3f:10.5 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 3 [8086:3cb5] (rev 07)
3f:10.6 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 2 [8086:3cb6] (rev 07)
3f:10.7 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 3 [8086:3cb7] (rev 07)
3f:11.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 DDRIO [8086:3cb8] (rev 07)
3f:13.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 R2PCIe [8086:3ce4] (rev 07)
3f:13.1 Performance counters [1101]: Intel Corporation Xeon E5/Core i7 Ring to PCI Express Performance Monitor [8086:3c43] (rev 07)
3f:13.4 Performance counters [1101]: Intel Corporation Xeon E5/Core i7 QuickPath Interconnect Agent Ring Registers [8086:3ce6] (rev 07)
3f:13.5 Performance counters [1101]: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 0 Performance Monitor [8086:3c44] (rev 07)
3f:13.6 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 1 Performance Monitor [8086:3c45] (rev 07)
 
Code: 
40:01.0 PCI bridge [0604]: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a [8086:3c02] (rev 07)
40:02.0 PCI bridge [0604]: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a [8086:3c04] (rev 07)
40:03.0 PCI bridge [0604]: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode [8086:3c08] (rev 07)
40:05.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management [8086:3c28] (rev 07)
40:05.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors [8086:3c2a] (rev 07)
41:00.0 Ethernet controller [0200]: QLogic Corp. cLOM8214 1/10GbE Controller [1077:8020] (rev 54)
41:00.1 Ethernet controller [0200]: QLogic Corp. cLOM8214 1/10GbE Controller [1077:8020] (rev 54)
7f:08.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 QPI Link 0 [8086:3c80] (rev 07)
7f:09.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 QPI Link 1 [8086:3c90] (rev 07)
7f:0a.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Power Control Unit 0 [8086:3cc0] (rev 07)
7f:0a.1 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Power Control Unit 1 [8086:3cc1] (rev 07)
7f:0a.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Power Control Unit 2 [8086:3cc2] (rev 07)
7f:0a.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Power Control Unit 3 [8086:3cd0] (rev 07)
7f:0b.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Interrupt Control Registers [8086:3ce0] (rev 07)
7f:0b.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Semaphore and Scratchpad Configuration Registers [8086:3ce3] (rev 07)
7f:0c.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
7f:0c.1 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
7f:0c.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
7f:0c.6 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 0 [8086:3cf4] (rev 07)
7f:0c.7 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 System Address Decoder [8086:3cf6] (rev 07)
7f:0d.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
7f:0d.1 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
7f:0d.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Unicast Register 0 [8086:3ce8] (rev 07)
7f:0d.6 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 1 [8086:3cf5] (rev 07)
7f:0e.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Processor Home Agent [8086:3ca0] (rev 07)
7f:0e.1 Performance counters [1101]: Intel Corporation Xeon E5/Core i7 Processor Home Agent Performance Monitoring [8086:3c46] (rev 07)
7f:0f.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers [8086:3ca8] (rev 07)
7f:0f.1 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller RAS Registers [8086:3c71] (rev 07)
7f:0f.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 0 [8086:3caa] (rev 07)
7f:0f.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 1 [8086:3cab] (rev 07)
7f:0f.4 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 2 [8086:3cac] (rev 07)
7f:0f.5 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 3 [8086:3cad] (rev 07)
7f:0f.6 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 4 [8086:3cae] (rev 07)
7f:10.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0 [8086:3cb0] (rev 07)
7f:10.1 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 1 [8086:3cb1] (rev 07)
7f:10.2 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 0 [8086:3cb2] (rev 07)
7f:10.3 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 1 [8086:3cb3] (rev 07)
7f:10.4 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 2 [8086:3cb4] (rev 07)
7f:10.5 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 3 [8086:3cb5] (rev 07)
7f:10.6 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 2 [8086:3cb6] (rev 07)
7f:10.7 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 3 [8086:3cb7] (rev 07)
7f:11.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 DDRIO [8086:3cb8] (rev 07)
7f:13.0 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 R2PCIe [8086:3ce4] (rev 07)
7f:13.1 Performance counters [1101]: Intel Corporation Xeon E5/Core i7 Ring to PCI Express Performance Monitor [8086:3c43] (rev 07)
7f:13.4 Performance counters [1101]: Intel Corporation Xeon E5/Core i7 QuickPath Interconnect Agent Ring Registers [8086:3ce6] (rev 07)
7f:13.5 Performance counters [1101]: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 0 Performance Monitor [8086:3c44] (rev 07)
7f:13.6 System peripheral [0880]: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 1 Performance Monitor [8086:3c45] (rev 07)

I did
echo "blacklist mpt3sas" >> /etc/modprobe.d/blacklist.conf
and all the stuff mentioned in the wiki for PT. I used PT several times before, so i am wondering where the problem is.
Its my first time setting up Proxmox on a Dell

VT-d enabled of course.

02:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 [1000:0087] (rev 05)
is the one we are talking about.
it doesnt show up in the VM

Only thing i see is
dmesg | grep 'remapping'
[ 0.539970] DMAR-IR: Enabled IRQ remapping in xapic mode
[ 0.539970] x2apic: IRQ remapping doesn't support X2APIC mode
but also thats ok as in one posting mentioned as it uses xapic instead of x2apic.

VM is set up with seabios, q35, Host cpu, 24gb ram, 30gb Disk

Somebody any clue??

Thanks a lot in advance!

Stefan[/CODE]
 
Since i have the same card, thought i write my configs down for comparison.
My VM settings are also smbios,q35,host cpu, numa.

I think all these options where set for my gpu passthrough already, and i still have to check what is really necessary. But it works, so ...
"/etc/default/grub"
Code: 
...
GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt transparent_hugepage=always pcie_port_pm=off nofb vfio-pci.ids=1002:7340,1002:ab38 video=efifb:off"
...


Nothing added in "/etc/modprobe.d/vfio.conf".


The following also already set for my gpu passthrough.
"/etc/modprobe.d/iommu_unsafe_interrupts.conf"
Code: 
options vfio_iommu_type1 allow_unsafe_interrupts=1


Don't know anymore... But it works, so ....
"/etc/modprobe.d/kvm.conf"
Code: 
options kvm ignore_msrs=1


And here especially the one difference that i think may be easy to try out,
although why i added the *2* version, is hazy .. maybe i was confused bc the internet said these cards use the *2* version or i just wanted to be sure, since it couldn't hurt my purpose as the host shouldnt get any hba cards for its own use.
"/etc/modprobe.d/pve-blacklist.conf"
Code: 
...
# disable lsi hba
blacklist mpt2sas
blacklist mpt3sas

Finally the definition for the VM. The "rombar=0" disables (the showing of) the raid boot menu, which is absolutely not needed, and in fact made the whole setup not work for me when left on.
"/etc/pve/quemu-server/100.conf"
Code: 
...
hostpci0: 03:00,pcie=1,rombar=0
...

The result for me:
Code: 
### on the host
root@pve-1:~# lspci -nn | grep -i sas
03:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 [1000:0087] (rev 05)

### in ubuntu 20.04 vm
root@srv-1:~# lspci -nn | grep -i sas
01:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 [1000:0087] (rev 05)

Now one port has an extender attached, which connects to 16drives via breakout cable.
The other port has a breakout cable and ssds attached.

Works stable.


ps: Just to make sure ... you did a "update-initramfs -u -k all" after adding to /etc/modprobe.d/blacklist.conf?

(also wasn't sure why your file is named differently .. but the internet to the rescure: https://lists.proxmox.com/pipermail/pve-user/2017-August/168716.html .. and learned my way is discouraged)


edited: whole post rewrite, added the question
 
Last edited:
HI,

thanks for the reply

Drivers are vfio like it should be. I blacklisted the mpt3sas driver before. This was used previous to my blacklisting


find /sys|grep drivers.*02:00
/sys/bus/pci/drivers/vfio-pci/0000:02:00.0




@jw-it
Thanks, but as you have an AMD cpu and use unsafe interrupts, this is not exactly what my enviroment looks like.
also i can see no difference besides that INtel vs AMD
You did the vfio stuff in grub, i did it in vfio.conf like mentioned in the manual ( https://pve.proxmox.com/wiki/PCI(e)_Passthrough#_host_device_passthrough )
and yes, i did a
update-initramfs -u -k all
;) but good to mention it!

Thanks again for your help!
 
Last edited:
You must log in or register to reply here.
Top Bottom