first pcie express disappear when passthrough another one

[bobzer]

Hi,

I got a pcie sas controller, (LSI 9211-8i SAS SA) plug in the first pcie
in another pcie i got a graphics card that i need to passthrough a VM.

so my problem is that both card works well but not together....

when i start the vm that got the graphics card the other card just disapear...
I really don't know what to do, even where to start investigation.
the only peace of information that i found is the console that show this :

[  969.449634] VFIO - User Level meta-driver version: 0.3
[  969.453604] vfio-pci 0000:02:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[  969.470012] vfio_pci: add [10de:1b80[ffff:ffff]] class 0x000000/00000000
[  969.470054] vfio_pci: add [10de:10f0[ffff:ffff]] class 0x000000/00000000
[  969.594135] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[  969.594186] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[  969.618208] mpt2sas_cm0: removing handle(0x000b), sas_addr(0x4433221103000000)
[  969.618235] mpt2sas_cm0: removing : enclosure logical id(0x500605b0013ca580), slot(0)
[  969.634134] sd 0:0:1:0: [sdb] Synchronizing SCSI cache
[  969.634187] sd 0:0:1:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[  969.658225] mpt2sas_cm0: removing handle(0x0009), sas_addr(0x4433221100000000)
[  969.658249] mpt2sas_cm0: removing : enclosure logical id(0x500605b0013ca580), slot(3)
[  969.678169] sd 0:0:2:0: [sdc] Synchronizing SCSI cache
[  969.678230] sd 0:0:2:0: [sdc] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[  969.702191] mpt2sas_cm0: removing handle(0x000a), sas_addr(0x4433221102000000)
[  969.702224] mpt2sas_cm0: removing : enclosure logical id(0x500605b0013ca580), slot(1)
[  969.722125] sd 0:0:3:0: [sdd] Synchronizing SCSI cache
[  969.722196] sd 0:0:3:0: [sdd] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[  969.766182] mpt2sas_cm0: removing handle(0x000c), sas_addr(0x4433221107000000)
[  969.766209] mpt2sas_cm0: removing : enclosure logical id(0x500605b0013ca580), slot(4)
[  969.766530] mpt2sas_cm0: sending message unit reset !!
[  969.768083] mpt2sas_cm0: message unit reset: SUCCESS
[  970.755992] device tap300i0 entered promiscuous mode
[  970.761675] vmbr0: port 2(tap300i0) entered blocking state
[  970.761697] vmbr0: port 2(tap300i0) entered disabled state
[  970.761803] vmbr0: port 2(tap300i0) entered blocking state
[  970.761822] vmbr0: port 2(tap300i0) entered forwarding state
[  971.426022] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003)
[  971.426192] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x900
[  976.158048] usb 1-1.2: reset full-speed USB device number 3 using ehci-pci
[  976.466036] usb 1-1.2: reset full-speed USB device number 3 using ehci-pci
[  981.394030] usb 1-1.2: reset full-speed USB device number 3 using ehci-pci
[  981.606022] usb 1-1.2: reset full-speed USB device number 3 using ehci-pci
[  981.838027] usb 1-1.2: reset full-speed USB device number 3 using ehci-pci
[  982.038062] usb 1-1.2: reset full-speed USB device number 3 using ehci-pci
[  986.487207] kvm [3449]: vcpu3, guest rIP: 0xfffff807909e62b6 ignored rdmsr: 0x611
[  986.487249] kvm [3449]: vcpu3, guest rIP: 0xfffff807909e62cc ignored rdmsr: 0x641
[  986.487275] kvm [3449]: vcpu3, guest rIP: 0xfffff807909e62e2 ignored rdmsr: 0x606
[  986.487300] kvm [3449]: vcpu3, guest rIP: 0xfffff807909e610e ignored rdmsr: 0x606
[  986.487326] kvm [3449]: vcpu3, guest rIP: 0xfffff807909e119c ignored rdmsr: 0x641
[  986.487351] kvm [3449]: vcpu3, guest rIP: 0xfffff807909e11e7 ignored rdmsr: 0x611
[  986.802039] usb 1-1.2: reset full-speed USB device number 3 using ehci-pci
[  986.998060] usb 1-1.2: reset full-speed USB device number 3 using ehci-pci
[  987.362040] usb 1-1.2: reset full-speed USB device number 3 using ehci-pci

the config file of the vm is :

agent: 1
balloon: 2048
bios: ovmf
bootdisk: virtio0
cores: 4
cpu: IvyBridge,hidden=1
efidisk0: data:vm-300-disk-1,size=4M
hostpci0: 02:00,pcie=1,x-vga=on
ide0: none,media=cdrom
machine: q35
memory: 8192
name: win-emma-gtx
net0: virtio=CE:E3:3A:68:AB:5F,bridge=vmbr0
numa: 0
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=60c6b50e-a94e-46b4-a7d5-740e03ffb35d
sockets: 1
usb0: host=1-1.2
vga: qxl
virtio0: ssd:vm-300-disk-1,cache=writeback,size=250G
virtio1: data:vm-300-disk-2,cache=writeback,size=500G

hope you can help me

thank you

 

[dcsapak]

can you post your iommu groups ?

#!/bin/bash
shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do
    n=${d#*/iommu_groups/*}; n=${n%%/*}
    printf 'IOMMU Group %s ' "$n"
    lspci -nns "${d##*/}"
done;

 

[bobzer]

thanks for you time
here is the results of your script :

IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller [8086:0150] (rev 09)
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)
IOMMU Group 1 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0155] (rev 09)
IOMMU Group 1 01:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
IOMMU Group 1 02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1080] [10de:1b80] (rev a1)
IOMMU Group 1 02:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
IOMMU Group 10 00:1c.5 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 6 [8086:1e1a] (rev c4)
IOMMU Group 11 00:1c.7 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 [8086:1e1e] (rev c4)
IOMMU Group 12 00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04)
IOMMU Group 13 00:1f.0 ISA bridge [0601]: Intel Corporation Z77 Express Chipset LPC Controller [8086:1e44] (rev 04)
IOMMU Group 13 00:1f.2 SATA controller [0106]: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] [8086:1e02] (rev 04)
IOMMU Group 13 00:1f.3 SMBus [0c05]: Intel Corporation 7 Series/C216 Chipset Family SMBus Controller [8086:1e22] (rev 04)
IOMMU Group 14 04:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]
IOMMU Group 15 05:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
IOMMU Group 16 06:01.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
IOMMU Group 17 06:04.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
IOMMU Group 18 06:05.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
IOMMU Group 19 06:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
IOMMU Group 2 00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller [8086:0162] (rev 09)
IOMMU Group 20 06:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
IOMMU Group 21 08:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11)
IOMMU Group 22 0b:00.0 PCI bridge [0604]: Integrated Technology Express, Inc. Device [1283:8892] (rev 30)
IOMMU Group 22 0c:02.0 FireWire (IEEE 1394) [0c00]: Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx] [104c:8023]
IOMMU Group 23 0d:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11)
IOMMU Group 3 00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04)
IOMMU Group 4 00:16.0 Communication controller [0780]: Intel Corporation 7 Series/C216 Chipset Family MEI Controller #1 [8086:1e3a] (rev 04)
IOMMU Group 5 00:19.0 Ethernet controller [0200]: Intel Corporation 82579V Gigabit Network Connection [8086:1503] (rev 04)
IOMMU Group 6 00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04)
IOMMU Group 7 00:1b.0 Audio device [0403]: Intel Corporation 7 Series/C216 Chipset Family High Definition Audio Controller [8086:1e20] (rev 04)
IOMMU Group 8 00:1c.0 PCI bridge [0604]: Intel Corporation 7 Series/C216 Chipset Family PCI Express Root Port 1 [8086:1e10] (rev c4)
IOMMU Group 9 00:1c.4 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 [8086:1e18] (rev c4)

 

[dcsapak]

IOMMU Group 1 01:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
IOMMU Group 1 02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1080] [10de:1b80] (rev a1)
IOMMU Group 1 02:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)

both pci devices are in the same iommu group, so you can only passthrough both, or none

 

[bobzer]

thank you for your really quick answer

i guess the only way around this problem is to physically change where areplug the card to make them in different groups if possible?