first pcie express disappear when passthrough another one

B

bobzer

Well-Known Member
Nov 14, 2017
37
8
48
37
Hi,

I got a pcie sas controller, (LSI 9211-8i SAS SA) plug in the first pcie
in another pcie i got a graphics card that i need to passthrough a VM.

so my problem is that both card works well but not together....

when i start the vm that got the graphics card the other card just disapear...
I really don't know what to do, even where to start investigation.
the only peace of information that i found is the console that show this :
Code: 
[  969.449634] VFIO - User Level meta-driver version: 0.3
[  969.453604] vfio-pci 0000:02:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[  969.470012] vfio_pci: add [10de:1b80[ffff:ffff]] class 0x000000/00000000
[  969.470054] vfio_pci: add [10de:10f0[ffff:ffff]] class 0x000000/00000000
[  969.594135] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[  969.594186] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[  969.618208] mpt2sas_cm0: removing handle(0x000b), sas_addr(0x4433221103000000)
[  969.618235] mpt2sas_cm0: removing : enclosure logical id(0x500605b0013ca580), slot(0)
[  969.634134] sd 0:0:1:0: [sdb] Synchronizing SCSI cache
[  969.634187] sd 0:0:1:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[  969.658225] mpt2sas_cm0: removing handle(0x0009), sas_addr(0x4433221100000000)
[  969.658249] mpt2sas_cm0: removing : enclosure logical id(0x500605b0013ca580), slot(3)
[  969.678169] sd 0:0:2:0: [sdc] Synchronizing SCSI cache
[  969.678230] sd 0:0:2:0: [sdc] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[  969.702191] mpt2sas_cm0: removing handle(0x000a), sas_addr(0x4433221102000000)
[  969.702224] mpt2sas_cm0: removing : enclosure logical id(0x500605b0013ca580), slot(1)
[  969.722125] sd 0:0:3:0: [sdd] Synchronizing SCSI cache
[  969.722196] sd 0:0:3:0: [sdd] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[  969.766182] mpt2sas_cm0: removing handle(0x000c), sas_addr(0x4433221107000000)
[  969.766209] mpt2sas_cm0: removing : enclosure logical id(0x500605b0013ca580), slot(4)
[  969.766530] mpt2sas_cm0: sending message unit reset !!
[  969.768083] mpt2sas_cm0: message unit reset: SUCCESS
[  970.755992] device tap300i0 entered promiscuous mode
[  970.761675] vmbr0: port 2(tap300i0) entered blocking state
[  970.761697] vmbr0: port 2(tap300i0) entered disabled state
[  970.761803] vmbr0: port 2(tap300i0) entered blocking state
[  970.761822] vmbr0: port 2(tap300i0) entered forwarding state
[  971.426022] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003)
[  971.426192] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x900
[  976.158048] usb 1-1.2: reset full-speed USB device number 3 using ehci-pci
[  976.466036] usb 1-1.2: reset full-speed USB device number 3 using ehci-pci
[  981.394030] usb 1-1.2: reset full-speed USB device number 3 using ehci-pci
[  981.606022] usb 1-1.2: reset full-speed USB device number 3 using ehci-pci
[  981.838027] usb 1-1.2: reset full-speed USB device number 3 using ehci-pci
[  982.038062] usb 1-1.2: reset full-speed USB device number 3 using ehci-pci
[  986.487207] kvm [3449]: vcpu3, guest rIP: 0xfffff807909e62b6 ignored rdmsr: 0x611
[  986.487249] kvm [3449]: vcpu3, guest rIP: 0xfffff807909e62cc ignored rdmsr: 0x641
[  986.487275] kvm [3449]: vcpu3, guest rIP: 0xfffff807909e62e2 ignored rdmsr: 0x606
[  986.487300] kvm [3449]: vcpu3, guest rIP: 0xfffff807909e610e ignored rdmsr: 0x606
[  986.487326] kvm [3449]: vcpu3, guest rIP: 0xfffff807909e119c ignored rdmsr: 0x641
[  986.487351] kvm [3449]: vcpu3, guest rIP: 0xfffff807909e11e7 ignored rdmsr: 0x611
[  986.802039] usb 1-1.2: reset full-speed USB device number 3 using ehci-pci
[  986.998060] usb 1-1.2: reset full-speed USB device number 3 using ehci-pci
[  987.362040] usb 1-1.2: reset full-speed USB device number 3 using ehci-pci

the config file of the vm is :
Code: 
agent: 1
balloon: 2048
bios: ovmf
bootdisk: virtio0
cores: 4
cpu: IvyBridge,hidden=1
efidisk0: data:vm-300-disk-1,size=4M
hostpci0: 02:00,pcie=1,x-vga=on
ide0: none,media=cdrom
machine: q35
memory: 8192
name: win-emma-gtx
net0: virtio=CE:E3:3A:68:AB:5F,bridge=vmbr0
numa: 0
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=60c6b50e-a94e-46b4-a7d5-740e03ffb35d
sockets: 1
usb0: host=1-1.2
vga: qxl
virtio0: ssd:vm-300-disk-1,cache=writeback,size=250G
virtio1: data:vm-300-disk-2,cache=writeback,size=500G

hope you can help me

thank you
 
can you post your iommu groups ?
Code: 
#!/bin/bash
shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do
    n=${d#*/iommu_groups/*}; n=${n%%/*}
    printf 'IOMMU Group %s ' "$n"
    lspci -nns "${d##*/}"
done;
 
thanks for you time
here is the results of your script :
Code: 
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller [8086:0150] (rev 09)
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)
IOMMU Group 1 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0155] (rev 09)
IOMMU Group 1 01:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
IOMMU Group 1 02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1080] [10de:1b80] (rev a1)
IOMMU Group 1 02:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
IOMMU Group 10 00:1c.5 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 6 [8086:1e1a] (rev c4)
IOMMU Group 11 00:1c.7 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 [8086:1e1e] (rev c4)
IOMMU Group 12 00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04)
IOMMU Group 13 00:1f.0 ISA bridge [0601]: Intel Corporation Z77 Express Chipset LPC Controller [8086:1e44] (rev 04)
IOMMU Group 13 00:1f.2 SATA controller [0106]: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] [8086:1e02] (rev 04)
IOMMU Group 13 00:1f.3 SMBus [0c05]: Intel Corporation 7 Series/C216 Chipset Family SMBus Controller [8086:1e22] (rev 04)
IOMMU Group 14 04:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]
IOMMU Group 15 05:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
IOMMU Group 16 06:01.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
IOMMU Group 17 06:04.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
IOMMU Group 18 06:05.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
IOMMU Group 19 06:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
IOMMU Group 2 00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller [8086:0162] (rev 09)
IOMMU Group 20 06:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba)
IOMMU Group 21 08:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11)
IOMMU Group 22 0b:00.0 PCI bridge [0604]: Integrated Technology Express, Inc. Device [1283:8892] (rev 30)
IOMMU Group 22 0c:02.0 FireWire (IEEE 1394) [0c00]: Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx] [104c:8023]
IOMMU Group 23 0d:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller [1b4b:9172] (rev 11)
IOMMU Group 3 00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04)
IOMMU Group 4 00:16.0 Communication controller [0780]: Intel Corporation 7 Series/C216 Chipset Family MEI Controller #1 [8086:1e3a] (rev 04)
IOMMU Group 5 00:19.0 Ethernet controller [0200]: Intel Corporation 82579V Gigabit Network Connection [8086:1503] (rev 04)
IOMMU Group 6 00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04)
IOMMU Group 7 00:1b.0 Audio device [0403]: Intel Corporation 7 Series/C216 Chipset Family High Definition Audio Controller [8086:1e20] (rev 04)
IOMMU Group 8 00:1c.0 PCI bridge [0604]: Intel Corporation 7 Series/C216 Chipset Family PCI Express Root Port 1 [8086:1e10] (rev c4)
IOMMU Group 9 00:1c.4 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 [8086:1e18] (rev c4)
 
bobzer said:
IOMMU Group 1 01:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
IOMMU Group 1 02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1080] [10de:1b80] (rev a1)
IOMMU Group 1 02:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
Click to expand...
both pci devices are in the same iommu group, so you can only passthrough both, or none
 
thank you for your really quick answer

i guess the only way around this problem is to physically change where areplug the card to make them in different groups if possible?
 
You must log in or register to reply here.
Top Bottom