GPU Passthrough issue after adding NVME drive

J

jtracy

Member
Aug 30, 2018
43
4
13
51
I had GPU passthrough working great on my system, but I added a NVME drive to the systems and am now having a strange problem. When I added the drive it renumbered a bunch of the PCI-E devices and a couple of those were passed into the VM (GPU and 2 USB devices). So after the upgrade I changed the device numbers in the VM config. After doing that the VM boots, but I don't see anything happening with the screens. But if I shutdown the VM it initializes the screens and shows me Windows shutdown process. After that I can boot the VM just find and interact with Windows.

Anybody have an idea of what is going on and why proxmox/Windows are not initializing the screens?

Code: 
# cat /etc/pve/qemu-server/100.conf
agent: 1
balloon: 0
bios: ovmf
bootdisk: scsi0
cores: 12
efidisk0: Fast-VMs:vm-100-disk-0,size=4M
hostpci0: 09:00,pcie=1,x-vga=1,romfile=GP104.rom
hostpci1: 06:00.1,pcie=1
hostpci2: 06:00.3,pcie=1
ide2: local:iso/virtio-win-0.1.171.iso,media=cdrom,size=363020K
machine: q35
memory: 12288
name: Windows
net0: virtio=4E:BF:4A:0A:B9:06,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win10
parent: Pre_WSL2
protection: 1
rng0: source=/dev/urandom
scsi0: BigNFast-VMs:vm-100-disk-0,discard=on,size=400G
scsi1: Fast-VMs:vm-100-disk-2,size=1G
scsihw: virtio-scsi-pci
smbios1: uuid=e2d2f136-6c4f-44e3-b12a-2cfe0128e8ba
sockets: 1
startup: order=1,up=15
unused0: NAS00:100/vm-100-disk-0.qcow2
unused1: NAS00:100/vm-100-disk-1.qcow2
unused2: Fast-VMs:vm-100-disk-1
vga: qxl
vmgenid: 3fcc234c-45a8-4823-878d-2a9d1b36316e

[Pre_WSL2]
#Before installing WSL2
agent: 1
balloon: 8192
bios: ovmf
bootdisk: scsi0
cores: 12
efidisk0: Fast-VMs:vm-100-disk-0,size=4M
hostpci0: 08:00,pcie=1,x-vga=1,romfile=GP104.rom
hostpci1: 05:00.1,pcie=1
hostpci2: 05:00.3,pcie=1
ide2: local:iso/virtio-win-0.1.171.iso,media=cdrom,size=363020K
machine: q35
memory: 16384
name: Windows
net0: virtio=4E:BF:4A:0A:B9:06,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
protection: 1
rng0: source=/dev/urandom
scsi0: Fast-VMs:vm-100-disk-1,discard=on,size=400G
scsi1: Fast-VMs:vm-100-disk-2,size=1G
scsihw: virtio-scsi-pci
smbios1: uuid=e2d2f136-6c4f-44e3-b12a-2cfe0128e8ba
snaptime: 1589393220
sockets: 1
startup: order=1,up=15
vga: qxl
vmgenid: 3fcc234c-45a8-4823-878d-2a9d1b36316e

Code: 
# pveversion --verbose
proxmox-ve: 6.1-2 (running kernel: 5.3.18-3-pve)
pve-manager: 6.1-8 (running version: 6.1-8/806edfe1)
pve-kernel-helper: 6.1-8
pve-kernel-5.3: 6.1-6
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-2-pve: 5.3.18-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 2.0.1-1+pve8
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-17
libpve-guest-common-perl: 3.0-5
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 4.0.1-pve1
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-23
pve-docs: 6.1-6
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.0-10
pve-firmware: 3.0-7
pve-ha-manager: 3.0-9
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-4
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-7
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1
 
can you post the output of 'dmesg' after such an attempt? also the lspci + iommu groups would be interesting
 
So I rebooted the server and the VM started but nothing was displayed. This time I accidentally stopped the VM instead of rebooting the VM but it came up fine either way.

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
00:08.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7
01:00.0 Non-Volatile memory controller: Sandisk Corp Device 5006
02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57ad
03:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3
03:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3
03:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
03:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
03:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
04:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981
05:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
06:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
06:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
06:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
07:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
08:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
09:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
09:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
0b:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
0b:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
0b:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller
0c:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
0d:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

/sys/kernel/iommu_groups/17/devices/0000:02:00.0
/sys/kernel/iommu_groups/7/devices/0000:00:05.0
/sys/kernel/iommu_groups/25/devices/0000:09:00.0
/sys/kernel/iommu_groups/25/devices/0000:09:00.1
/sys/kernel/iommu_groups/15/devices/0000:00:18.3
/sys/kernel/iommu_groups/15/devices/0000:00:18.1
/sys/kernel/iommu_groups/15/devices/0000:00:18.6
/sys/kernel/iommu_groups/15/devices/0000:00:18.4
/sys/kernel/iommu_groups/15/devices/0000:00:18.2
/sys/kernel/iommu_groups/15/devices/0000:00:18.0
/sys/kernel/iommu_groups/15/devices/0000:00:18.7
/sys/kernel/iommu_groups/15/devices/0000:00:18.5
/sys/kernel/iommu_groups/5/devices/0000:00:03.1
/sys/kernel/iommu_groups/23/devices/0000:04:00.0
/sys/kernel/iommu_groups/13/devices/0000:00:08.3
/sys/kernel/iommu_groups/31/devices/0000:0c:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:02.0
/sys/kernel/iommu_groups/21/devices/0000:03:09.0
/sys/kernel/iommu_groups/21/devices/0000:07:00.0
/sys/kernel/iommu_groups/11/devices/0000:00:08.1
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/28/devices/0000:0b:00.1
/sys/kernel/iommu_groups/18/devices/0000:03:01.0
/sys/kernel/iommu_groups/8/devices/0000:00:07.0
/sys/kernel/iommu_groups/26/devices/0000:0a:00.0
/sys/kernel/iommu_groups/16/devices/0000:01:00.0
/sys/kernel/iommu_groups/6/devices/0000:00:04.0
/sys/kernel/iommu_groups/24/devices/0000:05:00.0
/sys/kernel/iommu_groups/14/devices/0000:00:14.3
/sys/kernel/iommu_groups/14/devices/0000:00:14.0
/sys/kernel/iommu_groups/32/devices/0000:0d:00.0
/sys/kernel/iommu_groups/4/devices/0000:00:03.0
/sys/kernel/iommu_groups/22/devices/0000:08:00.0
/sys/kernel/iommu_groups/22/devices/0000:03:0a.0
/sys/kernel/iommu_groups/12/devices/0000:00:08.2
/sys/kernel/iommu_groups/30/devices/0000:0b:00.4
/sys/kernel/iommu_groups/2/devices/0000:00:01.2
/sys/kernel/iommu_groups/20/devices/0000:06:00.0
/sys/kernel/iommu_groups/20/devices/0000:03:08.0
/sys/kernel/iommu_groups/20/devices/0000:06:00.3
/sys/kernel/iommu_groups/20/devices/0000:06:00.1
/sys/kernel/iommu_groups/10/devices/0000:00:08.0
/sys/kernel/iommu_groups/29/devices/0000:0b:00.3
/sys/kernel/iommu_groups/0/devices/0000:00:01.0
/sys/kernel/iommu_groups/19/devices/0000:03:06.0
/sys/kernel/iommu_groups/9/devices/0000:00:07.1
/sys/kernel/iommu_groups/27/devices/0000:0b:00.0
 

Attachments

You must log in or register to reply here.
Top Bottom