GPU Passthrough issue after adding NVME drive

J

jtracy

Member
Aug 30, 2018
43
4
13
50
I had GPU passthrough working great on my system, but I added a NVME drive to the systems and am now having a strange problem. When I added the drive it renumbered a bunch of the PCI-E devices and a couple of those were passed into the VM (GPU and 2 USB devices). So after the upgrade I changed the device numbers in the VM config. After doing that the VM boots, but I don't see anything happening with the screens. But if I shutdown the VM it initializes the screens and shows me Windows shutdown process. After that I can boot the VM just find and interact with Windows.

Anybody have an idea of what is going on and why proxmox/Windows are not initializing the screens?

Code: 
# cat /etc/pve/qemu-server/100.conf
agent: 1
balloon: 0
bios: ovmf
bootdisk: scsi0
cores: 12
efidisk0: Fast-VMs:vm-100-disk-0,size=4M
hostpci0: 09:00,pcie=1,x-vga=1,romfile=GP104.rom
hostpci1: 06:00.1,pcie=1
hostpci2: 06:00.3,pcie=1
ide2: local:iso/virtio-win-0.1.171.iso,media=cdrom,size=363020K
machine: q35
memory: 12288
name: Windows
net0: virtio=4E:BF:4A:0A:B9:06,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win10
parent: Pre_WSL2
protection: 1
rng0: source=/dev/urandom
scsi0: BigNFast-VMs:vm-100-disk-0,discard=on,size=400G
scsi1: Fast-VMs:vm-100-disk-2,size=1G
scsihw: virtio-scsi-pci
smbios1: uuid=e2d2f136-6c4f-44e3-b12a-2cfe0128e8ba
sockets: 1
startup: order=1,up=15
unused0: NAS00:100/vm-100-disk-0.qcow2
unused1: NAS00:100/vm-100-disk-1.qcow2
unused2: Fast-VMs:vm-100-disk-1
vga: qxl
vmgenid: 3fcc234c-45a8-4823-878d-2a9d1b36316e

[Pre_WSL2]
#Before installing WSL2
agent: 1
balloon: 8192
bios: ovmf
bootdisk: scsi0
cores: 12
efidisk0: Fast-VMs:vm-100-disk-0,size=4M
hostpci0: 08:00,pcie=1,x-vga=1,romfile=GP104.rom
hostpci1: 05:00.1,pcie=1
hostpci2: 05:00.3,pcie=1
ide2: local:iso/virtio-win-0.1.171.iso,media=cdrom,size=363020K
machine: q35
memory: 16384
name: Windows
net0: virtio=4E:BF:4A:0A:B9:06,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
protection: 1
rng0: source=/dev/urandom
scsi0: Fast-VMs:vm-100-disk-1,discard=on,size=400G
scsi1: Fast-VMs:vm-100-disk-2,size=1G
scsihw: virtio-scsi-pci
smbios1: uuid=e2d2f136-6c4f-44e3-b12a-2cfe0128e8ba
snaptime: 1589393220
sockets: 1
startup: order=1,up=15
vga: qxl
vmgenid: 3fcc234c-45a8-4823-878d-2a9d1b36316e

Code: 
# pveversion --verbose
proxmox-ve: 6.1-2 (running kernel: 5.3.18-3-pve)
pve-manager: 6.1-8 (running version: 6.1-8/806edfe1)
pve-kernel-helper: 6.1-8
pve-kernel-5.3: 6.1-6
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-2-pve: 5.3.18-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 2.0.1-1+pve8
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-17
libpve-guest-common-perl: 3.0-5
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 4.0.1-pve1
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-23
pve-docs: 6.1-6
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.0-10
pve-firmware: 3.0-7
pve-ha-manager: 3.0-9
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-4
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-7
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1
 
can you post the output of 'dmesg' after such an attempt? also the lspci + iommu groups would be interesting
 
So I rebooted the server and the VM started but nothing was displayed. This time I accidentally stopped the VM instead of rebooting the VM but it came up fine either way.

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
00:08.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7
01:00.0 Non-Volatile memory controller: Sandisk Corp Device 5006
02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57ad
03:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3
03:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3
03:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
03:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
03:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
04:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981
05:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
06:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
06:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
06:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
07:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
08:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
09:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
09:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
0b:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
0b:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
0b:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller
0c:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
0d:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

/sys/kernel/iommu_groups/17/devices/0000:02:00.0
/sys/kernel/iommu_groups/7/devices/0000:00:05.0
/sys/kernel/iommu_groups/25/devices/0000:09:00.0
/sys/kernel/iommu_groups/25/devices/0000:09:00.1
/sys/kernel/iommu_groups/15/devices/0000:00:18.3
/sys/kernel/iommu_groups/15/devices/0000:00:18.1
/sys/kernel/iommu_groups/15/devices/0000:00:18.6
/sys/kernel/iommu_groups/15/devices/0000:00:18.4
/sys/kernel/iommu_groups/15/devices/0000:00:18.2
/sys/kernel/iommu_groups/15/devices/0000:00:18.0
/sys/kernel/iommu_groups/15/devices/0000:00:18.7
/sys/kernel/iommu_groups/15/devices/0000:00:18.5
/sys/kernel/iommu_groups/5/devices/0000:00:03.1
/sys/kernel/iommu_groups/23/devices/0000:04:00.0
/sys/kernel/iommu_groups/13/devices/0000:00:08.3
/sys/kernel/iommu_groups/31/devices/0000:0c:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:02.0
/sys/kernel/iommu_groups/21/devices/0000:03:09.0
/sys/kernel/iommu_groups/21/devices/0000:07:00.0
/sys/kernel/iommu_groups/11/devices/0000:00:08.1
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/28/devices/0000:0b:00.1
/sys/kernel/iommu_groups/18/devices/0000:03:01.0
/sys/kernel/iommu_groups/8/devices/0000:00:07.0
/sys/kernel/iommu_groups/26/devices/0000:0a:00.0
/sys/kernel/iommu_groups/16/devices/0000:01:00.0
/sys/kernel/iommu_groups/6/devices/0000:00:04.0
/sys/kernel/iommu_groups/24/devices/0000:05:00.0
/sys/kernel/iommu_groups/14/devices/0000:00:14.3
/sys/kernel/iommu_groups/14/devices/0000:00:14.0
/sys/kernel/iommu_groups/32/devices/0000:0d:00.0
/sys/kernel/iommu_groups/4/devices/0000:00:03.0
/sys/kernel/iommu_groups/22/devices/0000:08:00.0
/sys/kernel/iommu_groups/22/devices/0000:03:0a.0
/sys/kernel/iommu_groups/12/devices/0000:00:08.2
/sys/kernel/iommu_groups/30/devices/0000:0b:00.4
/sys/kernel/iommu_groups/2/devices/0000:00:01.2
/sys/kernel/iommu_groups/20/devices/0000:06:00.0
/sys/kernel/iommu_groups/20/devices/0000:03:08.0
/sys/kernel/iommu_groups/20/devices/0000:06:00.3
/sys/kernel/iommu_groups/20/devices/0000:06:00.1
/sys/kernel/iommu_groups/10/devices/0000:00:08.0
/sys/kernel/iommu_groups/29/devices/0000:0b:00.3
/sys/kernel/iommu_groups/0/devices/0000:00:01.0
/sys/kernel/iommu_groups/19/devices/0000:03:06.0
/sys/kernel/iommu_groups/9/devices/0000:00:07.1
/sys/kernel/iommu_groups/27/devices/0000:0b:00.0
 

Attachments

You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom