AMD gpu passthrough issue - windows 10 - steps included

Q-wulf

Well-Known Member
Mar 3, 2013
613
38
48
my test location
So, i am trying to pass my GPU through to my windows 10 VM using seabios.

I have so far overcome some issues, like e.g. Windows 10 updating Device drivers on its own, etc. I am actually able to boot into the VM using the GPU and its DVI-Ports as output for a single monitor. USB mouse + Keyboard works too. But a major one remains, which has put me on the end of my ropes.

I can not install the AMD-Driver without the screen going blank. (see at bottom of the post)

My System said:
  • I7-3930k
  • GA-x79-UD3
  • ATI x290 8GB
  • Onboard-Sound
  • Usb Keyboard, Old Logitech Cordless Mouse, Logitech T650 (unified Adapter)
  • running ontop of a local Ceph-Replication 2/1 SSD-Pool (6 OSD, single node)
  • Using virtio-win-0.1.1.iso (afaik from Fedora or redhat)

Following this guide:
https://pve.proxmox.com/wiki/Pci_passthrough
nano /etc/default/grub
change
Code:
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
to
Code:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
then
Code:
update-grub
reboot



Find the right USB and PCIE Device id's following the following guides:
https://pve.proxmox.com/wiki/Pci_passthrough
https://pve.proxmox.com/wiki/USB_physical_port_mapping#Add_the_device_to_the_VM
Code:
lsusb -t
lsusb
Bus 005 Device 004: ID 046d:c50d Logitech, Inc. Cordless Mouse
Bus 005 Device 003: ID 0566:3107 Monterey International Corp. Keyboard

Code:
lspci
...
00:1b.0 Audio device: Intel Corporation C600/X79 series chipset High Definition Audio Controller (rev 06)
...
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT [Radeon R9 290X]
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio
...







  • Set up VM.
    • Select Windows 8.1/2012
    • Add a new vDisk using Virtio Bus (iothread + Write back Cache) as Virtio0
    • add Qemu agent -> yes
    • Add windows 10 Iso as First DVD on Sata0
    • Add Virtio-win-0.1.110 as secondary DVD on Sata1
    • DO NOT assign any Network devices yet ( see https://forum.proxmox.com/threads/d...rivers-from-windows-update.25094/#post-125881 )
    • Select Virtio Controller via Options Tab (afaik best practice is to use SCSI ontop of Virtio controller for Windows on SSDs)
    • Verify your boot Orders. —> Disk then CD-Rom
  • Install windows 10
    • Start Windows VM - open noVNC console
    • Boot from Windows ISO (press any key to start from DVD/CD . . .)
    • select “Install Now”
    • select windows 10 pro x64
    • custom install
      • “load Driver” option
        • Select Virtio-DVD > Viostor > w8.1 > amd64
        • Install Red Hat VirtIO SCSI controller (driver for your virtio0)
      • “load Driver” option
        • Select Virtio-DVD > VioSCSI > w8.1 > amd64
        • uncheck the checkbox
        • Install Red Hat VirtIO SCSI pass-through controller (in case you need it later)
      • “load Driver” option
        • Select Virtio-DVD > netKVM > w8.1 > amd64
        • uncheck the checkbox
        • Install Red Hat VirtIO Ethernet Adapter (you will need this later)
    • Install windows regularly (like you'd normally do)
  • Fix windows 10 Windows Update settings
    • boot into windows
      • Disable Driver updates via Windows Update (important to prevent automatic updating by Microsoft’s menace called windows update)
        • Right click windows symbol —> Control Panel —> view by small Items —> System —> Advanced system settings —> Hardware -> Device installation Settings: Select “NO
      • Edit the power plan settings (while you are at it)
      • Right click start -> Control Pannel -> Power Options -> additional Plans -> High performance - change plan settings
        • Advanced Settings:
          • NEVER turn off HardDisks
          • Hibernate after : NEVER
          • Usb selective Suspend: NEVER
          • Sleep Button Action: Do nothing
          • NEVER turn off display
    • shutdown Vm
    • add a network device (virtio)
  • Create a snapshot (so you can roll back in case windows update does weird stuff)
  • Windows Update time
    • start VM
    • give yourself a chance to undo what microsoft in their Wisdom thinks you should do to your VM:
      • Go to windows Symbol in task bar -> settings -> update & security -> Windows Update -> Advanced options -> Notify to schedule restart + defer Upgrades.
    • Do a manual windows update
    • Restart Windows (as in reboot, verify it still works)
    • Shut down Windows VM
  • make snapshot (This on you will most likely use very often - I sure do)
  • Pass GPU + Usb through
    • nano /etc/pve/nodes/<hostname>/qemu-server/<vmnumber>.conf
      • usb0: host=046d:c50d
      • usb1: host=0566:3107
      • machine: q35
      • hostpci0: 02:00,x-vga=on,pcie=1
      • hostpci1: 00:1b.0
    • Start VM
    • Install Driver


Now as i mentioned above:
The second i try to install the AMD driver the screen goes blank, i can watch via "ceph-w" the data being written on the vDisk. Once its done i can only restart using ( qm stop <vmid>). keycombos via USB do not work anymore. A restart of the VM does not bring the GPU up. It seems the only way to get back into the vm is to:

  1. Rollback snapshot
  2. Restart Host

Any help, pointers, ideas or even theories how i might be able to pass that GPU through ?
 
Last edited:

Q-wulf

Well-Known Member
Mar 3, 2013
613
38
48
my test location
Do yo have tested with last testing kernel ?

https://forum.proxmox.com/threads/proxmox-4-0-pci-passthrough-broken-in-several-ways.24178/page-2
ftp://download1.proxmox.com/debian/dists/jessie/pvetest/binary-amd64/pve-kernel-4.2.6-1-pve_4.2.6-29_amd64.deb

I tested with
pve-kernel-4.2.6-1-pve: 4.2.6-28

Will try that kernel tonight and report back.

Do you have tested with ovmf bios ?
(bios: ovmf)

Yes, i have tried that. The Screen stays blank after passing it through (won't even show boot menu)

For omvf, do i need a GPU with UEFI bios ?
 

spirit

Famous Member
Apr 2, 2010
5,916
712
143
www.odiso.com
I tested with
pve-kernel-4.2.6-1-pve: 4.2.6-28

Will try that kernel tonight and report back.

this kernel version has specific fixes for pci passthrough (regressions introduced in kernel 4.1)


[QUOTE
Yes, i have tried that. The Screen stays blank after passing it through (won't even show boot menu)

For omvf, do i need a GPU with UEFI bios ?[/QUOTE]
[/QUOTE]

AFAIK, they are no GPU UEFI bios. UEFI is only inside motherboard. (in this case qemu virtual motherboard).
For ovmf, you need to have an uefi partition on your virtual harddrive to be able to boot.
(So you need to do a clean os install first, with ovmf enable, like this windows will create the uefi partition. You don't need to enable pci passthrough for this step)

Edit : But indeed, video card rom need to support UEFI.
http://vfio.blogspot.be/2014/08/does-my-graphics-card-rom-support-efi.html
 
Last edited:

Q-wulf

Well-Known Member
Mar 3, 2013
613
38
48
my test location
this kernel version has specific fixes for pci passthrough (regressions introduced in kernel 4.1)


Okay, i have now the following kernel running
pveversion -v
proxmox-ve: 4.1-28 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-2 (running version: 4.1-2/78c5f4a2)
pve-kernel-4.2.6-1-pve: 4.2.6-29

I noticed something (input from from my test last night with steam OS).
• When i set "default" under Hardware -> Display, windows will not recognise the GPU as AMD (generic display adapter)
• When i set "Standard VGA (std) under Hardware -> Display, windows will recognise the Adapter as the following:
Code:
AMD Atombios
Microsoft Basic Display Adapter
Bios Hawaii

Total available Graphics Memory: 4095 MB
Dedicated Video Memory: 0 MB
System Video Memory: 0 MB
Shared System Memory: 4095 MB

Note: it is a 8GB Variant of a MSI branded AMD 290x

I download the radeon-crimson-15.12-minimalsetup_web.exe and execute it.
Deselect everything except the "runtimes" and AMD-Driver.

The Installer keeps downloading/installing the Files, when it reaches around 60% (where it normally initializes the driver) the Monitor goes into sleep-mode. I can see from "Ceph -w" that the installer keeps writing stuff to vDisk for a good 120 seconds. Until it finishes.

At this pint i wait 2-3 more minutes (with only minimal 0-2 op/s via ceph-w)

I then issued "qm shutdown 1056" which seems to be hanging. After a while ,
I then issued "qm stop 1056" which stops the vm.

Syslog here (too long to post):
https://www.dropbox.com/s/fuz2u4584xrozhl/syslog_pcie_jan3_2016.txt

For omvf, do i need a GPU with UEFI bios ?
AFAIK, they are no GPU UEFI bios. UEFI is only inside motherboard. (in this case qemu virtual motherboard).

I ask because MSI (my GPU vendor) offer custom UEFI vBios On their support forums, that you can flash over your normal GPU bios when you send em a copy of your current bios. Found that out last night while tinkering with steamOS (out of frustration :) ).

Good to know that i do not need it normally for omvf.

For ovmf, you need to have an uefi partition on your virtual harddrive to be able to boot.
(So you need to do a clean os install first, with ovmf enable, like this windows will create the uefi partition. You don't need to enable pci passthrough for this step)
i'll do this next. with pve-kernel-4.2.6-1-pve: 4.2.6-29 on a separate VM.
 

spirit

Famous Member
Apr 2, 2010
5,916
712
143
www.odiso.com
by default,proxmox disable virtual vga card (vga:none) when xgva is enabled on pci passthough.
Until you setup manually a vgacard (std,vga,...).
I don't have any idea why this is change radeon detection...
 

Q-wulf

Well-Known Member
Mar 3, 2013
613
38
48
my test location
Any idea what else could try ? the fault definitely happens when the AMD-driver is initialised during install.

Any test i can perform (without the actual AMD chrimson driver ) to check if the GPU gets properly passed through ?
 

spirit

Famous Member
Apr 2, 2010
5,916
712
143
www.odiso.com
Any idea what else could try ? the fault definitely happens when the AMD-driver is initialised during install.

Any test i can perform (without the actual AMD chrimson driver ) to check if the GPU gets properly passed through ?

Try ovmf.(maybe this need video card rom patching to support uefi)
 

Q-wulf

Well-Known Member
Mar 3, 2013
613
38
48
my test location
Try with OMVF

Code:
pveversion -v
proxmox-ve: 4.1-28 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-2 (running version: 4.1-2/78c5f4a2)
pve-kernel-4.2.6-1-pve: 4.2.6-29

1060.conf
Code:
bios: ovmf
bootdisk: virtio0
cores: 6
cpu: kvm64
hostpci0: 02:00,x-vga=on,pcie=1
hostpci1: 00:1b.0
keyboard: de
machine: q35
memory: 8096
name: Win10-OMVF.local.lan
net0: virtio=32:62:64:62:39:38,bridge=vmbr0
numa: 0
ostype: win8
parent: WindowsUpdated
sata0: local:iso/Win10_1511_EnglishInternational_x64.iso,media=cdrom,size=3895734K
sata1: local:iso/virtio-win-0.1.110.iso,media=cdrom,size=55260K
scsihw: virtio-scsi-pci
smbios1: uuid=adb7a35f-ac72-491a-9ab6-a6c13269fbb8
sockets: 1
usb0: host=046d:c50d
usb1: host=0566:3107
virtio0: ceph_r2:vm-1060-disk-1,cache=writeback,iothread=on,size=300G


also with ovmf, you can try to remove "machine: q35".

Throws this error:
Code:
TASK ERROR: q35 machine model is not enabled at /usr/share/perl5/PVE/QemuServer.pm line 2768.

With "machine: q35" the monitor just freezes. It boots into windows in the background (as i can access network shares after a while)


Any ideas ?


ps.:
I shall see about getting a UEFI bios from MSI.
In the meantime i still have a AMD HD 7970 i can test later.
to Rule out that graphics card being the culprit (8GB variant) as i have seen lots of examples of people passing the GPU (4GB version) on a multitude on Systems (arch, fedora, ubuntu)
 

spirit

Famous Member
Apr 2, 2010
5,916
712
143
www.odiso.com
Throws this error:
Code:
TASK ERROR: q35 machine model is not enabled at /usr/share/perl5/PVE/QemuServer.pm line 2768.

also remove ",pcie" from hostpci if you disable q35.
 

Q-wulf

Well-Known Member
Mar 3, 2013
613
38
48
my test location
Throws this error:
Code:
TASK ERROR: q35 machine model is not enabled at /usr/share/perl5/PVE/QemuServer.pm line 2768.

also remove ",pcie" from hostpci if you disable q35.

Tried this just now with the
  • ATI x290 8GB
Does not throw that error anymore, but still freezes the Monitor before actually booting the win10-VM before even installing a GPU driver. (in Seabios it started freezing once you installed the GPU-Driver and/or booted into a win10-Vm that had the driver installed).

Will test with 7970 now and report back results. (this should definitely work from a GPU-being-the-issue pov as i have seen a multitude of threads regarding the specific GPU i'm using)
 

Q-wulf

Well-Known Member
Mar 3, 2013
613
38
48
my test location
I tried some more Combinations.
Red did not work.
Green did work.



7970 (TahitiXT) on OMVF Win10 results:

  • Without "machine: q35" --> Monitor goes straight to sleep. Only way to recover is to "reboot host"
  • With "machine: q35" --> Monitor goes straight to sleep. Only way to recover is to "reboot host"
7970 (TahitiXT) on SeaBios Win10 results:
  • Monitor Changes to Windows Boot. Boots into windows 10 just fine. Can change Screen resolution (indicates Windows installed a driver). Does not however recognise the Display adapter (all entries are <unavailable>).
  • First Boot: Crashes after 90 seconds with the following blue-screen: "System_thread_Exception_not_handled (atikmdad.sys)"
    • Can be recovered by doing a VM reset via GUI/qm.
  • Second Boot: Amd Autodetect utility does not recognise the GPU. "does not crash" for 15 Minutes
    • Install of Minmal Chrimson Edition 15.12 (Display Driver + Runtimes only) results in crash on step "currently installing: AMD Display Driver"
    • Bluescreen "System_thread_Exception_not_handled (atikmdad.sys)"
    • Can be recovered by doing a VM reset via GUI/qm.
    • Event Viewer "System" shows
      • The previous System Shutdown was unexspected
      • Driver Management concluded the process ti install driver amdkmafd.inf _amd64 .... with following status: 0x0





7970 (TahitiXT) on SeaBios Ubuntu 15.10 results :
Code:
machine: q35
hostpci0: 02:00,x-vga=on,pcie=1
Fresh install updated + steam installed.

Try one:
  • set Driver via "Software & updates" -> "Additional Drivers" --> Using Xorg X server AMD/ATI display driver wrapper
    • detects make and model properly. (HD 7970 IceQ X2)
      • Test with Game "Cities in Motion 2"
        • no crash.
        • Low performance (as expected)
  • set Driver via "Software & updates" -> "Additional Drivers" --> Using fglrx (propriatary)
    • Goes into endless loop of Boot to Ubuntu Account selection -> Reset.
    • Monitor does NOT go into deep sleep and i can just " qm stop" and rollback Snapshot


Pretty sure its not the GPU that is the issue.


Any more Ideas ?
 

Q-wulf

Well-Known Member
Mar 3, 2013
613
38
48
my test location
based on this:
https://forum.proxmox.com/threads/p...oken-in-several-ways.24178/page-2#post-127856

i have checked if i might have something else on the iommu_groups
Code:
ls /sys/kernel/iommu_groups/20/devices
0000:02:00.0 <-- VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X]
0000:02:00.1 <-- Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]


Code:
ls /sys/kernel/iommu_groups/10/devices
0000:00:1b.0  <-- 00:1b.0 Audio device: Intel Corporation C600/X79 series chipset High Definition Audio Controller (rev 06)

I guess that is not my issue then ?


Guess
 

spirit

Famous Member
Apr 2, 2010
5,916
712
143
www.odiso.com
maybe this could help:
https://www.redhat.com/archives/vfio-users/2015-October/msg00119.html

user describe how he's able to install catalyst driver. (he had the same bug than you)


"
I also had problems with installing Catalyst higher than 14.12 for my r9 290 on both win 8.1 and 10.
BSOD and rebooting cycle after installing Catalyst.
The solution was to add ioh3420 and pass the GPU through it.
I using qemu from commandline with following parameters for GPU:

-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0 \

Afterwards win10 automatically installed the latest non-beta drivers without problems.

"


I can make a test qemu-server package if you want
 

Q-wulf

Well-Known Member
Mar 3, 2013
613
38
48
my test location
I shall test them both after i get me some more coffee :p

I opened the settings for resolution and disabled the output on the qxl
gpu, which left me with no output.
What does he mean by "disabled the output on the qxl" ??
As in how did he do that ?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!