AMD gpu passthrough issue - windows 10 - steps included

Q-wulf · Jan 2, 2016

So, i am trying to pass my GPU through to my windows 10 VM using seabios.

I have so far overcome some issues, like e.g. Windows 10 updating Device drivers on its own, etc. I am actually able to boot into the VM using the GPU and its DVI-Ports as output for a single monitor. USB mouse + Keyboard works too. But a major one remains, which has put me on the end of my ropes.

I can not install the AMD-Driver without the screen going blank. (see at bottom of the post)

My System said:
I7-3930k

GA-x79-UD3

ATI x290 8GB

Onboard-Sound

Usb Keyboard, Old Logitech Cordless Mouse, Logitech T650 (unified Adapter)

running ontop of a local Ceph-Replication 2/1 SSD-Pool (6 OSD, single node)

Using virtio-win-0.1.1.iso (afaik from Fedora or redhat)

Following this guide:
https://pve.proxmox.com/wiki/Pci_passthrough

Code:
nano /etc/default/grub
change

Code:

GRUB_CMDLINE_LINUX_DEFAULT="quiet"

to

Code:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

then

Code:

update-grub reboot

Find the right USB and PCIE Device id's following the following guides:
https://pve.proxmox.com/wiki/Pci_passthrough
https://pve.proxmox.com/wiki/USB_physical_port_mapping#Add_the_device_to_the_VM

Code:
Code:

lsusb -t lsusb

Bus 005 Device 004: ID 046d:c50d Logitech, Inc. Cordless Mouse
Bus 005 Device 003: ID 0566:3107 Monterey International Corp. Keyboard

Code:

lspci

...
00:1b.0 Audio device: Intel Corporation C600/X79 series chipset High Definition Audio Controller (rev 06)
...
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT [Radeon R9 290X]
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio
...

Set up VM.
- Select Windows 8.1/2012
- Add a new vDisk using Virtio Bus (iothread + Write back Cache) as Virtio0
- add Qemu agent -> yes
- Add windows 10 Iso as First DVD on Sata0
- Add Virtio-win-0.1.110 as secondary DVD on Sata1
- DO NOT assign any Network devices yet ( see https://forum.proxmox.com/threads/d...rivers-from-windows-update.25094/#post-125881 )
- Select Virtio Controller via Options Tab (afaik best practice is to use SCSI ontop of Virtio controller for Windows on SSDs)
- Verify your boot Orders. —> Disk then CD-Rom
Install windows 10
- Start Windows VM - open noVNC console
- Boot from Windows ISO (press any key to start from DVD/CD . . .)
- select “Install Now”
- select windows 10 pro x64
- custom install
  - “load Driver” option
    - Select Virtio-DVD > Viostor > w8.1 > amd64
    - Install Red Hat VirtIO SCSI controller (driver for your virtio0)
  - “load Driver” option
    - Select Virtio-DVD > VioSCSI > w8.1 > amd64
    - uncheck the checkbox
    - Install Red Hat VirtIO SCSI pass-through controller (in case you need it later)
  - “load Driver” option
    - Select Virtio-DVD > netKVM > w8.1 > amd64
    - uncheck the checkbox
    - Install Red Hat VirtIO Ethernet Adapter (you will need this later)
- Install windows regularly (like you'd normally do)
Fix windows 10 Windows Update settings
- boot into windows
  - Disable Driver updates via Windows Update (important to prevent automatic updating by Microsoft’s menace called windows update)
    - Right click windows symbol —> Control Panel —> view by small Items —> System —> Advanced system settings —> Hardware -> Device installation Settings: Select “NO
  - Edit the power plan settings (while you are at it)
  - Right click start -> Control Pannel -> Power Options -> additional Plans -> High performance - change plan settings
    - Advanced Settings:
      - NEVER turn off HardDisks
      - Hibernate after : NEVER
      - Usb selective Suspend: NEVER
      - Sleep Button Action: Do nothing
      - NEVER turn off display
- shutdown Vm
- add a network device (virtio)
Create a snapshot (so you can roll back in case windows update does weird stuff)
Windows Update time
- start VM
- give yourself a chance to undo what microsoft in their Wisdom thinks you should do to your VM:
  - Go to windows Symbol in task bar -> settings -> update & security -> Windows Update -> Advanced options -> Notify to schedule restart + defer Upgrades.
- Do a manual windows update
- Restart Windows (as in reboot, verify it still works)
- Shut down Windows VM
make snapshot (This on you will most likely use very often - I sure do)
Pass GPU + Usb through
- nano /etc/pve/nodes/<hostname>/qemu-server/<vmnumber>.conf
  - usb0: host=046d:c50d
  - usb1: host=0566:3107
  - machine: q35
  - hostpci0: 02:00,x-vga=on,pcie=1
  - hostpci1: 00:1b.0
- Start VM
- Install Driver

Now as i mentioned above:
The second i try to install the AMD driver the screen goes blank, i can watch via "ceph-w" the data being written on the vDisk. Once its done i can only restart using ( qm stop <vmid>). keycombos via USB do not work anymore. A restart of the VM does not bring the GPU up. It seems the only way to get back into the vm is to:

Rollback snapshot
Restart Host

Any help, pointers, ideas or even theories how i might be able to pass that GPU through ?

spirit · Jan 3, 2016

Do yo have teste with last testing kernel ?

https://forum.proxmox.com/threads/proxmox-4-0-pci-passthrough-broken-in-several-ways.24178/page-2
ftp://download1.proxmox.com/debian/dists/jessie/pvetest/binary-amd64/pve-kernel-4.2.6-1-pve_4.2.6-29_amd64.deb

Do you have tested with ovmf bios ?
(bios: ovmf)

Q-wulf · Jan 3, 2016

spirit said:
Do yo have tested with last testing kernel ?

https://forum.proxmox.com/threads/proxmox-4-0-pci-passthrough-broken-in-several-ways.24178/page-2
ftp://download1.proxmox.com/debian/dists/jessie/pvetest/binary-amd64/pve-kernel-4.2.6-1-pve_4.2.6-29_amd64.deb

I tested with
pve-kernel-4.2.6-1-pve: 4.2.6-28

Will try that kernel tonight and report back.

spirit said:
Do you have tested with ovmf bios ?
(bios: ovmf)

Yes, i have tried that. The Screen stays blank after passing it through (won't even show boot menu)

For omvf, do i need a GPU with UEFI bios ?

spirit · Jan 3, 2016

Q-wulf said:
I tested with
pve-kernel-4.2.6-1-pve: 4.2.6-28

Will try that kernel tonight and report back.

this kernel version has specific fixes for pci passthrough (regressions introduced in kernel 4.1)

[QUOTE
Yes, i have tried that. The Screen stays blank after passing it through (won't even show boot menu)

For omvf, do i need a GPU with UEFI bios ?[/QUOTE]
[/QUOTE]

AFAIK, they are no GPU UEFI bios. UEFI is only inside motherboard. (in this case qemu virtual motherboard).
For ovmf, you need to have an uefi partition on your virtual harddrive to be able to boot.
(So you need to do a clean os install first, with ovmf enable, like this windows will create the uefi partition. You don't need to enable pci passthrough for this step)

Edit : But indeed, video card rom need to support UEFI.
http://vfio.blogspot.be/2014/08/does-my-graphics-card-rom-support-efi.html

Q-wulf · Jan 3, 2016

spirit said:
this kernel version has specific fixes for pci passthrough (regressions introduced in kernel 4.1)

Okay, i have now the following kernel running

pveversion -v
proxmox-ve: 4.1-28 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-2 (running version: 4.1-2/78c5f4a2)
pve-kernel-4.2.6-1-pve: 4.2.6-29

I noticed something (input from from my test last night with steam OS).
• When i set "default" under Hardware -> Display, windows will not recognise the GPU as AMD (generic display adapter)
• When i set "Standard VGA (std) under Hardware -> Display, windows will recognise the Adapter as the following:

Code:

AMD Atombios
Microsoft Basic Display Adapter
Bios Hawaii

Total available Graphics Memory: 4095 MB
Dedicated Video Memory: 0 MB
System Video Memory: 0 MB
Shared System Memory: 4095 MB

Note: it is a 8GB Variant of a MSI branded AMD 290x

I download the radeon-crimson-15.12-minimalsetup_web.exe and execute it.
Deselect everything except the "runtimes" and AMD-Driver.

The Installer keeps downloading/installing the Files, when it reaches around 60% (where it normally initializes the driver) the Monitor goes into sleep-mode. I can see from "Ceph -w" that the installer keeps writing stuff to vDisk for a good 120 seconds. Until it finishes.

At this pint i wait 2-3 more minutes (with only minimal 0-2 op/s via ceph-w)

I then issued "qm shutdown 1056" which seems to be hanging. After a while ,
I then issued "qm stop 1056" which stops the vm.

Syslog here (too long to post):
https://www.dropbox.com/s/fuz2u4584xrozhl/syslog_pcie_jan3_2016.txt

spirit said:
For omvf, do i need a GPU with UEFI bios ?

Click to expand...

AFAIK, they are no GPU UEFI bios. UEFI is only inside motherboard. (in this case qemu virtual motherboard).

I ask because MSI (my GPU vendor) offer custom UEFI vBios On their support forums, that you can flash over your normal GPU bios when you send em a copy of your current bios. Found that out last night while tinkering with steamOS (out of frustration

).

Good to know that i do not need it normally for omvf.

spirit said:
For ovmf, you need to have an uefi partition on your virtual harddrive to be able to boot.
(So you need to do a clean os install first, with ovmf enable, like this windows will create the uefi partition. You don't need to enable pci passthrough for this step)

i'll do this next. with pve-kernel-4.2.6-1-pve: 4.2.6-29 on a separate VM.

spirit · Jan 3, 2016

by default,proxmox disable virtual vga card (vga:none) when xgva is enabled on pci passthough.
Until you setup manually a vgacard (std,vga,...).
I don't have any idea why this is change radeon detection...

Q-wulf · Jan 3, 2016

Any idea what else could try ? the fault definitely happens when the AMD-driver is initialised during install.

Any test i can perform (without the actual AMD chrimson driver ) to check if the GPU gets properly passed through ?

spirit · Jan 3, 2016

Q-wulf said:
Any idea what else could try ? the fault definitely happens when the AMD-driver is initialised during install.

Any test i can perform (without the actual AMD chrimson driver ) to check if the GPU gets properly passed through ?

Try ovmf.(maybe this need video card rom patching to support uefi)

spirit · Jan 3, 2016

also with ovmf, you can try to remove "machine: q35".

you can also ask help here:
https://www.redhat.com/archives/vfio-users/

Alex Williamson maintain this maling list and he's the vfio guru.

Q-wulf · Jan 3, 2016

Try with OMVF

Code:

pveversion -v
proxmox-ve: 4.1-28 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-2 (running version: 4.1-2/78c5f4a2)
pve-kernel-4.2.6-1-pve: 4.2.6-29

1060.conf

Code:

bios: ovmf
bootdisk: virtio0
cores: 6
cpu: kvm64
hostpci0: 02:00,x-vga=on,pcie=1
hostpci1: 00:1b.0
keyboard: de
machine: q35
memory: 8096
name: Win10-OMVF.local.lan
net0: virtio=32:62:64:62:39:38,bridge=vmbr0
numa: 0
ostype: win8
parent: WindowsUpdated
sata0: local:iso/Win10_1511_EnglishInternational_x64.iso,media=cdrom,size=3895734K
sata1: local:iso/virtio-win-0.1.110.iso,media=cdrom,size=55260K
scsihw: virtio-scsi-pci
smbios1: uuid=adb7a35f-ac72-491a-9ab6-a6c13269fbb8
sockets: 1
usb0: host=046d:c50d
usb1: host=0566:3107
virtio0: ceph_r2:vm-1060-disk-1,cache=writeback,iothread=on,size=300G

spirit said:
also with ovmf, you can try to remove "machine: q35".

Throws this error:

Code:

TASK ERROR: q35 machine model is not enabled at /usr/share/perl5/PVE/QemuServer.pm line 2768.

With "machine: q35" the monitor just freezes. It boots into windows in the background (as i can access network shares after a while)

Any ideas ?

ps.:
I shall see about getting a UEFI bios from MSI.
In the meantime i still have a AMD HD 7970 i can test later.
to Rule out that graphics card being the culprit (8GB variant) as i have seen lots of examples of people passing the GPU (4GB version) on a multitude on Systems (arch, fedora, ubuntu)

spirit · Jan 4, 2016

Throws this error:

Code:

TASK ERROR: q35 machine model is not enabled at /usr/share/perl5/PVE/QemuServer.pm line 2768.

also remove ",pcie" from hostpci if you disable q35.

Q-wulf · Jan 4, 2016

spirit said:
Throws this error:

Code:

TASK ERROR: q35 machine model is not enabled at /usr/share/perl5/PVE/QemuServer.pm line 2768.

also remove ",pcie" from hostpci if you disable q35.

Tried this just now with the

ATI x290 8GB

Does not throw that error anymore, but still freezes the Monitor before actually booting the win10-VM before even installing a GPU driver. (in Seabios it started freezing once you installed the GPU-Driver and/or booted into a win10-Vm that had the driver installed).

Will test with 7970 now and report back results. (this should definitely work from a GPU-being-the-issue pov as i have seen a multitude of threads regarding the specific GPU i'm using)

Q-wulf · Jan 4, 2016

I tried some more Combinations.
Red did not work.
Green did work.

7970 (TahitiXT) on OMVF Win10 results:

Without "machine: q35" --> Monitor goes straight to sleep. Only way to recover is to "reboot host"
With "machine: q35" --> Monitor goes straight to sleep. Only way to recover is to "reboot host"

7970 (TahitiXT) on SeaBios Win10 results:

Monitor Changes to Windows Boot. Boots into windows 10 just fine. Can change Screen resolution (indicates Windows installed a driver). Does not however recognise the Display adapter (all entries are <unavailable>).
First Boot: Crashes after 90 seconds with the following blue-screen: "System_thread_Exception_not_handled (atikmdad.sys)"
- Can be recovered by doing a VM reset via GUI/qm.
Second Boot: Amd Autodetect utility does not recognise the GPU. "does not crash" for 15 Minutes
- Install of Minmal Chrimson Edition 15.12 (Display Driver + Runtimes only) results in crash on step "currently installing: AMD Display Driver"
- Bluescreen "System_thread_Exception_not_handled (atikmdad.sys)"
- Can be recovered by doing a VM reset via GUI/qm.
- Event Viewer "System" shows
  - The previous System Shutdown was unexspected
  - Driver Management concluded the process ti install driver amdkmafd.inf _amd64 .... with following status: 0x0

7970 (TahitiXT) on SeaBios Ubuntu 15.10 results :

Code:

machine: q35
hostpci0: 02:00,x-vga=on,pcie=1

Fresh install updated + steam installed.

Try one:

set Driver via "Software & updates" -> "Additional Drivers" --> Using Xorg X server AMD/ATI display driver wrapper
- detects make and model properly. (HD 7970 IceQ X2)
  - Test with Game "Cities in Motion 2"
    - no crash.
    - Low performance (as expected)
set Driver via "Software & updates" -> "Additional Drivers" --> Using fglrx (propriatary)
- Goes into endless loop of Boot to Ubuntu Account selection -> Reset.
- Monitor does NOT go into deep sleep and i can just " qm stop" and rollback Snapshot

Pretty sure its not the GPU that is the issue.

Any more Ideas ?

Q-wulf · Jan 8, 2016

based on this:
https://forum.proxmox.com/threads/p...oken-in-several-ways.24178/page-2#post-127856

i have checked if i might have something else on the iommu_groups

Code:

ls /sys/kernel/iommu_groups/20/devices
0000:02:00.0 <-- VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X]
0000:02:00.1 <-- Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]

Code:

ls /sys/kernel/iommu_groups/10/devices
0000:00:1b.0  <-- 00:1b.0 Audio device: Intel Corporation C600/X79 series chipset High Definition Audio Controller (rev 06)

I guess that is not my issue then ?

Guess

spirit · Jan 8, 2016

Q-wulf said:
I guess that is not my issue then ?

Guess

It's ok for you, you have an dedicated iommu group for your vga card

spirit · Jan 8, 2016

Q-wulf said:
Any more Ideas ?

can you test last packages from pvetest repository ?
I have made a modification to disable hyper-v enlightment for windows machines when vga passthrough is done.
It's known to have problem with nvidia geforce driver, I'm not sure about radeon.

spirit · Jan 8, 2016

maybe this could help:
https://www.redhat.com/archives/vfio-users/2015-October/msg00119.html

user describe how he's able to install catalyst driver. (he had the same bug than you)

"
I also had problems with installing Catalyst higher than 14.12 for my r9 290 on both win 8.1 and 10.
BSOD and rebooting cycle after installing Catalyst.
The solution was to add ioh3420 and pass the GPU through it.
I using qemu from commandline with following parameters for GPU:

-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0 \

Afterwards win10 automatically installed the latest non-beta drivers without problems.

"

I can make a test qemu-server package if you want

spirit · Jan 8, 2016

I have also see some notes for ovmf:

I seem that x-vga=on option should be removed.

Q-wulf · Jan 8, 2016

I shall test them both after i get me some more coffee

I opened the settings for resolution and disabled the output on the qxl
gpu, which left me with no output.

What does he mean by "disabled the output on the qxl" ??
As in how did he do that ?

spirit · Jan 8, 2016

Q-wulf said:
I shall test them both after i get me some more coffee

What does he mean by "disabled the output on the qxl" ??
As in how did he do that ?

qxl is spice vga card.

AMD gpu passthrough issue - windows 10 - steps included

Renowned Member

Distinguished Member

Renowned Member

Distinguished Member

Renowned Member

Distinguished Member

Renowned Member

Distinguished Member

Distinguished Member

Renowned Member

Distinguished Member

Renowned Member

Renowned Member

Renowned Member

Distinguished Member

Distinguished Member

Distinguished Member

Distinguished Member

Renowned Member

Distinguished Member