mmm, for this one, it should already be the config of proxmox with q35 (with a different command line).
Can you give me the kvm process command line, when q35 enabled ?
AMD gpu passthrough issue - windows 10 - steps included
- Thread starter Q-wulf
- Start date
mmm, for this one, it should already be the config of proxmox with q35 (with a different command line).
Can you give me the kvm process command line, when q35 enabled ?
I know it is spice, but what does he specifically mean by the steps he described above ??
I assume the "I opened the settings for resolution" part means in windows rightclick -> display settings -> advanced settings
How did he acheive this part: "and disabled the output on the qxl gpu,". Thats what i am not understanding.
Edit:
Will do when i get home and can test it (soon TM)
also found this (it is a follow up of the thread you posted) :
https://www.redhat.com/archives/vfio-users/2015-October/msg00120.html
How'd one go about doing this via VMID.conf ??
https://www.redhat.com/archives/vfio-users/2015-October/msg00120.html
How'd one go about doing this via VMID.conf ??
This one is the same than your qemu command line
-device vfio-pci,host=02:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-vga=on,multifunction=on
(machine:q35 + hostpci: ...,pcie=1 )
(from the example, bus=root.1 = bus=ich9-pcie-port-1).
So, it's ok for this point.
Another thing to test, with
bios: ovmf.
hostpci: .....,x-vga=on
can you edit
/usr/share/perl5/PVE/QemuServer.pm
and change
Code:
my $xvga = $d->{'x-vga'} && $d->{'x-vga'} eq 'on' ? ",x-vga=on" : "";
if ($xvga && $xvga ne '') {
push @$cpuFlags, 'kvm=off';
$vga = 'none';
$nohyperv = 1;
}by
Code:
my $xvga = $d->{'x-vga'} && $d->{'x-vga'} eq 'on' ? ",x-vga=on" : "";
if ($xvga && $xvga ne '') {
push @$cpuFlags, 'kvm=off';
$vga = 'none';
$nohyperv = 1;
if ($conf->{bios} && $conf->{bios} eq 'ovmf') {
$xvga='';
}
}then start the vm with "qm start VMID".
(or with gui, but you need to restart pvedaemon service first)
for the mailing example:
This is the same than proxmox config
Code:
qemu-system-x86_64 \
-name starscream \
-enable-kvm \
-m 8192 \
-cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF-pure-efi.fd \
-drive if=pflash,format=raw,file=/home/bchow/kvm/starscream/OVMF-starscream.fd \
-drive file=/home/bchow/kvm/starscream/starscream.img,if=virtio,format=raw \
-drive file=/home/bchow/Win10_English_x64.iso,format=raw,id=win10cd,if=none -device ide-cd,bus=ide.1,drive=win10cd \
-drive file=/home/bchow/virtio-win-0.1.110.iso,format=raw,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \
-device vfio-pci,host=01:00.0 \
-device vfio-pci,host=01:00.1 \
-usb -usbdevice host:24f0:0141 \
-usb -usbdevice host:046d:c00e \
-usbdevice tablet \
-vga qxl \
-boot menu=on \
-show-cursorThis is the same than proxmox config
Code:
bios: ovmf
-> no machine config q35
hostpci : 01:00, xvga=on (+my previous patch ) -> no pcie=1 here
vga: qxlthe original actually reads:
Code:
my $xvga = $d->{'x-vga'} && $d->{'x-vga'} eq 'on' ? ",x-vga=on" : "";
if ($xvga && $xvga ne '') {
push @$cpuFlags, 'kvm=off';
$vga = 'none';
}$nohyperv = 1;
not present.
Edit: if i actually add the "$nohyperv = 1;" line in, after a reboot my Proxmox-GUI does not come back up. When i restore a backup of /usr/share/perl5/PVE/QemuServer.pm it does.
If i apply your complete patch i end up with
Code:
nano /usr/share/perl5/PVE/QemuServer.pm
root@node:~# qm start 1060
Global symbol "$nohyperv" requires explicit package name at /usr/share/perl5/PVE/QemuServer.pm line 2779, <DATA> line 751.
Compilation failed in require at /usr/share/perl5/PVE/CLI/qm.pm line 19, <DATA> line 751.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/CLI/qm.pm line 19, <DATA> line 751.
Compilation failed in require at /usr/sbin/qm line 6, <DATA> line 751.
BEGIN failed--compilation aborted at /usr/sbin/qm line 6, <DATA> line 751.
Last edited:
For the record i am on "pve-no-subscription" (up to date) with the following applied (as per your suggestion)
ftp://download1.proxmox.com/debian/dists/jessie/pvetest/binary-amd64/pve-kernel-4.2.6-1-pve_4.2.6-29_amd64.deb
ftp://download1.proxmox.com/debian/dists/jessie/pvetest/binary-amd64/pve-kernel-4.2.6-1-pve_4.2.6-29_amd64.deb
use pvetest repository
"deb http://download.proxmox.com/debian jessie pvetest"
or if you use no-subscription, you can remove "nohyperv"
(But it's better to test pvetest, we have already made some optiimisations (pve-kernel,qemu-server packages) no yet available in pve-no-subscription)
"deb http://download.proxmox.com/debian jessie pvetest"
or if you use no-subscription, you can remove "nohyperv"
Code:
my $xvga = $d->{'x-vga'} && $d->{'x-vga'} eq 'on' ? ",x-vga=on" : "";
if ($xvga && $xvga ne '') {
push @$cpuFlags, 'kvm=off';
$vga = 'none';
if ($conf->{bios} && $conf->{bios} eq 'ovmf') {
$xvga='';
}
}(But it's better to test pvetest, we have already made some optiimisations (pve-kernel,qemu-server packages) no yet available in pve-no-subscription)
Last edited:
So, i am running tests with todays update to pve-no-subscription and based on the rewritten wiki here:
https://pve.proxmox.com/wiki/Pci_passthrough
First run
AMD 7970 3GB
Ignoring the following advice:
test 1.1: Fail
GPU Seabios PCI PASSTHROUGH
(as per wiki)
GPU Seabios PCI EXPRESS PASSTHROUGH
(as per wiki)
test 1.2: Fail
GPU OVMF PCI PASSTHROUGH
(as per wiki)
After this crash it seems my vNas is really, really, really slow on network through put until the host is restarted. Before 250 MB/s -> crash ->2 MB/s -> reboot --> 250 MB/s
So reboot was in order.
Test 1.3: Fail
GPU OMVF PCI EXPRESS PASSTHROUGH
(as per wiki)
Seems like i better go and get myself a UEFI cable GPU Bios...
Note to self: Next time choose a smaller disk for Tests - 2 TB vDisk rollback on a EC-Pool takes ages....
https://pve.proxmox.com/wiki/Pci_passthrough
First run
AMD 7970 3GB
Ignoring the following advice:
$ lspci -n -s 1:
01:00.0 0300: 10de:1381 (rev a2)
01:00.1 0403: 10de:0fbc (rev a1)
The Vendorevice IDs for my GPU and audio functions are therefore 10de:1381, 10de:0fbc.
Then, create a file
echo "options vfio-pci ids=10de:1381,10de:0fbc" > /etc/modprobe.d/vfio.conf
test 1.1: Fail
GPU Seabios PCI PASSTHROUGH
(as per wiki)
- 1st. Boot: Starts into windows fine.
- After a while BSOD with the error message: SYSTEM_Thread_Exception_NOT_Handled (atikmdag.sys). That is the Windows GPU driver .
- Monitor stays/GPu stays responsive (does not free into sleep mode)
- Seems like other VM's on the network loose network connections.
- 2nd Boot:
- Install of 15.7.1 ATI driver (Driver Only - no CCC)
- SYSTEM_Thread_Exception_NOT_Handled (atikmdag.sys) after switch to ATI driver by installer.
- Time to reset system to previous snapshot.
- Note: Proxmox would not restart afterwards without hitting the Hard-reset button.
GPU Seabios PCI EXPRESS PASSTHROUGH
(as per wiki)
- 1st. Boot: Starts into windows fine.
- After a while BSOD with the error message: SYSTEM_Thread_Exception_NOT_Handled (atikmdag.sys). That is the Windows GPU driver .
- Monitor stays/GPu stays responsive (does not free into sleep mode)
- Seems like other VM's on the network loose network connections.
test 1.2: Fail
GPU OVMF PCI PASSTHROUGH
(as per wiki)
- 1st. try - Freezes Monitor before going to sleep.
- Throws error message:
Code:
tart failed: command '/usr/bin/systemd-run --scope --slice qemu --unit 1060 -p 'KillMode=none' -p 'CPUShares=1000' /usr/bin/kvm -id 1060 -chardev 'socket,id=qmp,path=/var/run/qemu-server/1060.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/1060.vnc,x509,password -pidfile /var/run/qemu-server/1060.pid -daemonize -smbios 'type=1,uuid=adb7a35f-ac72-491a-9ab6-a6c13269fbb8' -drive 'if=pflash,format=raw,readonly,file=/usr/share/kvm/OVMF-pure-efi.fd' -drive 'if=pflash,format=raw,file=/tmp/1060-OVMF_VARS-pure-efi.fd' -name Win10-OMVF.datamile.lan -smp '6,sockets=1,cores=6,maxcpus=6' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000' -vga none -no-hpet -cpu 'kvm64,kvm=off,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce' -m 8096 -k de -object 'iothread,id=iothread-virtio0' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -readconfig /usr/share/qemu-server/pve-usb.cfg -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'vfio-pci,host=02:00.0,id=hostpci0,bus=pci.0,addr=0x10' -device 'usb-host,vendorid=0x046d,productid=0xc50d' -device 'usb-host,vendorid=0x0566,productid=0x3107' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:c18a2cf727d' -drive 'file=/dev/rbd/P2__HDD_OSD_REPLICATED_R2/vm-1060-disk-1,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,iothread=iothread-virtio0,bootindex=100' -device 'ahci,id=ahci0,multifunction=on,bus=pci.0,addr=0x7' -drive 'if=none,id=drive-sata1,media=cdrom,aio=threads' -device 'ide-drive,bus=ahci0.1,drive=drive-sata1,id=sata1' -drive 'if=none,id=drive-sata0,media=cdrom,aio=threads' -device 'ide-drive,bus=ahci0.0,drive=drive-sata0,id=sata0' -netdev 'type=tap,id=net0,ifname=tap1060i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=32:62:64:62:39:38,netdev=net0,bus=pci.0,addr=0x12,id=net0' -rtc 'driftfix=slew,base=localtime' -global 'kvm-pit.lost_tick_policy=discard'' failed: got timeoutAfter this crash it seems my vNas is really, really, really slow on network through put until the host is restarted. Before 250 MB/s -> crash ->2 MB/s -> reboot --> 250 MB/s
So reboot was in order.
Test 1.3: Fail
GPU OMVF PCI EXPRESS PASSTHROUGH
(as per wiki)
- 1st. Boot: freezes Monitor - seems to boot into windows in background.
- shows error messages on start:
Code:
# qm start 1060
Use of uninitialized value $kvmver in pattern match (m//) at /usr/share/perl5/PVE/QemuServer.pm line 6394.
Use of uninitialized value $current_major in numeric ge (>=) at /usr/share/perl5/PVE/QemuServer.pm line 6400.
/dev/rbd0
Running as unit 1060.scope.
libust[13073/13073]: Error: Error cancelling global ust listener thread: No such process (in lttng_ust_exit() at lttng-ust-comm.c:1592)
libust[13073/13073]: Error: Error cancelling local ust listener thread: No such process (in lttng_ust_exit() at lttng-ust-comm.c:1601)Seems like i better go and get myself a UEFI cable GPU Bios...
Note to self: Next time choose a smaller disk for Tests - 2 TB vDisk rollback on a EC-Pool takes ages....
you can ignore this, It's just a warning from machine:q35. (I'll send a fix)
This is ceph related, it's a bug in current hammer rbd driver. It's already fixed in infernalis release.
you can ignore it too, they are no impact.
could be great too see if it's work with uefi rom. (with display working when it's booting)
That's pretty strange ... your host don't loose network connection ? are your sure that network card is not is same iommu group ?
At this point i am not sure of anything. It seems that i have broken the vDisk(s) on some of my VM's and i am not yet sure if it is related to this test, or related to the last update. Didn't have time to look into it yet.
edit1: virtio vdisks have a broken filesystem. Happened on the Windows-10 VM i was working on and on a OMV VM that uses XFS internally.
edit2: it is not on a similar iommu group.
The network connection drop and then "slow performance until node reboot" only happens when the GPU passthrough failed. I had it on both omvf tests.
edit3: still not sure if related to the GPU passthrough, the latest Proxmox update, or something completely different and random.
Last edited:
Can confirm its related to neither Proxmox Update, nor GPU passthrough and totally random in origin.
So i use 6 small sized SSD's on this node and recently added 4 HDD's. just old stuff i had laying around (altho i had tested em), since i needed some more more storage space for a VM I was testing. I use'd em as a single node ceph pool (SSD cache-pool over EC_HDD pool) turns out that 2 of those HDD's have developed
Once i plugged the disks into my Zabbix Monitoring system at home it became apparent from the long Smart tests that those drives were faulty.
i'll probably have to redo all the tests i made above once i get a couple new disks. Hopefully by then i have the MSI ATI card with UEFI Bios, so i can test that first.
so .. stay ... tuned .. (goes into corner with a picard facepalm sticker)
Did you solve this? I've just posted a similar thread as I have exactly the same problem. Didn't find this one on first search.
yes and no.
No, since i have never been able to pass the AMD GPU through to the windows VM.
Yes, in the sense that i found a workaround that works for me.
Basically i created a windows VM and passed a whole SSD through to it. On that SSD i installed the boot loader and the OS:
I can run the windows VM fine the regular way via noVNC/console. In that random case i want to use the GPU for e.g. Gaming i reboot into windows natively.
It only makes up 3-5% of my regular usage of this system - which runs backup services for my primary Proxmox-system(s) at home - so i can live with it, definitely more economical and less time consuming then trying to get this working
No, since i have never been able to pass the AMD GPU through to the windows VM.
Yes, in the sense that i found a workaround that works for me.
Basically i created a windows VM and passed a whole SSD through to it. On that SSD i installed the boot loader and the OS:
I can run the windows VM fine the regular way via noVNC/console. In that random case i want to use the GPU for e.g. Gaming i reboot into windows natively.
It only makes up 3-5% of my regular usage of this system - which runs backup services for my primary Proxmox-system(s) at home - so i can live with it, definitely more economical and less time consuming then trying to get this working
Last edited:
OK. That wouldn't work for me as I need this server to be on 24/7 running other VM's.
I've made some progress actually...
There's no UEFI BIOS available for my card (AMD Radeon 7750) however I've found on some Mac forums how to edit the firmware. So I've created a hybrid UEFI / Legacy BIOS for the card which I flashed on it. So I now have a UEFI BIOS which gives me a screen during boot up on UEFI so it works the same as SeaBIOS there. However low and behold this lets me load the Radeon drivers without it then killing the machine so this is far better.
Clearly there's a catch of course because otherwise that's it all solved. The problem is trying DOTA 2 on it I might get 5 - 15 mins or so before the graphics drivers crash and restart (TLR). This stops DOTA so you have to force quit it. I cannot find a solution to this currently, I've tried a few different drivers, added in things like:
options kvm ignore_msrs=1
but nothing seems to solve this. So somewhat out of ideas but currently plan to spin up ESXi again to test if it's stable on that. Just to see if it's an AMD as a whole issue or a KVM issue. The problem with ESXi is I cannot pass through any USB devices on my machine plus it's Mini ITX so no room to put in a USB controller card. So the solution here is USB over Ethernet.
As you say, maybe this is too much hassle but I'm on a mission and not quite ready to give up
I've made some progress actually...
There's no UEFI BIOS available for my card (AMD Radeon 7750) however I've found on some Mac forums how to edit the firmware. So I've created a hybrid UEFI / Legacy BIOS for the card which I flashed on it. So I now have a UEFI BIOS which gives me a screen during boot up on UEFI so it works the same as SeaBIOS there. However low and behold this lets me load the Radeon drivers without it then killing the machine so this is far better.
Clearly there's a catch of course because otherwise that's it all solved. The problem is trying DOTA 2 on it I might get 5 - 15 mins or so before the graphics drivers crash and restart (TLR). This stops DOTA so you have to force quit it. I cannot find a solution to this currently, I've tried a few different drivers, added in things like:
options kvm ignore_msrs=1
but nothing seems to solve this. So somewhat out of ideas but currently plan to spin up ESXi again to test if it's stable on that. Just to see if it's an AMD as a whole issue or a KVM issue. The problem with ESXi is I cannot pass through any USB devices on my machine plus it's Mini ITX so no room to put in a USB controller card. So the solution here is USB over Ethernet.
As you say, maybe this is too much hassle but I'm on a mission and not quite ready to give up
So just as an update but ESXi is exactly the same, drivers crash in DOTA so seems it could just be AMD drivers / hardware not working well with this or this particular motherboard (SuperMicro server board).
Probably going to have to give up on this
Probably going to have to give up on this
Hello; I hope I am not too late, but I have successfully passed-through an AMD r9 FURY with drivers installed
My problem also was: during the driver install, resolution/VGA reset I'd get all sorts of atikmpag.sys BSOD.
HARDWARE:
motherboard: ASUS X99-E WS USB3.1
cpu: Intel Xeon e5-1650
gpu: Sapphire Tri-X Radeon R9 Fury
bios: seabios
/etc/pve/qemu-server/100.conf:
Please take note of:
I do not use the "pcie=1" attribute: I am passing it through as a normal pci device.
Romfiles can be found at techpowerup.com/vgabios/ and must be placed in /usr/share/kvm/
NOTES:
I just did a fresh windows install onto an SSD (.raw) with the latest (Crimson 17.7.2) drivers and the VM survived every reboot/update
My problem also was: during the driver install, resolution/VGA reset I'd get all sorts of atikmpag.sys BSOD.
HARDWARE:
motherboard: ASUS X99-E WS USB3.1
cpu: Intel Xeon e5-1650
gpu: Sapphire Tri-X Radeon R9 Fury
bios: seabios
/etc/pve/qemu-server/100.conf:
Code:
balloon: 0
boot: dcn
bootdisk: virtio0
cores: 8
cpu: host
hostpci0: 09:00,romfile=fury.rom,x-vga=on
ide0: local:iso/virtio-win-0.1.126.iso,media=cdrom,size=152204K
ide2: local:iso/windows10.iso,media=cdrom
machine: q35
memory: 16384
name: win10pro
net0: e1000=CA:75:A2:2F:2C:D5,bridge=vmbr0
numa: 0
ostype: win10
parent: final
scsihw: virtio-scsi-pci
smbios1: uuid=2ccd6b8c-17ce-434d-b824-56ce4fb03c1b
sockets: 1
unused0: storage:vm-100-disk-2
usb0: host=3-10.4
virtio0: storage:vm-100-disk-1,size=32GPlease take note of:
Code:
hostpci0: 09:00,romfile=fury.rom,x-vga=onI do not use the "pcie=1" attribute: I am passing it through as a normal pci device.
Romfiles can be found at techpowerup.com/vgabios/ and must be placed in /usr/share/kvm/
NOTES:
- VFIO successfully binds the GPU so "pci-stub" was not needed
- GPU unresponsive using UEFI (as pci and pcie). VNC hangs too. Only using SeaBIOS would the GPU power on
- I installed/updated Windows through VNC-console. After that was all setup, i shutdown the VM, add "hostpci0: ..." then fire up VM. You can exclude "x-vga=on" and still work in VNC if you need to toy around with the drivers
I just did a fresh windows install onto an SSD (.raw) with the latest (Crimson 17.7.2) drivers and the VM survived every reboot/update
Last edited: