GPU passthrough, possible with my hardware?

[mdurkin]

Hi All,
I've recently installed proxmox, moved to pvetest, updated, and installed windows8.1 in a vm, config below.
My hardware supports Vtd, IOMMU etc, and I have previously run xen 4.4 passing through a number of devices - soundcard, usb chipsets, and the onboard intel HD4600 GPU.
Performance with XEN is great, but I was really attracted to kvm, and proxmox, for other reasons, such as the management console, and to address a few minor problems I've had with xen.

Config here:

bootdisk: virtio0
cores: 2
ide0: local:iso/virtio-win-0.1-81.iso,media=cdrom,size=72406K
ide2: local:iso/win8.iso,media=cdrom
memory: 1984
name: Windows8
net0: virtio=B6:37:0C:BC:1D:A0,bridge=vmbr0
ostype: win8
sockets: 1
virtio0: local:100/vm-100-disk-1.qcow2,format=qcow2,cache=writeback,size=70G
machine: q35
#hostpci0: 00:02.0,pcie=1,driver=vfio
hostpci0: 00:02.0,x-vga=on,pcie=1,driver=vfio
#hostpci0: 00:02.0

the above config won't start, with the following errors:

qm start 100
kvm: -device vfio-pci,host=00:02.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,x-vga=on: vfio: Device does not support requested feature x-vga
kvm: -device vfio-pci,host=00:02.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,x-vga=on: vfio: failed to get device 0000:00:02.0
kvm: -device vfio-pci,host=00:02.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,x-vga=on: Device initialization failed.
kvm: -device vfio-pci,host=00:02.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,x-vga=on: Device 'vfio-pci' could not be initialized
start failed: command '/usr/bin/kvm -id 100 -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/100.vnc,x509,password -pidfile /var/run/qemu-server/100.pid -daemonize -name Windows8 -smp 'sockets=1,cores=2' -nodefaults -boot 'menu=on' -vga std -no-hpet -cpu 'kvm64,kvm=off,hv_spinlocks=0xffff,hv_relaxed,+lahf_lm,+x2apic,+sep' -k en-gb -m 1984 -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=00:02.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,x-vga=on' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:c3ff81b4e81b' -drive 'file=/var/lib/vz/template/iso/win8.iso,if=none,id=drive-ide2,media=cdrom,aio=native' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/var/lib/vz/images/100/vm-100-disk-1.qcow2,if=none,id=drive-virtio0,format=qcow2,cache=writeback,aio=native' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -drive 'file=/var/lib/vz/template/iso/virtio-win-0.1-81.iso,if=none,id=drive-ide0,media=cdrom,aio=native' -device 'ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=201' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=B6:37:0C:BC:1D:A0,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -machine 'type=q35' -global 'kvm-pit.lost_tick_policy=discard'' failed: exit code 1

if I remove the x-vga=on parameter, the vm does boot, makes it to windows which sees an additional display adaptor, but is unstable and crashes very quickly (too quickly to unstall the drivers).

versions:
pveversion -v
proxmox-ve-2.6.32: 3.2-132 (running kernel: 3.10.0-3-pve)
pve-manager: 3.2-18 (running version: 3.2-18/e157399a)
pve-kernel-2.6.32-29-pve: 2.6.32-126
pve-kernel-3.10.0-3-pve: 3.10.0-11
pve-kernel-2.6.32-31-pve: 2.6.32-132
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-1
pve-cluster: 3.0-14
qemu-server: 3.1-28
pve-firmware: 1.1-3
libpve-common-perl: 3.0-19
libpve-access-control: 3.0-15
libpve-storage-perl: 3.0-21
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-7
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.1-1
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.2-1

I had assumed, rightly or wrongly, that as I had this working perfectly well in xen that it ought to be possible in kvm, but maybe not. Perhaps graphics card support is much more limited. That said, it feels close.

If anyone has any ideas on this, I'd really appreciate them!

Many thanks,
Matthew

 

[aderumier]

Hi, what gpu model do you try to passthrough ?

from https://bbs.archlinux.org/viewtopic.php?id=162768

" AMD RADEON 5xxx, 6xxx, 7xxx and NVIDIA GEFORCE 7, 8, 4xx, 5xx, 6xx, 7xx have been reported working with this, passing though an intel IGD'S WONT WORK."

 

[mdurkin]

Hi, what gpu model do you try to passthrough ?

from https://bbs.archlinux.org/viewtopic.php?id=162768

" AMD RADEON 5xxx, 6xxx, 7xxx and NVIDIA GEFORCE 7, 8, 4xx, 5xx, 6xx, 7xx have been reported working with this, passing though an intel IGD'S WONT WORK."

Yes - I'm trying to passthrough the intel IGD. Works well on Xen. Why not kvm? The device does get passed through, the proxmox console login disappears so the hardware is mapped into the vm, but it doesn't work. Seems like the mechanism works, but something's not quite there.
Is there a specific reason that IGDs aren't supported?

Matt

 

[optim]

I'm in the same boat:

vfio: Device does not support requested feature x-vga

I'm using a motherboard (GA-Z77X-UD3H, i3770) and video card combo (HD4670) that works perfectly with XenServer v6.2 SP1 passthrough. Thinking maybe it was something to do with the video card I also tried 4 others (2 nvidia, 2 amd/ati), all return the same x-vga not supported error. I also blacklisted the noveau and radeon drivers in my grub setup. It is so close to working, and I know the hardware works as I've passed through SATA controllers and USB without issue.

Granted, VGA is much harder to pass though, but I'm sure we are almost there as hours of googling has taught me that KVM can do it and much of the hard work has been done already as per the pve-devel list. I'm just not smart enough to figured out what I am missing!

BTW, thanks to all those working to get passthrough into Proxmox. Once it works it will be a killer platform for the home user to consolidate server and desktop roles to save heat, power and hardware expense.

Daniel

 

[spirit]

Hi,

I just notice that pve-kernel 3.10 from pvetest repostory don't have vfio xvga module enabled !
It's already ok in proxmox git, but not pvetest repo.

I can build the kernel for you if you want

 

[mcflym]

Hi,

x-vga is necessary to passthrough a gpu? So please release it to the pvetest repository

thanks!

 

[dietmar]

x-vga is necessary to passthrough a gpu? So please release it to the pvetest repository

Just uploaded a new version to pvetest.

 

[mdurkin]

Brill! So I can now boot with x-vga set, load windows, install the HD4600 drivers. But then the device shows with a yellow exclamation and says 'This device cannot find enough free resources that it can use'

so not looking good with intel IGD...

 

[mdurkin]

another thing I noticed, I don't seem to be able to pass the parameter "options kvm allow_unsafe_assigned_interrupts=1" in /etc/modprobe.d/interrupts.conf
I get: proxmox kernel: [ 12.589359] kvm: Unknown parameter `allow_unsafe_assigned_interrupts'
in my log. also a
modinfo kvm
filename: /lib/modules/3.10.0-3-pve/kernel/arch/x86/kvm/kvm.ko
license: GPL
author: Qumranet
srcversion: B24E22D8ACB39E9477759A5
depends:
intree: Y
vermagic: 3.10.0-3-pve SMP mod_unload modversions
parm: ignore_msrs:bool
parm: min_timer_period_us:uint
parm: tsc_tolerance_ppm:uint


doesn't show as an available parameter. I was expecting to have to set this parameter. I have tried adding:


GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1"


I have tried adding
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1"
to my /etc/default/grub - I can see it is passed at boot, but am unsure if it actually works.
just scrabbling around for anything else that might get this thing working!!

 

[mdurkin]

anyone?

 

[spirit]

anyone?

Hi,reading the archlinux forum,
https://bbs.archlinux.org/viewtopic.php?id=162768&p=84

seem that we need

a kernel patch:
https://lkml.org/lkml/2014/5/9/517

and boot option : i915.enable_hd_vgaarb=1

 

[XueSheng]

I have tried adding
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1"
to my /etc/default/grub - I can see it is passed at boot, but am unsure if it actually works.
just scrabbling around for anything else that might get this thing working!!

vfio_iommu_type1 is a module. You'll have to add a module parameter:

# cat /etc/modprobe/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

 

[mdurkin]

thanks for explaining, very useful. I have passed this, though am not sure how to check it's actually taken and has an effect? I don't see any errors, and can see that the parameter is applicable through modinfo, so suspect it has taken. No change to being able to pass through the GPU thus far...

On the kernel patch - is there anyone who could patch the kernel? I've not done any kernel work before. This is looking so close and I would love to test with the patch applied.

Thanks,
Matt

 

[spirit]

thanks for explaining, very useful. I have passed this, though am not sure how to check it's actually taken and has an effect? I don't see any errors, and can see that the parameter is applicable through modinfo, so suspect it has taken. No change to being able to pass through the GPU thus far...

On the kernel patch - is there anyone who could patch the kernel? I've not done any kernel work before. This is looking so close and I would love to test with the patch applied.

Thanks,
Matt

I tried to apply the patch on kernel 3.10, but it' don't work.
So I think it's better to wait that redhat backport it.

 

[mdurkin]

how long would that take usually? How would I know? (sorry not very familiar with backports). I just tried the latest experimental from debian to see if had made it that far, but it hasn't...
thanks,
Matt

 

[mdurkin]

reading this thread:
http://patchwork.freedesktop.org/patch/25700/

I'm not convinced the patch has been accepted. Seems like the discussion ended without conclusion, though I'm not familiar with the process for patches being officially included or not. I've never built or patched a kernel before, but in theory, should the patch apply to a later kernel such as 3.16? I'm running 3.16-rc6 at the moment as that was the latest experimental debian.

 

[optim]

Just uploaded a new version to pvetest.

Sorry for the delay, was on vacation and just got a chance to test this.

When I use a fresh Proxmox setup with pvetest updates and pci passthrough customizations as per the wiki, on a Win 7 vm, I run into the same error as mdurkin (code 12 - not enough resources).

I also noticed that the standard vga adapter was still showing, so I started to tinker and see if I could get kvm to work with the card directly (not using the Proxmox GUI). To start me off, I figured I would copy the command Proxmox issues to start the vm (vmid: 100) by doing:

qm showcmd 100 >>  test.sh

This resulted in the following being pasted into "test.sh":

/usr/bin/kvm -id 100 -chardev socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait -mon chardev=qmp,mode=control -vnc unix:/var/run/qemu-server/100.vnc,x509,password -pidfile /var/run/qemu-server/100.pid -daemonize -smbios type=1,uuid=1bd57999-ac3e-4b60-9701-7ba2b37d277f -name TestPassthrough -smp sockets=1,cores=4 -nodefaults -boot menu=on -vga std -no-hpet -cpu kvm64,kvm=off,hv_spinlocks=0xffff,hv_relaxed,+lahf_lm,+x2apic,+sep -k en-us -m 8192 -readconfig /usr/share/qemu-server/pve-q35.cfg -device usb-tablet,id=tablet,bus=ehci.0,port=1 -device vfio-pci,host=01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-vga=on,multifunction=on -device vfio-pci,host=01:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:b4ee086fd8 -drive file=/var/lib/vz/template/iso/Windows_7_SP1_Integrated_-_X17-24395.iso,if=none,id=drive-ide2,media=cdrom,aio=native -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -device ahci,id=ahci0,multifunction=on,bus=pci.0,addr=0x7 -drive file=/var/lib/vz/images/100/vm-100-disk-1.qcow2,if=none,id=drive-sata0,format=qcow2,aio=native,cache=none -device ide-drive,bus=ahci0.0,drive=drive-sata0,id=sata0,bootindex=100 -netdev type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown -device e1000,mac=1E:C9:2F:C1:83:D5,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 -rtc driftfix=slew,base=localtime -machine type=q35 -global kvm-pit.lost_tick_policy=discard

The interesting part is the "-vga std" switch. I changed this to "-vga none", then ran the script. Low and behold, my IGPU had some video corruption, and the monitor connected to my passthorugh video card turned on and showed the "Starting Windows" log. Progress!

Unfortunately Win 7 blue-screens and I still haven't passed through my USB keyboard and mouse, but at least the basic passthrough is working. When I have more time to test, I will post if I make any further progress. But I figured I'd pass this info along in case it helps others get their configs working.

 

[mdurkin]

Sorry for the delay, was on vacation and just got a chance to test this.

When I use a fresh Proxmox setup with pvetest updates and pci passthrough customizations as per the wiki, on a Win 7 vm, I run into the same error as mdurkin (code 12 - not enough resources).

I also noticed that the standard vga adapter was still showing, so I started to tinker and see if I could get kvm to work with the card directly (not using the Proxmox GUI). To start me off, I figured I would copy the command Proxmox issues to start the vm (vmid: 100) by doing:

qm showcmd 100 >>  test.sh

This resulted in the following being pasted into "test.sh":

/usr/bin/kvm -id 100 -chardev socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait -mon chardev=qmp,mode=control -vnc unix:/var/run/qemu-server/100.vnc,x509,password -pidfile /var/run/qemu-server/100.pid -daemonize -smbios type=1,uuid=1bd57999-ac3e-4b60-9701-7ba2b37d277f -name TestPassthrough -smp sockets=1,cores=4 -nodefaults -boot menu=on -vga std -no-hpet -cpu kvm64,kvm=off,hv_spinlocks=0xffff,hv_relaxed,+lahf_lm,+x2apic,+sep -k en-us -m 8192 -readconfig /usr/share/qemu-server/pve-q35.cfg -device usb-tablet,id=tablet,bus=ehci.0,port=1 -device vfio-pci,host=01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-vga=on,multifunction=on -device vfio-pci,host=01:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:b4ee086fd8 -drive file=/var/lib/vz/template/iso/Windows_7_SP1_Integrated_-_X17-24395.iso,if=none,id=drive-ide2,media=cdrom,aio=native -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -device ahci,id=ahci0,multifunction=on,bus=pci.0,addr=0x7 -drive file=/var/lib/vz/images/100/vm-100-disk-1.qcow2,if=none,id=drive-sata0,format=qcow2,aio=native,cache=none -device ide-drive,bus=ahci0.0,drive=drive-sata0,id=sata0,bootindex=100 -netdev type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown -device e1000,mac=1E:C9:2F:C1:83:D5,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 -rtc driftfix=slew,base=localtime -machine type=q35 -global kvm-pit.lost_tick_policy=discard

The interesting part is the "-vga std" switch. I changed this to "-vga none", then ran the script. Low and behold, my IGPU had some video corruption, and the monitor connected to my passthorugh video card turned on and showed the "Starting Windows" log. Progress!

Unfortunately Win 7 blue-screens and I still haven't passed through my USB keyboard and mouse, but at least the basic passthrough is working. When I have more time to test, I will post if I make any further progress. But I figured I'd pass this info along in case it helps others get their configs working.

ooo! That sounds interesting. I was wondering how to see the whole kvm command string - most useful. I'll see if I have time to test this evening.
One thing worth trying, if you haven't already, is to disable passthrough, boot windows, disable the 'microsoft basic display adaptor' you'll find in device manager, then reboot with passthrough enabled. My wind 8.1 vm blue screens after a short while with both enabled. I don't really know what the other device is, but I had to do exactly the same in xen as well. fingers crossed...

 

[spirit]

Sorry for the delay, was on vacation and just got a chance to test this.

When I use a fresh Proxmox setup with pvetest updates and pci passthrough customizations as per the wiki, on a Win 7 vm, I run into the same error as mdurkin (code 12 - not enough resources).

I also noticed that the standard vga adapter was still showing, so I started to tinker and see if I could get kvm to work with the card directly (not using the Proxmox GUI). To start me off, I figured I would copy the command Proxmox issues to start the vm (vmid: 100) by doing:

qm showcmd 100 >>  test.sh

This resulted in the following being pasted into "test.sh":

/usr/bin/kvm -id 100 -chardev socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait -mon chardev=qmp,mode=control -vnc unix:/var/run/qemu-server/100.vnc,x509,password -pidfile /var/run/qemu-server/100.pid -daemonize -smbios type=1,uuid=1bd57999-ac3e-4b60-9701-7ba2b37d277f -name TestPassthrough -smp sockets=1,cores=4 -nodefaults -boot menu=on -vga std -no-hpet -cpu kvm64,kvm=off,hv_spinlocks=0xffff,hv_relaxed,+lahf_lm,+x2apic,+sep -k en-us -m 8192 -readconfig /usr/share/qemu-server/pve-q35.cfg -device usb-tablet,id=tablet,bus=ehci.0,port=1 -device vfio-pci,host=01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-vga=on,multifunction=on -device vfio-pci,host=01:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:b4ee086fd8 -drive file=/var/lib/vz/template/iso/Windows_7_SP1_Integrated_-_X17-24395.iso,if=none,id=drive-ide2,media=cdrom,aio=native -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -device ahci,id=ahci0,multifunction=on,bus=pci.0,addr=0x7 -drive file=/var/lib/vz/images/100/vm-100-disk-1.qcow2,if=none,id=drive-sata0,format=qcow2,aio=native,cache=none -device ide-drive,bus=ahci0.0,drive=drive-sata0,id=sata0,bootindex=100 -netdev type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown -device e1000,mac=1E:C9:2F:C1:83:D5,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 -rtc driftfix=slew,base=localtime -machine type=q35 -global kvm-pit.lost_tick_policy=discard

The interesting part is the "-vga std" switch. I changed this to "-vga none", then ran the script. Low and behold, my IGPU had some video corruption, and the monitor connected to my passthorugh video card turned on and showed the "Starting Windows" log. Progress!

Unfortunately Win 7 blue-screens and I still haven't passed through my USB keyboard and mouse, but at least the basic passthrough is working. When I have more time to test, I will post if I make any further progress. But I figured I'd pass this info along in case it helps others get their configs working.

Thanks for the report. I'll make a patch to disable vga when gpu passthrough is enabled

 

[mdurkin]

Just had a quick chance to test. The VM starts but windows doesn't boot.
Optim - have you made other settings to the proxmox install? Module parameters or grub params?
Work now, do probably not much chance to try again for a while...