GPU passthrough, possible with my hardware?

[optim]

mdurkin,

I had tried a lot of things before I posted above. In order to see what was the change that enabled the passthrough I spent some time this morning undoing the changes and testing again.

In my post above as I was trying many of the fixes detailed here: https://bbs.archlinux.org/viewtopic.php?id=162768

But what I did not realize when posting above was that I had tried the following previously, before actually running the vm (excerpt from above link):

Binding a device to vfio-pci

Assuming the kernel, qemu and seabios are built and working, lets bind some devices.
You can use this script to make life easier:

#!/bin/bash

modprobe vfio-pci

for dev in "$@"; do
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done

Save it as /usr/bin/vfio-bind
chmod 755 /usr/bin/vfio-bind

Bind the GPU:
vfio-bind 0000:07:00.0 0000:07:00.1

After performing the above steps (binding in my case, 01:00.0 and 01:00.1 + all my usb), starting the vm with the correct hostpci options enables the passthrough. My USB ports come alive in the VM and the new hardware wizard finds the Radeon card. Unfortunately, on reboot the vm crashes (no blue screen, at least that I can see, just reboots).

The only other change I made was to QemuServer.pm, I set MAX_HOSTPCI_DEVICES = 8 as I had 4 USB controllers to passthrough + 1 HDMI + HDMI Audio.

I'm still playing around, but I don't have a firm grasp of vfio or the intricacies of passthrough yet. Hopefully we can iron out the details and get passthrough stable under Proxmox as I'm looking to migrate my setup from Xen and this is the only feature I need to do that...

Daniel

 

[mdurkin]

unfortunately this doesn't seem to work for me. vfio-bind runs ok, but I'm not sure if it does anything. Certainly the proxmox login console still appears on the IGD, so possibly not. When I attempt to start the VM, the IGD is switched away from the proxmox console, as it always does, but windows won't boot with -vga none switch applied.
the thread you posted does a lot more though - the kernel patch, pci-stub. Did you do any of that? Assuming not?

 

[spirit]

mdurkin,

I had tried a lot of things before I posted above. In order to see what was the change that enabled the passthrough I spent some time this morning undoing the changes and testing again.

In my post above as I was trying many of the fixes detailed here: https://bbs.archlinux.org/viewtopic.php?id=162768

But what I did not realize when posting above was that I had tried the following previously, before actually running the vm (excerpt from above link):



After performing the above steps (binding in my case, 01:00.0 and 01:00.1 + all my usb), starting the vm with the correct hostpci options enables the passthrough. My USB ports come alive in the VM and the new hardware wizard finds the Radeon card. Unfortunately, on reboot the vm crashes (no blue screen, at least that I can see, just reboots).

The only other change I made was to QemuServer.pm, I set MAX_HOSTPCI_DEVICES = 8 as I had 4 USB controllers to passthrough + 1 HDMI + HDMI Audio.

I'm still playing around, but I don't have a firm grasp of vfio or the intricacies of passthrough yet. Hopefully we can iron out the details and get passthrough stable under Proxmox as I'm looking to migrate my setup from Xen and this is the only feature I need to do that...

Daniel

Hi,
Proxmox manage the unbind for you normally.
you can also pass the gpu (video+audio) with 01:00 (without dot).

Does it work for you with normal proxmox code ?

 

[spirit]

the thread you posted does a lot more though - the kernel patch, pci-stub. Did you do any of that? Assuming not?

Optim passthrough a radeon card I think, not an intel igp.
you need a kernel patch or a recent kernel (>3.16) I think to get it work with intel.

 

[optim]

I'm using the IGD to provide the Proxmox console only. I have a Radeon HD4670 that I am passing through to the Windows. In my bios settings, I have the IGD set to be the first card initialized when the system starts.

Like I said in the previous post, I had done a lot of tinkering, but I went back to a plain setup.

Here's the details on the changes to make it a little more concrete:

/etc/default/grub.conf

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on quiet"

lspci

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04)
00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04)
00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4)
00:1c.1 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 2 (rev c4)
00:1c.2 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 3 (rev c4)
00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4)
00:1c.5 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c4)
00:1c.6 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 7 (rev c4)
00:1c.7 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 (rev c4)
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04)
00:1f.0 ISA bridge: Intel Corporation Z77 Express Chipset LPC Controller (rev 04)
00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04)
00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV730 XT [Radeon HD 4670]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] RV710/730 HDMI Audio [Radeon HD 4000 series]
03:00.0 RAID bus controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01)
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 01)
05:00.0 USB controller: VIA Technologies, Inc. VL80x xHCI USB 3.0 Controller (rev 03)
06:00.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 41)
08:00.0 Ethernet controller: Qualcomm Atheros AR8151 v2.0 Gigabit Ethernet (rev c0)

/etc/pve/qemu-server/100.conf

bootdisk: sata0
cores: 4
ide2: local:iso/Windows_7_SP1_Integrated_-_X17-24395.iso,media=cdrom
memory: 8192
name: TestPassthrough
net0: e1000=1E:C9:2F:C1:83:D5,bridge=vmbr0
ostype: win7
parent: PreVideoPassthrough
sata0: local:100/vm-100-disk-1.qcow2,format=qcow2,size=32G
smbios1: uuid=1bd57999-ac3e-4b60-9701-7ba2b37d277f
sockets: 1
machine: q35
hostpci0: 01:00,x-vga=on,pcie=1,driver=vfio
hostpci1: 00:14.0,pcie=1,driver=vfio
hostpci2: 00:1a.0,pcie=1,driver=vfio
hostpci3: 00:1d.0,pcie=1,driver=vfio

lsmod | grep -i vfio

nothing appears

Now, here's what the sequence looks like from a cold boot:

qm start 100

The vm starts, but I do not get any video on the Radeon card.

qm stop 100

Yes, it's a hard shutdown but I'm testing and I have a snapshot of a clean Win7 vm to revert to anyway.

lsmod | grep -i vfio

vfio_pci               36578  0
vfio_iommu_type1       17636  0
vfio                   20885  2 vfio_iommu_type1,vfio_pci

So the vfio modules did get loaded when the vm was started.

qm showcmd 100 > test.sh
nano test.sh

I change the "-vga std" to "-vga none".

sh test.sh

The result:

kvm: -device vfio-pci,host=01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-vga=on,multifunction=on: vfio: error opening /dev/vfio/1: No such file or directory
kvm: -device vfio-pci,host=01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-vga=on,multifunction=on: vfio: failed to get group 1
kvm: -device vfio-pci,host=01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-vga=on,multifunction=on: Device initialization failed.
kvm: -device vfio-pci,host=01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-vga=on,multifunction=on: Device 'vfio-pci' could not be initialized

vfio-bind 0000:01:00.0 0000:01:00.1
sh test.sh

Boom. My Radeon video is alive and showing the "Starting Windows" logo.

I don't know what it is with the vfio-bind step, but it seems to be the key to enabling the passthrough. Without it the vfio devices don't seem to get enumerated.

Hopefully that helps clear up what I am doing. Sorry for all the code tags, but I wanted to be clear with what was being typed.

Daniel

 

[optim]

Proxmox manage the unbind for you normally. Does it work for you with normal proxmox code ?

No.

As I detailed in a post above, I need that step or for some reason it doesn't work.

I'm also noticing that the Radeon card does not reset when the VM is stopped. I killed the vm (qm stop) and I still have the Windows Recovery screen painted on the passthrough monitor. I know there is bus reset fixes and patches related to this in the 3.12 kernel, but not sure if these might have been backported to the 3.10 kernel, which I believe is the LTS kernel.

BTW, I am passing through the 01:00 and not 01:00.0 and 01:00.1. Thanks for the tips!

Daniel

 

[spirit]

I think the problem is that currently proxmox don't use -vga none.

copy-paste the kvm command line in test.sh, don't do the vfio unbind/bind.


can you try this package :

http://odisoweb1.odiso.net/qemu-server_3.0-18_amd64.deb


it'll do the "-vga none" if x-vga is enabled.

 

[optim]

It breaks the setup. When applied I can no longer start the VM's because of a perl compilation error.

root@pve-desktop:~# dpkg -i qemu-server_3.0-18_amd64.deb
dpkg: warning: downgrading qemu-server from 3.1-30 to 3.0-18
(Reading database ... 36647 files and directories currently installed.)
Preparing to replace qemu-server 3.1-30 (using qemu-server_3.0-18_amd64.deb) ...
Unpacking replacement qemu-server ...
Setting up qemu-server (3.0-18) ...
Processing triggers for pve-manager ...
Restarting PVE Daemon: pvedaemonregister method PVE::API2::Qemu/{vmid}/config - duplicate method definition
Compilation failed in require at /usr/share/perl5/PVE/API2/Nodes.pm line 32.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Nodes.pm line 32.
Compilation failed in require at /usr/share/perl5/PVE/API2.pm line 14.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2.pm line 14.
Compilation failed in require at /usr/bin/pvedaemon line 14.
BEGIN failed--compilation aborted at /usr/bin/pvedaemon line 14.
 (warning).
Restarting PVE Status Daemon: pvestatd.
Restarting PVE API Proxy Server: pveproxyregister method PVE::API2::Qemu/{vmid}/config - duplicate method definition
Compilation failed in require at /usr/share/perl5/PVE/API2/Nodes.pm line 32.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Nodes.pm line 32.
Compilation failed in require at /usr/share/perl5/PVE/API2.pm line 14.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2.pm line 14.
Compilation failed in require at /usr/bin/pveproxy line 22.
BEGIN failed--compilation aborted at /usr/bin/pveproxy line 22.
 (warning).
Restarting PVE SPICE Proxy Server: spiceproxyregister method PVE::API2::Qemu/{vmid}/config - duplicate method definition
Compilation failed in require at /usr/share/perl5/PVE/API2/Nodes.pm line 32.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Nodes.pm line 32.
Compilation failed in require at /usr/share/perl5/PVE/API2.pm line 14.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2.pm line 14.
Compilation failed in require at /usr/bin/spiceproxy line 16.
BEGIN failed--compilation aborted at /usr/bin/spiceproxy line 16.
 (warning).
Processing triggers for pve-firewall ...
Restarting Proxmox VE firewall: pve-firewall.
Processing triggers for man-db ...
root@pve-desktop:~#

 

[mdurkin]

All very interesting. I'm going to go back to Xen for the time being as I'm not sure if the kernel patch required for the IGD was accepted, and I don't really want to run on patched code. Perhaps in a month or so the patch might have been accepted. Thanks for all the information and help though...
Matt

 

[spirit]

It breaks the setup. When applied I can no longer start the VM's because of a perl compilation error.

root@pve-desktop:~# dpkg -i qemu-server_3.0-18_amd64.deb
dpkg: warning: downgrading qemu-server from 3.1-30 to 3.0-18
(Reading database ... 36647 files and directories currently installed.)
Preparing to replace qemu-server 3.1-30 (using qemu-server_3.0-18_amd64.deb) ...
Unpacking replacement qemu-server ...
Setting up qemu-server (3.0-18) ...
Processing triggers for pve-manager ...
Restarting PVE Daemon: pvedaemonregister method PVE::API2::Qemu/{vmid}/config - duplicate method definition
Compilation failed in require at /usr/share/perl5/PVE/API2/Nodes.pm line 32.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Nodes.pm line 32.
Compilation failed in require at /usr/share/perl5/PVE/API2.pm line 14.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2.pm line 14.
Compilation failed in require at /usr/bin/pvedaemon line 14.
BEGIN failed--compilation aborted at /usr/bin/pvedaemon line 14.
 (warning).
Restarting PVE Status Daemon: pvestatd.
Restarting PVE API Proxy Server: pveproxyregister method PVE::API2::Qemu/{vmid}/config - duplicate method definition
Compilation failed in require at /usr/share/perl5/PVE/API2/Nodes.pm line 32.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Nodes.pm line 32.
Compilation failed in require at /usr/share/perl5/PVE/API2.pm line 14.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2.pm line 14.
Compilation failed in require at /usr/bin/pveproxy line 22.
BEGIN failed--compilation aborted at /usr/bin/pveproxy line 22.
 (warning).
Restarting PVE SPICE Proxy Server: spiceproxyregister method PVE::API2::Qemu/{vmid}/config - duplicate method definition
Compilation failed in require at /usr/share/perl5/PVE/API2/Nodes.pm line 32.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Nodes.pm line 32.
Compilation failed in require at /usr/share/perl5/PVE/API2.pm line 14.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2.pm line 14.
Compilation failed in require at /usr/bin/spiceproxy line 16.
BEGIN failed--compilation aborted at /usr/bin/spiceproxy line 16.
 (warning).
Processing triggers for pve-firewall ...
Restarting Proxmox VE firewall: pve-firewall.
Processing triggers for man-db ...
root@pve-desktop:~#

pvetest has been updated, so just do

#apt-get update
#apt-get dist-upgrade

 

[optim]

Success!!!

Thanks spirit!

I ran an apt-get update, apt-get upgrade and apt-get dist-upgrade. Once updated, the vga=none was correct out of the box, so I no longer needed to use a bash script to launch. However, when launching the vm I still got the vfio errors:

kvm: -device vfio-pci,host=01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-vga=on,multifunction=on: vfio: error opening /dev/vfio/1: No such file or directory
kvm: -device vfio-pci,host=01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-vga=on,multifunction=on: vfio: failed to get group 1
kvm: -device vfio-pci,host=01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-vga=on,multifunction=on: Device initialization failed.
kvm: -device vfio-pci,host=01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-vga=on,multifunction=on: Device 'vfio-pci' could not be initialized

But I decided to try adding "pcie_acs_override=downstream" to my grub boot cmdline and the vfio devices populated correctly, removing the above error.

So now the VM started correctly, without any hacks or workarounds. The only problem was that the VM was still crashing after installation.

The trick for me was this. If I install Win7 without passhrough enabled, all is well. Once I enable passthrough Windows will detect the Radeon card upon boot up. The critical thing in my case was that I needed to install the drivers as soon as it was detected. If I reboot the vm without installing the driver as soon as the card is detected, it would crash upon startup from then on. Once the driver was installed, I reboot the vm and I am greeted with a dual monitor Win7 desktop working flawlessly on 2560 x 1440 monitors!

The usb mapping is working perfectly too, so my keyboard and mouse are detected and running fine without requiring passthrough like Xen.

I still have more playing to do to ensure that is is a stable setup, but if all goes well there is nothing stopping me from converting over from Xen!

Thanks again to spirit and all the other Proxmox devs, you guys are awesome!

Daniel

 

[XueSheng]

But I decided to try adding "pcie_acs_override=downstream" to my grub boot cmdline and the vfio devices populated correctly, removing the above error.

Did you use a patched kernel for pcie_acs_override? Because pvetest kernel does not include pcie_acs_override... (if you used pvetest kernel, pcie_acs_override is ignored). I'm just curious if pcie_acs_override really was the reason that your iommu_groups were split up.

 

[optim]

I'm away from my computers (at cottage) so I can't answer for sure until tomorrow, but I can tell you that the pcie_acs_override switch was the only thing I changed and the iommu stuff came alive. I plan on testing further tomorrow when I get back, but maybe spirit (or dietmar) added the patch you spoke of to the -13 3-10 kernel in pvetest. Just a guess.

In any case it does something to fix the vfio error I was getting. Thanks for posting it to the forums as it was your post that inspired me to try it.

Daniel

 

[spirit]

I'm away from my computers (at cottage) so I can't answer for sure until tomorrow, but I can tell you that the pcie_acs_override switch was the only thing I changed and the iommu stuff came alive. I plan on testing further tomorrow when I get back, but maybe spirit (or dietmar) added the patch you spoke of to the -13 3-10 kernel in pvetest. Just a guess.

In any case it does something to fix the vfio error I was getting. Thanks for posting it to the forums as it was your post that inspired me to try it.

Daniel

No, we don't have added any patch to pve-kernel 3.10 (based on rhel7 kernel).
Indeed, I don't see any reference to pcie_acs_override in current proxmox kernel.

 

[spirit]

The good news, the pci_acs_override patch apply correctly to pve-kernel 3.10.
I'll send a patch to the dev mailing list to have it applied by default

 

[optim]

No, we don't have added any patch to pve-kernel 3.10 (based on rhel7 kernel).
Indeed, I don't see any reference to pcie_acs_override in current proxmox kernel.

You are absolutely correct. Sorry if I mislead anyone, but I was in a rush to test and post my findings before leaving for the cottage, so I think I may have messed up somewhere.

I tested everything again last night by completely wiping out my Proxmox install and starting over, performing all the steps I mentioned in a previous email, minus the pcie_acs_override flag and using the -4 release of the kernel instead of -3. Passthrough worked right away, and has been stable ever since. I also tried replacing the Radeon with a GTX 650 and that works perfect too.

So it wasn't the pcie_acs_override, but instead (I think) the hard work of spirit and dietmar with the new kernel builds/Proxmx patches that fixed it all for me.

It's hard to sift through all the clutter sometimes with passthrough, as I was trying all sorts of stuff, but in the end it is pretty simple if you stick to the wiki steps...

Thanks again.

 

[jpbaril]

I would like to do video passthrough on a VM.

If I read the thread correctly, in addition to use the 3.10 kernel, I must use a PCI GPU card as I cannot passthrough the Intel IGD present on my Intel Vt-d CPU.

What I don't understand is that I read everywhere -- and it was also mentioned at some point here -- that the kernel also needs to have the VGA arbiter patch for i915 driver. Yet, it does not appear the actual 3.10 pvetest kernel has it. So how Optim was able to passthrough his Radeon GPU ?

I want to be sure I'll be able to do GPU passthrough before actually buying a GPU card.

Thank you.