VGA Passthrough : "vfio: failed to set iommu for container: Operation not permitted"

[phguy]

Hi all,
I really need help.
I've an up-to-date proxmox VE installation and I need to passthrough a GPU Card to a Windows 10 VM.

Server : HP ML350 G6 with 2x Xeon
GPU : AMD Radeon RX 560 4GDDR5

I've followed the steps in the wiki :
_in /etc/default/grub , i've changed the line --> GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
_then i've ran : update-grub
_in /etc/modules , I've added lines :
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

_I've looked for the pci address with "lspci" and found
0b:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Polaris11] (rev cf)

_I've ran : lspci -n -s 0b:00.0 which gives :
0b:00.0 0300: 1002:67ff (rev cf)
0b:00.1 0403: 1002:aae0

_i've done :
echo "options vfio-pci ids=1002:67ff,1002:aae0 disable_vga=1" > /etc/modprobe.d/vfio.conf

_then I've blacklisted the drivers :
echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf

_the I've added in my VMxxx.conf file :
machine: q35
hostpci0: 0b:00.0,pcie=1,x-vga=on


And now, when I try to start the VM, I've the following message :
"vfio: failed to set iommu for container : Operation not permitted"

Full message :
kvm: -device vfio-pci,host=0b:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on: vfio: failed to set iommu for container: Operation not permitted
kvm: -device vfio-pci,host=0b:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on: vfio: failed to setup container for group 20
kvm: -device vfio-pci,host=0b:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on: vfio: failed to get group 20
kvm: -device vfio-pci,host=0b:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on: Device initialization failed
TASK ERROR: start failed: command '/usr/bin/kvm -id 121 -chardev 'socket,id=qmp,path=/var/run/qemu-server/121.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/121.pid -daemonize -smbios 'type=1,uuid=36558434-eb42-4564-84e6-03236c1b9ac2' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/kvm/OVMF_CODE-pure-efi.fd' -drive 'if=pflash,unit=1,id=drive-efidisk0,format=qcow2,file=/mnt/pve/SRV05-Vol1/images/121/vm-121-disk-2.qcow2' -name SRV21-Win10-CoinMiner -smp '24,sockets=2,cores=12,maxcpus=24' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,kvm=off' -m 8192 -object 'memory-backend-ram,id=ram-node0,size=4096M' -numa 'node,nodeid=0,cpus=0-11,memdev=ram-node0' -object 'memory-backend-ram,id=ram-node1,size=4096M' -numa 'node,nodeid=1,cpus=12-23,memdev=ram-node1' -k fr -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=0b:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on' -device 'vfio-pci,host=0b:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1' -chardev 'socket,path=/var/run/qemu-server/121.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:dd455418cf5e' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/mnt/pve/SRV05-Vol1/images/121/vm-121-disk-1.qcow2,if=none,id=drive-virtio0,format=qcow2,cache=none,aio=native,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap121i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=82:F1:85:59:3F:65,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -machine 'type=q35' -global 'kvm-pit.lost_tick_policy=discard'' failed: exit code 1


I've tried several combinaisons, but I haven't found one that work. Any ideas guys and girls?
I'm desesperated.
kind regards,
Philippe

 

[imrazor]

What do you see if you run:

# dmesg | grep -e vfio

You might also need to passthrough your video card as a multifunction device, since it has an HDMI audio chip onboard.

In your VMID.conf file, change:

machine: q35
hostpci0: 0b:00.0,pcie=1,x-vga=on

to:

machine: q35
hostpci0: 0b:00,pcie=1,x-vga=on

 

[phguy]

Hello Imrazor,
Thank you for your answer.

I've changed to what you proposed, but it didn't change. I've found and tried with :

hostpci0: 0b:00,pcie=1,x-vga=on,driver=vfio

It helps and the VM start, BUT, the card does appear in the VM when started in Windows.
Then I don't know how to do. Last time I've passthrough something, it was SSD and it worked fine. Here, I don't why I don't see the GPU Card.
Do you have an idea?
kind regards,
Philippe

 

[imrazor]

Can you run the command I put in my first post?

# dmesg | grep -e vfio

I want to make sure vfio is grabbing your video card's device IDs.

Have you specified ovmf in your vmid.conf file,

bios: ovmf

or are you using one of the SeaBios methods from here: https://pve.proxmox.com/wiki/Pci_passthrough ?

You can also try dropping "pcie=1" from the hostpci0 line.

 

[phguy]

Hello @imrazor,
Thanks for your help.
I've read too quickly, so I forgot. Here is the result of :
Code:
# dmesg | grep -e vfio
[ 19.138331] vfio_pci: add [1002:67ff[ffff:ffff]] class 0x000000/00000000
[ 19.154319] vfio_pci: add [1002:aae0[ffff:ffff]] class 0x000000/00000000

for th VM, I've tried in omvf, but then my windows doesn't start. So I only tried with the seabios methods. Do you think it can change my result? If you think it could be a solution, I can try to reinstall the whole windows VM but it could be long.

So now I try with the next line in my VM as you suggest :
hostpci0: 0b:00,x-vga=on,driver=vfio

and it starts, but the Graphic Card is not detect in the VM. I've tried the Graphic Card in a physical machine with the same version of Windows and it is detected.
Somehow, something is not going threw I think. I don't know what and how to solve it.
do you have another idea?
kind regards,
philippe

 

[imrazor]

I know I had to use ovmf to get my 1050ti working. I would suggest trying that again, especially since you're using a modern graphics card. I would also try dropping driver=vfio, and maybe pcie=1 as well.

 

[phguy]

Hello,
Ok, I'll try and come back to you.
kind regards,
Philippe

 

[phguy]

I'm back after having tried for hours to install a win 10 with Omvf in proxmox. I can't or I don't know. each time I boot, it finish to a SHELL console proposed by the UEFI bios. I can't boot on my iso as usual. can you give me the trick please?
kind regards,
Philippe

 

[imrazor]

I had the same problem after experimenting with ovmf and SeaBios. After you get to the EFI shell, type 'exit', then select 'Boot Manager'. From the Boot Manager select 'Misc UEFI Device'. Unfortunately, I have to do that every time I boot the VM with GPU passthrough. I'm working on a permanent solution...

EDIT: Actually, if you're still trying to boot the ISO you may need to select something other than Misc UEFI Device. See if there is a CD/DVD ROM option in the Boot Manager.

 

[phguy]

i've try without vfio option, but the VM refuse to boot. so for the moment, my line is :
hostpci0: 0b:00,pcie=1,x-vga=on,driver=vfio
machine: q35
EDIT : this message crossed yours. sorry

 

[imrazor]

So try booting with the same settings that got you to the EFI shell, type 'exit' then select Boot Manager and try to boot with the CD/DVD ROM option. Hopefully that'll work...

 

[phguy]

The UEFI refuse to boot on the CD UEFI after entering exit and selecting boot manager. when did that, were your system already installed?

 

[imrazor]

The UEFI refuse to boot on the CD UEFI after entering exit and selecting boot manager. when did that, were your system already installed?

My OS was already installed, so I was able to select 'Misc UEFI Device' in the boot manager.

Are you sure that a bootable ISO is selected in your VM settings/config file?

 

[phguy]

I've tried on the VM1 (the current one with the card) installed with Win10 to change the bios to OMVF : the VM freezes on boot and the PROXMOX logo stay on screen.
For the moment on my original VM, here is my config file :
agent: 1
balloon: 1024
bootdisk: virtio0
cores: 8
cpu: host
hotplug: disk,network,usb,memory,cpu
ide2: none,media=cdrom
machine: q35
memory: 8192
name: SRV21-Win10
net0: virtio=82:F1:85:59:3F:65,bridge=vmbr0
numa: 1
onboot: 1
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=36558434-eb42-4564-84e6-03236c1b9ac2
sockets: 2
startup: order=5,up=60,down=60
virtio0: MyA4-NFS-VMachines:121/vm-121-disk-1.qcow2,size=33G
hostpci0: 0b00:0,pcie=1,x-vga=on,driver=vfio
hostpci1: 0b00:1,pcie=1,driver=vfio


on VM2 (a new one with OMVF, to install fresh system) :
The 2 iso I'm using for my test are Win Iso that works too boot. I have just try with a debian iso, and it boots on. may be my win iso boots on normal bios and not on OMVF bios. I'll try to find another one. I've found an iso with win8.1 that boots up. I start a new install and get back to you with it.

 

[imrazor]

Sometimes if you wait a few minutes at the Proxmox logo, it'll drop to the UEFI shell.

Good idea creating a new VM. IIRC, Win8 or later is required for UEFI booting. Also, I think you need

balloon: 0

for GPU passthrough to work properly.

 

[phguy]

Hi,
Well the VM in UEFI is now available. So I've tried to passthrough the VGA Card but nothing appear in the VM and the driver installation process doesn't detect it neither. I've tried also with Balloon=0 or 1024
here is the config file :
agent: 1
balloon: 0
bios: ovmf
boot: cdn
bootdisk: virtio0
cores: 6
efidisk0: SRV05-Vol1:127/vm-127-disk-2.qcow2,size=128K
ide0: none,media=cdrom
machine: q35
memory: 4096
name: SRV27-Win10-TestUEFI
net0: virtio=860:5E:5B:B6:2A,bridge=vmbr0
numa: 1
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=cbc868ab-4589-48d6-90fa-4fbfcc87cb49
sockets: 2
virtio0: SRV05-Vol1:127/vm-127-disk-1.qcow2,size=32G
hostpci0: 0b:00,pcie=1,x-vga=on,driver=vfio


when it is set like this, the too lines disappear after 1 VM on and VM off :
hostpci0: 0b:00.0,pcie=1,driver=vfio
hostpci1: 0b:00.1,pcie=1,x-vga=on,driver=vfio

any clue?
kind regards,
Philippe

 

[imrazor]

Hi,

hostpci0: 0b:00,pcie=1,x-vga=on,driver=vfio


when it is set like this, the too lines disappear after 1 VM on and VM off :
hostpci0: 0b:00.0,pcie=1,driver=vfio
hostpci1: 0b:00.1,pcie=1,x-vga=on,driver=vfio

I think the top example is closer to the one you want. I'm a little confused. If you have x-vga=on, you should only be able to see the Windows desktop on a monitor attached to the GPU you're passing through. So are you viewing the Windows desktop from the Proxmox console, or on a seperate monitor attached to your passthrough GPU?

 

[phguy]

I'm viewing the windows desktop from proxmox console. I didn't try with a screen plugged on the Graphic Card. As the card is not detected by Windows VM, I supposed it wouldn't work. My goal is to use the Graphic Card for the GPU capabilities but not for the material output.
So you mean I should use only one line like ? :
hostpci0: 0b:00,pcie=1,driver=vfio

 

[imrazor]

That looks right to me assuming you want to keep the Proxmox console available. The one that works for me looks like:

hostpci0: 22:00,pcie=1,x-vga=on

You know if you're using Proxmox for serious use (ie, there's money involved) you might want to consider getting a support contract.

 

[phguy]

Hi Imrazor,
You are right, I'll get a contract. I hope they will find something.
Thank you for your help anyway.
have a nice evening.
Philippe