VGA Passthrough : "vfio: failed to set iommu for container: Operation not permitted"

[Thies88]

Im using a HD 6850 with passtrough (Windows 10) using SMBIOS with the followwing commands. Also on a second host an IGD HD4600 (i5 4670):

Did you try (with smbios) removing the "disable_vga=1" because that doesnt work with smbios.

But maby its a good idea to start from the beginning:

Please post output of the following commands

pveversion -v
dmesg | grep -e "Directed I/O"
dmesg | grep ecap
lspci | grep VGA
lspci -n -s XX:XX (where XX:XX is the output lspci port from lspci | grep VGA (example 01:00) )
cat /etc/modprobe.d/kvm.conf
cat /etc/modprobe.d/vfio.conf
Optional: cat /etc/modprobe.d/iommu_unsafe_interrupts.confnano pve

In wich PCIe slot did you put the GPU (first slot or maby second ?)

 

[phguy]

Hello, thanks for your help.

I only have one port for my GPU in 16x PCIe.

Where should I find "disable_vga=1"? I only have "x-vga=on" in my VM.conf file
I've already tried with seabios and x-vga=on ==> doesn't work
I've already tried with seabios and no x-vga=on ==> doesn't work
should I try with disable_vga=1 ?

here is bellow the result of the "cat" you asked :

root@MyA-N08:~# pveversion -v
proxmox-ve: 4.4-90 (running kernel: 4.4.67-1-pve)
pve-manager: 4.4-13 (running version: 4.4-13/7ea56165)
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.4.35-1-pve: 4.4.35-77
pve-kernel-4.4.35-2-pve: 4.4.35-79
pve-kernel-4.4.21-1-pve: 4.4.21-71
pve-kernel-4.4.15-1-pve: 4.4.15-60
pve-kernel-4.4.59-1-pve: 4.4.59-87
pve-kernel-4.4.44-1-pve: 4.4.44-84
pve-kernel-4.4.16-1-pve: 4.4.16-64
pve-kernel-4.4.67-1-pve: 4.4.67-90
pve-kernel-4.4.24-1-pve: 4.4.24-72
pve-kernel-4.4.19-1-pve: 4.4.19-66
pve-kernel-4.4.49-1-pve: 4.4.49-86
pve-kernel-4.4.40-1-pve: 4.4.40-82
pve-kernel-4.4.62-1-pve: 4.4.62-88
lvm2: 2.02.116-pve3
corosync-pve: 2.4.2-2~pve4+1
libqb0: 1.0.1-1
pve-cluster: 4.0-52
qemu-server: 4.0-110
pve-firmware: 1.1-11
libpve-common-perl: 4.0-95
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-76
pve-libspice-server1: 0.12.8-2
vncterm: 1.3-2
pve-docs: 4.4-4
pve-qemu-kvm: 2.7.1-4
pve-container: 1.0-100
pve-firewall: 2.0-33
pve-ha-manager: 1.0-41
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-4
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-9
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.9-pve15~bpo80


root@MyA-N08:~# dmesg | grep -e "Directed I/O"
[    1.211871] DMAR: Intel(R) Virtualization Technology for Directed I/O



root@MyA-N08:~# dmesg | grep ecap
[    1.211021] DMAR: dmar0: reg_base_addr cfffe000 ver 1:0 cap c90780106f0462 ecap f0207e

root@MyA-N08:~# lspci | grep VGA
01:03.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] ES1000 (rev 02)
0b:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Polaris11] (rev cf)

root@MyA-N08:~# lspci -n -s XX:XX (where XX:XX is the output lspci port from lspci | grep VGA (example 01:00) )
-bash: syntax error near unexpected token `('

root@MyA-N08:~# cat /etc/modprobe.d/kvm.conf
cat: /etc/modprobe.d/kvm.conf: No such file or directory

root@MyA-N08:~# cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=1002:67ff,1002:aae0 disable_vga=1

root@MyA-N08:~# Optional: cat /etc/modprobe.d/iommu_unsafe_interrupts.confnano pve
-bash: Optional:: command not found

 

[Thies88]

disable_vga=1can be found in vfio.conf maby you can remove it.

I wil take a look when i'm at home. I see you have 2 gpu' s installed ?

 

[Thies88]

Ok please rerun the following and post them here in a replay:

lspci -n -s 0b:00.0
find /sys/kernel/iommu_groups/ -type l
cat /proc/cpuinfo

meanwhile you can try the following:
edit your config files as follows (just copy paste the following )

mv /etc/modprobe.d/vfio.conf /etc/modprobe.d/vfio.conf.old
echo "options vfio-pci ids=1002:67ff,1002:aae0" > /etc/modprobe.d/vfio.conf
echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf

and remove the following lines from /etc/modules:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

execute this

update-initramfs -u

REBOOT HOST

and for your vmid.conf make sure the following is set(use SeaBIOS):

machine: q35
hostpci0: 0b:00,pcie=1,x-vga=on

I can see that you have updated this host a couple of times (a lot of diffrent kernels) maby after setting above settings you could try to boot from a diffrent kernel like: 4.4.35-1-pve

 

[phguy]

Thanks for this new advices.

disable_vga=1can be found in vfio.conf maby you can remove it.

I wil take a look when i'm at home. I see you have 2 gpu' s installed ?

Yes I have 2 GPU, one intégrated for the physical machine, and one that I want to dedicate to one VM.
I've removed the whole line in vfio.conf has it was for OMVF.
then I've tried both test in a smbios VM and in a UEFI VM : none detects the GPU.

here is the result of your commands :
SEE attached file (cause it was too long to be accepted)

hope this will help.

I've tried with the removing vfio options in /etc/modules and reboot, and try again the VM with the line :
hostpci0: 0b:00,pcie=1,x-vga=on

but the VM refuse to boot with message :

kvm: -device vfio-pci,host=0b:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-vga=on,multifunction=on: vfio: failed to set iommu for container: Operation not permitted[/QUOTE]
the end of the message is in the attached file also.

 

[phguy]

I've try with :

hostpci0: 0b:00,pcie=1,x-vga=on,driver=vfio

the VM starts, but doesn't detect the card in the VM.

 

[Thies88]

did you add allow_unsafe_interrupts :

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf

also try to to set the following in your VMID.conf:

vga: none

I also found this on the forum... please read it it could be a HP problem:
https://forum.proxmox.com/threads/pcie-passthrough-on-4-2-acces-denied-its-a-hp-problem-with-rmrr.27608/

and this one:
https://forum.proxmox.com/threads/h...ontroller-hp-p410-passthrough-probleme.30547/
Read post #14 for possible solution

they mention the exact same error code:

vfio: failed to set iommu for container: Operation not permitted
vfio: failed to setup container for group xx
vfio: failed to get group xx

 

[phguy]

Hello,

I've tried with omvf and with seabios. I've tried with and without each option of the hostpci0 line in the vm.conf file.
I haven't tried with the allowunsafeinterrupts option as it doesn't seem to be the problem. the VT-d option in my physical machine was ON.
I've found with dialog with the proxmox support team that nobody managed yet to do passthrough with some HP servers so I suppose this is a motherboard limitation.
So I have to close my ticket without solution. Sorry for those who read this because they have the same problem.
Thank you for each person who helped me !
kind regards,
Philippe