proxmox 6.2 breaks gpu passthru

[Joris L.]

Hello Joris!

Sorry for the delay in responding. My PSU went bad and just got another one installed. Whew.

Let me see if passthrough is working....

if the monitor goes black and only VNC is working that is a good thing

 

[scrapiron]

Well, I guess. If I remember correctly, when the GPU passthrough was working, the console wouldn't connect (which makes sense).
So at VM boot, the proxmox screen comes up for the VM, then goes to black and nothing.. have to go to the console. I wonder if my video drivers are messed up. I'll create a new VM to see.

 

[Joris L.]

Well, I guess. If I remember correctly, when the GPU passthrough was working, the console wouldn't connect (which makes sense).
So at VM boot, the proxmox screen comes up for the VM, then goes to black and nothing.. have to go to the console. I wonder if my video drivers are messed up. I'll create a new VM to see.

1. You must distinguish a reliable way of testing GPU passthrough on the VM first, don't mind about what weirdness you observe at this point
2. the console blanking on the physical console is influenced by video=vseafb:off video=efifb:off and also nomodeset I don't think i forget something here
3. you have to set the GPU to virtio-GPU, i assigned 512 memory
4. you have to create and enable a PCI device for the PCI id of the video card, here i enabled all options (primary GPU, pci express, rom bar, all functions and also a bios file configured
5. i'm not sure i remember correctly but i think i observed only rom files ending in .bin load correctly, i mention this just in case

 

[scrapiron]

still no go.

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1 nofb nomodese
t video=vesafbff video=efifbff"

GPU is virtio-GPU
PCI device is identified and all selected.
No idea on the rom file.. hmmm

 

[Joris L.]

still no go.

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1 nofb nomodese
t video=vesafbff video=efifbff"

GPU is virtio-GPU
PCI device is identified and all selected.
No idea on the rom file.. hmmm

make this cmdline as simple as possible, mabye so maybe not, write it as iommu=pt intel_iommu=on nobar noats nomodeset video=vseafbff video=efifbff

this configuration below should be a minimum

Configuration required


Code:
/etc/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

aufs
overlay

I trust you know to replace x, y, z with the correct information.
disable_vga and disabled_idel_d3 are optional

Code:
/etc/modprobe.d/

blacklist amdgpu

# GPU and AUDIO

options vfio-pci ids=xxxx:yyyyy,xxxxx:zzzz disable_vga=1 disable_idle_d3=1
options kvm ignore_msrs

 

[scrapiron]

First off Joris thank you so much for all your time and assistance!

Okay, adding a more simple grub line. Same result,

Here's all my settings..

GRUB--
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
#_NEW_GRUB_CMDLINE_LINUX_DEFAULT="iommu=on amd_iommu=forced_isolation nobar video=vsafbff video=efifbff pci=pcie_bus_perf,pcie_scan_all,big_root_window pcie_ports=native"
# OLDGRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafbff efifbff"
#GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1 nofb nomodeset video=vesafbff video=efifbff"
GRUB_CMDLINE_LINUX_DEFAULT="iommu=pt intel_iommu=on nobar noats nomodeset video=vesafbff video=efifbff"

# Disable os-prober, it might add menu entries for each guest
GRUB_DISABLE_OS_PROBER=true

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"
_______________________________________________________
MODULES
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
______________________________________________________
blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia
_________________________________________
iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1
_______________________________________________________
kvm.conf
options kvm ignore_msrs=1
___________________________________________________
pve-blacklist.conf
# This file contains a list of modules which are not supported by Proxmox VE

# nidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
blacklist nvidia
blacklist radeon
blacklist nouveau
__________________________________________________
vfio.conf
options vfio-pci ids=10de:1c03 disable_vga=1 disable_idle_d3=1
options kvm ignore_msrs
_____________________________________________-
VM.conf

agent: 1
#args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,kvm=off'
audio0: device=AC97,driver=spice
balloon: 8032
bios: ovmf
bootdisk: scsi0
cores: 1
efidisk0: local-lvm:vm-102-disk-0,size=4M
hostpci0: 04:00,pcie=1,x-vga=1
ide2: local:iso/pop-os_20.10_amd64_nvidia_1.iso,media=cdrom
machine: q35
memory: 16032
name: POP
net0: e1000=DA:1F:8D:A1:F6:71,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: Fedora:vm-102-disk-0,cache=writethrough,size=476G
scsihw: virtio-scsi-pci
smbios1: uuid=31f37015-d24e-4bef-9b9e-2a879b7042d5
sockets: 4
usb0: host=05ac:024f,usb3=1
usb1: host=03f0:a407,usb3=1
vga: virtio,memory=512
vmgenid: c25dc664-5081-4c1f-82a7-1ef55b7acb8c

 

[Joris L.]

First off Joris thank you so much for all your time and assistance!

Okay, adding a more simple grub line, adding in second line to vfio.conf file. The external screen now just displays the ProxMox Boot screen but instead of cutting out, it just is frozen there. The console works.

Here's all my settings..

GRUB--
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
#_NEW_GRUB_CMDLINE_LINUX_DEFAULT="iommu=on amd_iommu=forced_isolation nobar video=vsafbff video=efifbff pci=pcie_bus_perf,pcie_scan_all,big_root_window pcie_ports=native"
# OLDGRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafbff efifbff"
#GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1 nofb nomodeset video=vesafbff video=efifbff"
GRUB_CMDLINE_LINUX_DEFAULT="iommu=pt intel_iommu=on nobar noats nomodeset video=vesafbff video=efifbff"

# Disable os-prober, it might add menu entries for each guest
GRUB_DISABLE_OS_PROBER=true

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"
_______________________________________________________
MODULES
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
______________________________________________________
blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia
_________________________________________
iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1
_______________________________________________________
kvm.conf
options kvm ignore_msrs=1
___________________________________________________
pve-blacklist.conf
# This file contains a list of modules which are not supported by Proxmox VE

# nidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
blacklist nvidia
blacklist radeon
blacklist nouveau
__________________________________________________
vfio.conf
options vfio-pci ids=10de:1c03 disable_vga=1 disable_idle_d3=1
options kvm ignore_msrs
_____________________________________________-
VM.conf

agent: 1
#args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,kvm=off'
audio0: device=AC97,driver=spice
balloon: 8032
bios: ovmf
bootdisk: scsi0
cores: 1
efidisk0: local-lvm:vm-102-disk-0,size=4M
hostpci0: 04:00,pcie=1,x-vga=1
ide2: local:iso/pop-os_20.10_amd64_nvidia_1.iso,media=cdrom
machine: q35
memory: 16032
name: POP
net0: e1000=DA:1F:8D:A1:F6:71,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: Fedora:vm-102-disk-0,cache=writethrough,size=476G
scsihw: virtio-scsi-pci
smbios1: uuid=31f37015-d24e-4bef-9b9e-2a879b7042d5
sockets: 4
usb0: host=05ac:024f,usb3=1
usb1: host=03f0:a407,usb3=1
vga: virtio,memory=512
vmgenid: c25dc664-5081-4c1f-82a7-1ef55b7acb8c

you are welcome, i miss the collaborative spirit in newsgroups and forums so i try to keep the flame burning

good, in my experience this means GPU passthrough is working

i've yet to test this but writing the cmdline differntly for video=vesafbff video=efifbff could change that, even rearranging the order

 

[scrapiron]

In the console, I see the Nvidia GPU... even third party apps can see it. So yeah, it's working, just not shooting the display to where I want it. Using ProxMox 6.2-12. This is PopOS 21.10 with Nvidia drivers 455.28

alright, changed the order of those two commands. same result.

 

[Joris L.]

In the console, I see the Nvidia GPU... even third party apps can see it. So yeah, it's working, just not shooting the display to where I want it. Using ProxMox 6.2-12. This is PopOS 21.10 with Nvidia drivers 455.28

alright, changed the order of those two commands. same result.

ehm, what do you want to do with the display ?

if you're trying to make it work on the physical display, that is possible i think

i assume not using nomodeset should fix that, if i remember well this is not an ideall approach, i started using nomachine for remote desktop as it is GPU accelerated

 

[scrapiron]

That's the goal- to use an external display with the VM.

Using the console or another VNC client doesn't seem to leverage the GPU. Which makes sense I guess.

 

[Joris L.]

That's the goal- to use an external display with the VM.

Using the console or another VNC client doesn't seem to leverage the GPU. Which makes sense I guess.

i'm not sure you understand why you needed GPU passthrough
like i said, try nomachine.com (free version works fine) unless it is for gaming this will work just fine

i accidentay had this working at one point, from memory i'd say, experiment with nomodeset, video= , efifb= by changing the order, putting them right after intel_iommu= and iommu= , removing them

 

[scrapiron]

it is for gaming. I'll keep experimenting... it worked fine under ProxMox 6.1, then after the update failed.

 

[Joris L.]

it is for gaming. I'll keep experimenting... it worked fine under ProxMox 6.1, then after the update failed.

please, let us know what makes it work

 

[scrapiron]

when I look here:
root@scrapiron:~# dmesg | grep 'remapping'
[ 1.009151] DMAR-IR: This system BIOS has enabled interrupt remapping
interrupt remapping is being disabled. Please

perhaps this is the issue?

 

[Joris L.]

can't say

when I look here:
root@scrapiron:~# dmesg | grep 'remapping'
[ 1.009151] DMAR-IR: This system BIOS has enabled interrupt remapping
interrupt remapping is being disabled. Please

perhaps this is the issue?

for output to screen ? no, not to my understanding, this is exclusively because of boot time parameters video= / efifb= / nomodeset
to have output to screen you may also need to set the VGA output to not use virtio i guess, i'm happy with nomachine so i did not try more

first check if intel_iommu= is set/loading correctly

try dmesg | grep -i -e iommu -e dmar -e bar -e passthrough -e vfio for a full picture

to my understanding remapping is disabled because you set a boot time option ?

based on https://www.kernel.org/doc/html/v5.1/admin-guide/kernel-parameters.html

you may consider intremap=on to force enable remapping (at your own risk, not sure why it is off at all)


_______ this suggests a configuration issue in the cmdline

intel_iommu=    [DMAR] Intel IOMMU driver (DMAR) option
                on
                        Enable intel iommu driver.
                off
                        Disable intel iommu driver.
                igfx_off [Default Off]
                        By default, gfx is mapped as normal device. If a gfx
                        device has a dedicated DMAR unit, the DMAR unit is
                        bypassed by not enabling DMAR with this option. In
                        this case, gfx device will use physical address for
                        DMA.
                forcedac [x86_64]
                        With this option iommu will not optimize to look
                        for io virtual address below 32-bit forcing dual
                        address cycle on pci bus for cards supporting greater
                        than 32-bit addressing. The default is to look
                        for translation below 32-bit and if not available
                        then look in the higher range.
                strict [Default Off]
                        With this option on every unmap_single operation will
                        result in a hardware IOTLB flush operation as opposed
                        to batching them for performance.
                sp_off [Default Off]
                        By default, super page will be supported if Intel IOMMU
                        has the capability. With this option, super page will
                        not be supported.
                sm_on [Default Off]
                        By default, scalable mode will be disabled even if the
                        hardware advertises that it has support for the scalable
                        mode translation. With this option set, scalable mode
                        will be used on hardware which claims to support it.
                tboot_noforce [Default Off]
                        Do not force the Intel IOMMU enabled under tboot.
                        By default, tboot will force Intel IOMMU on, which
                        could harm performance of some high-throughput
                        devices like 40GBit network cards, even if identity
                        mapping is enabled.
                        Note that using this option lowers the security
                        provided by tboot because it makes the system
                        vulnerable to DMA attacks.

 

[scrapiron]

Hmmm, results:
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.4.65-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 nobar pci=pcie_bus_perf,pcie_scan_all,big_root_window pcie_ports=native video=vesafbff video=efifbff transparent_hugepage=always
[ 0.030206] ACPI: DMAR 0x00000000000FD2C8 0000F8 (v01 DELL B10K 00000015 ASL 00000061)
[ 0.500505] Booting paravirtualized kernel on bare hardware
[ 0.502337] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.4.65-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 nobar pci=pcie_bus_perf,pcie_scan_all,big_root_window pcie_ports=native video=vesafbff video=efifbff transparent_hugepage=always
[ 0.502431] DMAR: IOMMU enabled
[ 1.009271] DMAR-IR: This system BIOS has enabled interrupt remapping
[ 1.030364] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
[ 1.725910] iommu: Default domain type: Passthrough (set via kernel command line)
[ 1.730376] pci 0000:00:1f.2: can't claim BAR 5 [mem 0xff970000-0xff9707ff]: no compatible bridge window
[ 1.754798] pnp 00:00: disabling [io 0x0800-0x085f] because it overlaps 0000:00:1f.0 BAR 13 [io 0x0800-0x087f]
[ 1.754801] pnp 00:00: disabling [io 0x0860-0x08ff] because it overlaps 0000:00:1f.0 BAR 13 [io 0x0800-0x087f]
[ 1.794111] pci 0000:00:1f.2: BAR 5: assigned [mem 0xc0000000-0xc00007ff]
[ 2.598752] DMAR: Host address width 40
[ 2.598755] DMAR: DRHD base: 0x000000cfffe000 flags: 0x0
[ 2.598779] DMAR: dmar0: reg_base_addr cfffe000 ver 1:0 cap c90780106f0462 ecap f020fe
[ 2.598780] DMAR: DRHD base: 0x000000fedc0000 flags: 0x1
[ 2.598789] DMAR: dmar1: reg_base_addr fedc0000 ver 1:0 cap c90780106f0462 ecap f020f6
[ 2.598790] DMAR: RMRR base: 0x000000bfe58000 end: 0x000000bfe6ffff
[ 2.598792] DMAR: ATSR flags: 0x0
[ 2.598794] DMAR: ATSR flags: 0x0
[ 2.599161] DMAR: dmar0: Using Queued invalidation
[ 2.599183] DMAR: dmar1: Using Queued invalidation
[ 2.599645] pci 0000:00:00.0: Adding to iommu group 0
[ 2.599674] pci 0000:00:01.0: Adding to iommu group 1
[ 2.599700] pci 0000:00:03.0: Adding to iommu group 2
[ 2.599725] pci 0000:00:07.0: Adding to iommu group 3
[ 2.599751] pci 0000:00:14.0: Adding to iommu group 4
[ 2.599777] pci 0000:00:14.1: Adding to iommu group 5
[ 2.599801] pci 0000:00:14.2: Adding to iommu group 6
[ 2.599883] pci 0000:00:1a.0: Adding to iommu group 7
[ 2.599906] pci 0000:00:1a.1: Adding to iommu group 7
[ 2.599928] pci 0000:00:1a.2: Adding to iommu group 7
[ 2.599952] pci 0000:00:1a.7: Adding to iommu group 7
[ 2.599978] pci 0000:00:1b.0: Adding to iommu group 8
[ 2.600039] pci 0000:00:1c.0: Adding to iommu group 9
[ 2.600070] pci 0000:00:1c.5: Adding to iommu group 9
[ 2.600155] pci 0000:00:1d.0: Adding to iommu group 10
[ 2.600179] pci 0000:00:1d.1: Adding to iommu group 10
[ 2.600203] pci 0000:00:1d.2: Adding to iommu group 10
[ 2.600226] pci 0000:00:1d.7: Adding to iommu group 10
[ 2.600250] pci 0000:00:1e.0: Adding to iommu group 11
[ 2.600668] pci 0000:00:1e.0: Using iommu dma mapping
[ 2.600738] pci 0000:00:1f.0: Adding to iommu group 12
[ 2.600765] pci 0000:00:1f.2: Adding to iommu group 12
[ 2.600789] pci 0000:00:1f.3: Adding to iommu group 12
[ 2.600826] pci 0000:01:00.0: Adding to iommu group 13
[ 2.601216] pci 0000:01:00.0: Using iommu dma mapping
[ 2.601257] pci 0000:03:00.0: Adding to iommu group 14
[ 2.601351] pci 0000:04:00.0: Adding to iommu group 15
[ 2.601391] pci 0000:04:00.1: Adding to iommu group 15
[ 2.601409] pci 0000:05:00.0: Adding to iommu group 9
[ 2.601437] pci 0000:06:00.0: Adding to iommu group 9
[ 2.601452] pci 0000:07:0a.0: Adding to iommu group 11
[ 2.601485] pci 0000:20:03.0: Adding to iommu group 16
[ 2.601512] pci 0000:20:07.0: Adding to iommu group 17
[ 2.601536] pci 0000:20:09.0: Adding to iommu group 18
[ 2.601569] pci 0000:20:14.0: Adding to iommu group 19
[ 2.601594] pci 0000:20:14.1: Adding to iommu group 20
[ 2.601620] pci 0000:20:14.2: Adding to iommu group 21
[ 2.601657] pci 0000:24:00.0: Adding to iommu group 22
[ 2.601685] pci 0000:24:00.1: Adding to iommu group 23
[ 2.601744] pci 0000:3e:00.0: Adding to iommu group 24
[ 2.601773] pci 0000:3e:00.1: Adding to iommu group 24
[ 2.601880] pci 0000:3e:02.0: Adding to iommu group 25
[ 2.601908] pci 0000:3e:02.1: Adding to iommu group 25
[ 2.601935] pci 0000:3e:02.2: Adding to iommu group 25
[ 2.601962] pci 0000:3e:02.3: Adding to iommu group 25
[ 2.601989] pci 0000:3e:02.4: Adding to iommu group 25
[ 2.602019] pci 0000:3e:02.5: Adding to iommu group 25
[ 2.602098] pci 0000:3e:03.0: Adding to iommu group 26
[ 2.602126] pci 0000:3e:03.1: Adding to iommu group 26
[ 2.602154] pci 0000:3e:03.2: Adding to iommu group 26
[ 2.602182] pci 0000:3e:03.4: Adding to iommu group 26
[ 2.602261] pci 0000:3e:04.0: Adding to iommu group 27
[ 2.602293] pci 0000:3e:04.1: Adding to iommu group 27
[ 2.602322] pci 0000:3e:04.2: Adding to iommu group 27
[ 2.602351] pci 0000:3e:04.3: Adding to iommu group 27
[ 2.602430] pci 0000:3e:05.0: Adding to iommu group 28
[ 2.602460] pci 0000:3e:05.1: Adding to iommu group 28
[ 2.602490] pci 0000:3e:05.2: Adding to iommu group 28
[ 2.602522] pci 0000:3e:05.3: Adding to iommu group 28
[ 2.602602] pci 0000:3e:06.0: Adding to iommu group 29
[ 2.602633] pci 0000:3e:06.1: Adding to iommu group 29
[ 2.602663] pci 0000:3e:06.2: Adding to iommu group 29
[ 2.602694] pci 0000:3e:06.3: Adding to iommu group 29
[ 2.602752] pci 0000:3f:00.0: Adding to iommu group 30
[ 2.602786] pci 0000:3f:00.1: Adding to iommu group 30
[ 2.602895] pci 0000:3f:02.0: Adding to iommu group 31
[ 2.602928] pci 0000:3f:02.1: Adding to iommu group 31
[ 2.602960] pci 0000:3f:02.2: Adding to iommu group 31
[ 2.602992] pci 0000:3f:02.3: Adding to iommu group 31
[ 2.603023] pci 0000:3f:02.4: Adding to iommu group 31
[ 2.603057] pci 0000:3f:02.5: Adding to iommu group 31
[ 2.603138] pci 0000:3f:03.0: Adding to iommu group 32
[ 2.603171] pci 0000:3f:03.1: Adding to iommu group 32
[ 2.603204] pci 0000:3f:03.2: Adding to iommu group 32
[ 2.603237] pci 0000:3f:03.4: Adding to iommu group 32
[ 2.603317] pci 0000:3f:04.0: Adding to iommu group 33
[ 2.603353] pci 0000:3f:04.1: Adding to iommu group 33
[ 2.603386] pci 0000:3f:04.2: Adding to iommu group 33
[ 2.603420] pci 0000:3f:04.3: Adding to iommu group 33
[ 2.603501] pci 0000:3f:05.0: Adding to iommu group 34
[ 2.603535] pci 0000:3f:05.1: Adding to iommu group 34
[ 2.603569] pci 0000:3f:05.2: Adding to iommu group 34
[ 2.603605] pci 0000:3f:05.3: Adding to iommu group 34
[ 2.603686] pci 0000:3f:06.0: Adding to iommu group 35
[ 2.603721] pci 0000:3f:06.1: Adding to iommu group 35
[ 2.603758] pci 0000:3f:06.2: Adding to iommu group 35
[ 2.603793] pci 0000:3f:06.3: Adding to iommu group 35
[ 2.603801] DMAR: Intel(R) Virtualization Technology for Directed I/O
[ 2.955482] ehci-pci 0000:00:1a.7: DMAR: Setting identity map [0xbfe58000 - 0xbfe6ffff]
[ 2.955514] ehci-pci 0000:00:1a.7: DMAR: 32bit DMA uses non-identity mapping
[ 2.959889] mptsas 0000:05:00.0: DMAR: 32bit DMA uses non-identity mapping
[ 2.974677] uhci_hcd 0000:00:1a.0: DMAR: Setting identity map [0xbfe58000 - 0xbfe6ffff]
[ 2.974709] uhci_hcd 0000:00:1a.0: DMAR: 32bit DMA uses non-identity mapping
[ 2.977783] uhci_hcd 0000:00:1a.1: DMAR: Setting identity map [0xbfe58000 - 0xbfe6ffff]
[ 2.977812] uhci_hcd 0000:00:1a.1: DMAR: 32bit DMA uses non-identity mapping
[ 2.979057] ehci-pci 0000:00:1d.7: DMAR: Setting identity map [0xbfe58000 - 0xbfe6ffff]
[ 2.979085] ehci-pci 0000:00:1d.7: DMAR: 32bit DMA uses non-identity mapping
[ 2.998579] uhci_hcd 0000:00:1a.2: DMAR: Setting identity map [0xbfe58000 - 0xbfe6ffff]
[ 2.998714] uhci_hcd 0000:00:1a.2: DMAR: 32bit DMA uses non-identity mapping
[ 3.001984] uhci_hcd 0000:00:1d.0: DMAR: Setting identity map [0xbfe58000 - 0xbfe6ffff]
[ 3.002013] uhci_hcd 0000:00:1d.0: DMAR: 32bit DMA uses non-identity mapping
[ 3.003836] uhci_hcd 0000:00:1d.1: DMAR: Setting identity map [0xbfe58000 - 0xbfe6ffff]
[ 3.003865] uhci_hcd 0000:00:1d.1: DMAR: 32bit DMA uses non-identity mapping
[ 3.004066] ata1: SATA max UDMA/133 abar m2048@0xc0000000 port 0xc0000100 irq 34
[ 3.004069] ata2: SATA max UDMA/133 abar m2048@0xc0000000 port 0xc0000180 irq 34
[ 3.004071] ata3: SATA max UDMA/133 abar m2048@0xc0000000 port 0xc0000200 irq 34
[ 3.004074] ata6: SATA max UDMA/133 abar m2048@0xc0000000 port 0xc0000380 irq 34
[ 3.006702] uhci_hcd 0000:00:1d.2: DMAR: Setting identity map [0xbfe58000 - 0xbfe6ffff]
[ 3.006731] uhci_hcd 0000:00:1d.2: DMAR: 32bit DMA uses non-identity mapping
[ 23.527225] VFIO - User Level meta-driver version: 0.3
[ 23.536728] vfio-pci 0000:04:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=nonewns=none
[ 23.536775] vfio_pci: add [10de:1c03[ffffffff:ffffffff]] class 0x000000/00000000
[ 155.423688] vfio-pci 0000:04:00.0: enabling device (0002 -> 0003)
[ 155.424226] vfio-pci 0000:04:00.0: vfio_ecap_init: hiding ecap 0x19@0x900

 

[Joris L.]

well, no problem there i guess. don't be distracted by messages too much.

 

[scrapiron]

Alright, it is fixed! GPU passthrough to an external monitor is working. GTX 1060 being seen and used. Had to change the VMID.conf file to vga: none then it worked. Thank you Joris for ALL your effort. Your help got me to this last change. Meaning it may not have worked without the changes you suggested.

From this thread: https://forum.proxmox.com/threads/gpu-pcie-passthrough-stopped-working-after-update.71183/page-2

 

[Joris L.]

Alright, it is fixed! GPU passthrough to an external monitor is working. GTX 1060 being seen and used. Had to change the VMID.conf file to vga: none then it worked. Thank you Joris for ALL your effort. Your help got me to this last change. Meaning it may not have worked without the changes you suggested.

From this thread: https://forum.proxmox.com/threads/gpu-pcie-passthrough-stopped-working-after-update.71183/page-2

eh, thanks for the thanks i guess

i'm documenting it in my redmine so i don't forget again