GPU passthrough | IOMMU not present

fidays

New Member
May 15, 2021
18
4
3
32
Hi I am running Proxmox VE 6.4-6 in one of my old desktop with the following configuration.
MSI Motherboard: H310M PRO-M2 (MS-7B28)
Processor: Intel i5 - 8600K
GPU: GeForce GT 710B

I have enabled Intel VT-d from motherboard.


I am running few VMs and one of them is a Kali linux box. I am trying to passthrough my GPU so that I can use to to do password cracking (with hashcat)
I have updated GRUB (content pasted bellow)
I have added kernel modules (content pasted bellow)
I have added GPU drivers on the blacklist (content pasted bellow)
I created a Kali VM (content pasted bellow)

The issue is
1. The GPU is still attached to host. I have a monitor plugged in to the GPU card. When I reboot/start, I can see boot process on the monitor and then stays on login screen (where it shows the IP and port to connect from a browser). As I understand, since I have added NVIDIA driver to the host's blacklist. There should not be any display from the host on the monitor. Please correct me if I am wrong.

2. When I start my Kali VM, it won't start and will throw error saying that IOMMU not present. If I remove the PCI device from VM's hardware list then the VM starts normally.

This is what my iommu groups look like

Bash:
for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
IOMMU group * 00:00.0 Host bridge [0600]: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers [8086:3ec2] (rev 07)
00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 07)
00:08.0 System peripheral [0880]: Intel Corporation Skylake Gaussian Mixture Model [8086:1911]
00:12.0 Signal processing controller [1180]: Intel Corporation Cannon Lake PCH Thermal Controller [8086:a379] (rev 10)
00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)
00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)
00:17.0 SATA controller [0106]: Intel Corporation Cannon Lake PCH SATA AHCI Controller [8086:a352] (rev 10)
00:1f.0 ISA bridge [0601]: Intel Corporation Devic[ICODE][/ICODE]e [8086:a303] (rev 10)
00:1f.3 Audio device [0403]: Intel Corporation Cannon Lake PCH cAVS [8086:a348] (rev 10)
00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-V [8086:15bc] (rev 10)
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 710B] [10de:128b] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)

VM's config
Bash:
agent: 1
balloon: 2048
bios: ovmf
boot: order=ide2;scsi1
cores: 4
cpu: host
efidisk0: ssd-pool-pmx1:vm-101-disk-1,size=4M
hostpci0: 01:00,pcie=1,x-vga=1
ide2: nas-global:iso/kali-linux-2021.1-installer-amd64.iso,media=cdrom
machine: q35
memory: 8192
name: kali
net0: virtio=06:33:51:DF:FC:C9,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi1: ssd-pool-pmx1:vm-101-disk-0,cache=unsafe,size=38G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=b9cf2968-ec31-491d-98b3-6bae816b4f49
sockets: 1
vga: none
vmgenid: ff7bb64e-e4e3-464f-a3a2-199236797092

Grub
Bash:
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
#GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
GRUB_CMDLINE_LINUX=""

# Disable os-prober, it might add menu entries for each guest
GRUB_DISABLE_OS_PROBER=true

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"


modules
Bash:
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

/etc/modprobe.d/blacklist.conf
Code:
blacklist radeon
blacklist nouveau
blacklist nvidia



Please let me know what I am doing wrong here. I do not want my GPU to be used as a display, it is just for computation.
Thank you.
 

Attachments

  • Screenshot 2021-06-08 133235.png
    Screenshot 2021-06-08 133235.png
    120.4 KB · Views: 7

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
8,276
1,049
164
34
Vienna
can you please post the full output of the command 'dmesg' ?

did you execute 'update-grub' after editing /etc/default/grub ? did you reboot afterwards?

. As I understand, since I have added NVIDIA driver to the host's blacklist.
not necessarily, the kernel can also use the basic vga driver

please check your bios if you can set the default display to the integrated graphics of your processor
 

fidays

New Member
May 15, 2021
18
4
3
32
dmesg's output is attached as a text file.

I did update-grub2 after editing the grub file.

I manually rebooted after adding the blacklist.
 

Attachments

  • dmesg.txt
    65.6 KB · Views: 13

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
8,276
1,049
164
34
Vienna
i know you said you activated it, but from the dmesg it looks like vt-d is not enabled on a bios level...
 

leesteken

Famous Member
May 31, 2020
2,052
435
88
From you dmesg.txt, it looks like you did the right things with the kernel parameters, but there are no Adding to iommu group messages. This is also clear from the IOMMU group * output. It confirms that there are no iOMMU groups (and thus the GPU is not in an isolated group) and you cannot use PCI passthrough. Both the H310 and i5-8600 are capable of VT-d, according to Intel.Unless you did not do a update initramfs -u before the update-grub, it looks like it is not (fully) enabled in the BIOS.
 

fidays

New Member
May 15, 2021
18
4
3
32
Sorry, I found out that VT-d was not enabled in the BIOS. After doing that when I boot my Kali, Nothing appears on the monitor. On the vnc window i get this error.

1623163317602.png

Also how do I create a different iommu group for GPU passthrough
 

leesteken

Famous Member
May 31, 2020
2,052
435
88
You cannot "create" IOMMU groups, and I don't think you need to. What does for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done show now that VT-d is enabled?
It looks like PCI passthrough is working. The GPU is passed to the VM and the Proxmox is also still working fine. Maybe the open source nouveau driver does not support your GPU? Maybe the GPU does not reset properly? Have you tried x-vga=on, which seems to help with NVidia GPUs? Maybe dump the ROM file and add it to the hostpci
 

fidays

New Member
May 15, 2021
18
4
3
32
Here is the output
Bash:
IOMMU group 0 00:00.0 Host bridge [0600]: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers [8086:3ec2] (rev 07)
IOMMU group 1 00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 07)
IOMMU group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 710B] [10de:128b] (rev a1)
IOMMU group 1 01:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
IOMMU group 2 00:08.0 System peripheral [0880]: Intel Corporation Skylake Gaussian Mixture Model [8086:1911]
IOMMU group 3 00:12.0 Signal processing controller [1180]: Intel Corporation Cannon Lake PCH Thermal Controller [8086:a379] (rev 10)
IOMMU group 4 00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)
IOMMU group 4 00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
IOMMU group 5 00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)   
IOMMU group 6 00:17.0 SATA controller [0106]: Intel Corporation Cannon Lake PCH SATA AHCI Controller [8086:a352] (rev 10)       
IOMMU group 7 00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a303] (rev 10)
IOMMU group 7 00:1f.3 Audio device [0403]: Intel Corporation Cannon Lake PCH cAVS [8086:a348] (rev 10)
IOMMU group 7 00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
IOMMU group 7 00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
IOMMU group 7 00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-V [8086:15bc] (rev 10)

I added x-vga=on, when I started the VM, host crashed. Had to reboot
 

fidays

New Member
May 15, 2021
18
4
3
32
When I start the KALI VM on the console this is the output. Also the VM takes a lot of time (3 minutes) here
I am not experienced enough to make sense of these.
Please can someone help me to figure out the actual issue.

1623213944484.png

then I do get this on the console
1623214036936.png


Also while this is going on on the console. here is the output of journalctl -b 0
Bash:
root@pmx1:~# journalctl -b 0
-- Logs begin at Wed 2021-06-09 10:16:20 IST, end at Wed 2021-06-09 10:19:15 IST. --
Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]Jun 09 10:16:20 pmx1 kernel: vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xa8000000-0xa9ffffff 64bit pref]lines 1-46


Then after a while, the console goes black and stays like that forever


On the task list bellow, this is what it shows
1623214486322.png
 

fidays

New Member
May 15, 2021
18
4
3
32
Also, this is happening.
The error log is growing up so fast, it is actually ending up using entire disk space.
1623214876387.png
 

leesteken

Famous Member
May 31, 2020
2,052
435
88
Can you post the VM configuration file /etc/pve/qemu-server/101.conf?
This post has the same error message, maybe the video= settings work can help?
I did not realize before but the IOMMU groups indicate that this is the only GPU in the system. If you are using this card for boot/host display, it needs more tricks for passthrough. Would it be possible for you to boot from another (integrated) GPU?

EDIT: Maybe this can help with BAR 3 issues?
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!