GPU pci pass-through help

[jorgeg73]

Hello to all. This is my setup
I just install proxmox
CPU ryzen 5 5600x
motherboard asrock steel legend
32 GB memory
512GB m.2 were proxmox is isntall.
4 HDD's, 1 512Gb, 320GB, 500GB and 1TB
I have also install a 2 port nic and a GPU GEFORCE VGA GTX 970.
I have follow several tutorial to put my GPU PCI-passthrough. None work what i whant to achive is video transcoding.
I have enable everything that i can in the motherboard, but nothing. I stalled in ext4.
I made a ubuntu VM and when i add PCI GPU i get
TASK ERROR: cannot prepare PCI pass-through, IOMMU not present
This is my grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafbff,efifbff"
root@pve:/etc# update-initramfs -u
update-initramfs: Generating /boot/initrd.img-5.15.83-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.

This is other printouts.
If someone can help me. Thank you

 

[leesteken]

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafbff,efifbff"

IOMMU is always enabled on AMD Ryzen in Proxmox, so you don't need amd_iommu=on. You most likely also don't need the other kernel parameters (and some are just invalid and do nothing).

TASK ERROR: cannot prepare PCI pass-through, IOMMU not present

If IOMMU is not working, it is because it is not (fully) Enabled in your motherboard BIOS. Also make sure you did all the steps like loading all vfio modules.

 

[jorgeg73]

IOMMU is always enabled on AMD Ryzen in Proxmox, so you don't need amd_iommu=on. You most likely also don't need the other kernel parameters (and some are just invalid and do nothing).

If IOMMU is not working, it is because it is not (fully) Enabled in your motherboard BIOS. Also make sure you did all the steps like loading all vfio modules.

. On the motherboard I have enable vt-d and virtualization I am also reading to see if something else need to be enable or disable. Now I just leave at GRUB_CMDLINE_LINUX_DEFAULT="quiet". ?
Thanks

 

[leesteken]

. On the motherboard I have enable vt-d and virtualization I am also reading to see if something else need to be enable or disable.

VT-d is Intel's name for IOMMU and this makes me confused. Several different motherboards are called ASRock Steel Legend, which one do you have. Do you have an Intel or and AMD system?

Now I just leave at GRUB_CMDLINE_LINUX_DEFAULT="quiet". ?

For AMD Ryzen, that's usually enough But for a single (or boot) GPU passthrough, you might need additional (and different than you had before) parameters. Are you sure that your system using GRUB and not systemd-boot? What is the output of cat /proc/cmdline? Did you read the manual about PCI(e) passthrough and load the modules?

 

[jorgeg73]

VT-d is Intel's name for IOMMU and this makes me confused. Several different motherboards are called ASRock Steel Legend, which one do you have. Do you have an Intel or and AMD system?

For AMD Ryzen, that's usually enough But for a single (or boot) GPU passthrough, you might need additional (and different than you had before) parameters. Are you sure that your system using GRUB and not systemd-boot? What is the output of cat /proc/cmdline? Did you read the manual about PCI(e) passthrough and load the modules?

Is a AMD, b550m steel legend. Yes I have read the manual and also different tutorials online. Is basically the same on all of them. I try whit cmdline too. I install it whit ext4 format. if I am not mistaking and read well I have to modify grub. I will put my printout later when I get home. But I also mess whit cmdline just in case, I commented out on grub what I did and then try cmdline. I don't know if I have to install some drivers in proxmo for the GPU?.
I appreciate the time and help, thank you. I am learning so be patient.

 

[Ventority]

Hey, I have a b450m, from asrock as well, so the bios shouldn't be that different i guess. I didn't get it, did you enable IOMMU support in your Bios? in addition, you can enable acs. It's under the Advanced Tab in AMD PBS (I think, I will look it up later). If you've done so, try to run

dmesg | grep -e DMAR -e IOMMU -e AMD-Vi

and

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done

and post your outputs.
EDIT:
No, you don't have to install any drivers in proxmox.

 

[jorgeg73]

Hey, I have a b450m from asrock as well so the bos shouldn't be that different i guess. I didn't get it, did you enable IOMMU support in your Bios? in addition, you can enable acs. It's under the Advanced Tab in AMD PBS (I think, I will look it up later). If you've done so, try to run

dmesg | grep -e DMAR -e IOMMU -e AMD-Vi

and

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done

and post your outputs.
EDIT:
No, you don't have to install any drivers in proxmox.

Thanks to you i was able to find the IOMMU option on BIOS i downloaded your MB manual en search for the IOMMU, on my manual if you search for it doesn't appear is buries under a lot of sobmenus. So now is enable. Here are my prin outs after i enable IOMMU. Later i will try to enable the GPU again whit this changes. Thanks again. If you find something in the print out, please let me now.


dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[ 0.677042] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.679918] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 0.679920] AMD-Vi: Extended features (0x58f77ef22294a5a): PPR NX GT IA PC GA_vAPIC
[ 0.679924] AMD-Vi: Interrupt remapping enabled
[ 0.680183] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; do
ne
IOMMU group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 0 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 0 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 0 01:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation PS5013 E13 NVMe Controller [1987:5013] (rev 01)
IOMMU group 0 02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ee]
IOMMU group 0 02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43eb]
IOMMU group 0 02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43e9]
IOMMU group 0 03:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU group 0 03:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU group 0 03:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU group 0 04:00.0 Non-Volatile memory controller [0108]: MAXIO Technology (Hangzhou) Ltd. NVMe SSD Controller MAP1202 [1e4b:1202] (rev 01)
IOMMU group 0 05:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe [14e4:165f]
IOMMU group 0 05:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe [14e4:165f]
IOMMU group 0 06:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
IOMMU group 10 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
IOMMU group 10 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
IOMMU group 10 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
IOMMU group 10 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
IOMMU group 10 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
IOMMU group 10 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
IOMMU group 10 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
IOMMU group 10 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
IOMMU group 11 08:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU group 12 09:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU group 13 09:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU group 14 09:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 15 09:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
IOMMU group 1 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 2 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 2 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 2 07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
IOMMU group 2 07:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
IOMMU group 3 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 4 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 5 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 6 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 7 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 8 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 9 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
IOMMU group 9 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
root@pve:~#

 

[Ventority]

I’ve looked through the output and everything seems to be alright. IOMMU groups should be separated enough even if the device 00:03:01 is in the same group as the gpu. If it still doesn’t work you can try the acs override patch but first we should try the first suggestion. Another question: Because you only have one gpu, this should be the gpu that displays your Bios, the grub boot loader and later the os, right? After you’ve booted proxmox, what is on the screen?

 

[jorgeg73]

I’ve looked through the output and everything seems to be alright. IOMMU groups should be separated enough even if the device 00:03:01 is in the same group as the gpu. If it still doesn’t work you can try the acs override patch but first we should try the first suggestion. Another question: Because you only have one gpu, this should be the gpu that displays your Bios, the grub boot loader and later the os, right? After you’ve booted proxmox, what is on the screen?

Its working now.

this should be the gpu that displays your Bios, the grub boot loader and later the os, right?

yes. GPU is the only way to display proxmox the onboard MB display does not display anything

After you’ve booted proxmox, what is on the screen?

The proxmox login, the IP to connect to the proxmiox GUI



@debian:~$ lspci | grep VGA
00:01.0 VGA compatible controller: Device 1234:1111 (rev 02)
01:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] (rev a1)

 

[Ventority]

So your passthrough is fully working?

 

[jorgeg73]

So your passthrough is fully working?

Yes. I have also made a passthrough to and lxc container. It took me until 2 am. I also install Nvidia graphics accelerator. install jellyfin on a docker container have not tested yet. Thanks again.
Let me see what else I can do.