Proxmox as an Ultimate Desktop Setup ( Dual Gpu Pass )

[rifaterdemsahin]

I am trying to set up a Proxmox ultimate workstation challenge for myself.

- Appreciate the feedback
- I would create a video on it when it becomes my daily driver.

Outlook / Physical


Goal :
- DevOps Workstation
- Video Editing Station
- Video Recording Station
- Video Streaming Station

Objective And Key Results
https://lucid.app/lucidchart/invitations/accept/inv_09128978-aa04-44f2-973c-ebce4a25d831


Hardware specs compute:
- CPU 128 thread 64 core AMD 3995x
- MOTHERBOARD Pro-WS-WRX80E-SAGE-SE-WIFI
- HARDDISC Intel Optane P900 450 GB
- HARDDISC Firecuda 1 TB Seagate
- RAM 96 GB ECC RAM(starters)
- GPU Radeon vii
- GPU Radeon 6900 xt
- MONITOR Samsung 49 Odessy
- Dark Base Pro V2 Case

Video Production Hardware specs:
- Sony A7 S3
- Camlik 4k ( 4 Hdmi captures )

Accessories
- Dummy HDMI adapter
- Dummy DisplayPort adapter
- Stream deck x2
- Atem mini pro
- Drop Alt USB c keyboard
- Razer Orb keyboard


Steps Setup For Programs
- install Proxmox from a USB
- set the main system on intel optane
- Install windows VM
- Install RealVNC pro on windows VM
- Install Linux boxes for different use cases ( different teams accounts, docker, Kubernetes )

Steps Proxmox Setup GPU Pass Through
- Go to Proxmox Workstation Console

- nano /etc/default/grub

Find what works for you in my case both AMD cards and passing through to VMS

video=vesafb:off video=efifb:off video=simplefb:off

https://imgur.com/a/Ix3vZpm

Grub as the main source of metadata of hardware

update-grub

- Reboot ( use the button )
- See VMs shutting down( give 5 minutes)

nano /etc/modules

Add to modules make the system understand that you are using virtual io action and get ready for passthrough

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

- Reboot ( use the button )
IOMMU remapping
The Input-Output Memory Management Unit (IOMMU) is a component in a memory controller that translates device virtual addresses

nano  /etc/modprobe.d/iommu_unsafe_interrupts.conf

Add
options vfio_iommu_type1 allow_unsafe_interrupts=1

Enable Nested Virtualization

nano /etc/modprobe.d/kvm.conf

Add
options kvm ignore_msrs=1

Blacklist Drivers> you only want the windows main to be passed for you to work not the base Debian console

nano /etc/modprobe.d/blacklist.conf

Add

blacklist radeon
blacklist nouveau
blacklist nvidia
blacklist nvidiafb

Find your GPUs
2 Cards / 4 numbers popping up ignore the audio

lspci -v

lspci -n -s 03:00;lspci -n -s 43:00

Get Output for the GPU Number that the AMD Vendor provided for my system

03:00.0 0300: 1002:66af (rev c1)
03:00.1 0403: 1002:ab20
43:00.0 0300: 1002:73bf (rev c0)
43:00.1 0403: 1002:ab28

Tell vfio what you want to passthrough
Virtual Function I/O (or VFIO) allows a virtual machine (VM) direct access to a PCI hardware resource, such as a graphics processing unit (GPU). Virtual machines with setting up GPU passthrough can gain close to the bare metal performance, which makes running games in a Windows virtual machine possible.

nano /etc/modprobe.d/vfio.conf

options vfio-pci ids=1002:66af,1002:ab20,1002:73bf,1002:ab28 disable_vga=1

live-update-initramfs is a wrapper replacing the update-initramfs command provided by the initramfs-tools package. In live-systems with a persistence layer, live-update-initramfs will write out updated kernel and initrd images to the live media.

update-initramfs -u

- Reboot
- Get your windows Keys
-- Windows Pro 11 ( With Remote Access )
-- Windows Server 2022 DataCenter ( If Proxmox Fails and Daily Driver with a more stable system )

SetUp Windows Virtual Machine
https://youtu.be/5ce-CcYjqe8?t=733
- Set the Discs on Sata not IDE ( multiple throws errors)
- Enable Remote Desktop
- Add proxmox hardware network card ( if vfio does not work )
- Get IP

ipconfig /all

- Add proxmox hardware Graphics( check all )
- Add proxmox hardware mouse
- Add proxmox hardware keyboard
- Do windows updates
- Restart ( check if the proxmox oranges changes have been applied )

- Install the graphics drivers ( Check the Devices )

Software
- Access > RealVNC pro
- Recording > Xsplit
- Editing > Adobe Premiere Pro
- Coding > VSCode
- Vmware Workstation > Run Windows Development Environment

Operating Systems and Use Cases
- Windows 11 Pro > Real VNC Host
- Ubuntu 22 > Teams / Kubernetes / Docker
- Windows Server 2022 > Alternative sandbox if proxmox Fails


Todos
- increase remote access resolution
- System-wide backup to 50 TB Synology NAS

GPU I USE CASE
Radeon 6900 XT ( Run Black Lion )
Windows Server 2022
Desktop with all tools

GPU II USE CASE
Radeon v7 ( Davinci Render Box )


Diagnosis
Radeon v7 monitor
https://imgur.com/a/g5pOfjR

Tips from Erdem
- Get Fast Discs Big file moves
- Disaster If All Fails to Work as a Workstation Rollback to Windows Server 2022 DataCenter
- Get a basic blue tooth USB to get audio
- move from CPU KVM to host login issue is gone.
- Don't do GPU pass on windows 2022 server / not support on the AMD side
- If the keyboard is problematic. Make sure it is only passed to one VM
- Don't pass more than 5 USBs
- Pass the PCI express and the controller and connect the hubs to one PCIe card
- Radeon 7 needs a reset-bug to be implemented.
- Radeon 7 works with display none in windows11 ( get the remote desktop IP and put it to the notes )
- Set AMD to quiet in the grub if you got AMD cards

Reset Bug

apt-get update
apt install pve-kernel-5.15

crontab -e

0 5 * * 1 echo 'device_specific' > /sys/bus/pci/devices/1002:ab28/reset_method
0 5 * * 1 echo 'device_specific' > /sys/bus/pci/devices/1002:66af/reset_method

Vm Debug Commands

 dmesg | grep AMD-Vi

Host Debug Commands

lspci -nnk
dmesg -w
dmesg | grep -e DMAR -e IOMMU
pveversion -v

https://github.com/rifaterdemsahin/workstation/blob/master/proxmox/diagnosis.txt

Config File 9800 XT ( GPU I )

root@workstation:/etc/pve/qemu-server# cat 100.conf
#realvnc based action > login to the box
#use the usb monitor as backup and better resolution when needed
#serial works in here
#no guest agent error!
bios: ovmf
boot: order=virtio0;net0;ide0;ide2
cores: 24
cpu: host
efidisk0: zfsonfirecuda:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:43:00,pcie=1,x-vga=1
hostpci1: 0000:44:00,pcie=1
hostpci2: 0000:65:00,pcie=1
ide2: local:iso/virtio-win-0.1.215.iso,media=cdrom,size=528322K
machine: pc-q35-6.1
memory: 22048
meta: creation-qemu=6.1.0,ctime=1651135806
name: BlackLionWP11Screens
net0: virtio=22:CE:C5:98:69:EA,bridge=vmbr0,firewall=1
net1: rtl8139=CE:A0:66:CF:54:D5,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win11
parent: acronisbackup
runningcpu: kvm64,enforce,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep
runningmachine: pc-q35-6.1+pve0
scsihw: virtio-scsi-pci
smbios1: uuid=2e349ac4-75bc-494f-aaea-a40cb31de45d
sockets: 2
startup: order=1
tpmstate0: zfsonfirecuda:vm-100-disk-1,size=4M,version=v2.0
usb0: host=04d8:eed2,usb3=1
usb1: host=17e9:4300,usb3=1
usb2: host=0bda:2550,usb3=1
usb3: host=145f:01ad,usb3=1
usb4: host=1235:8210,usb3=1
virtio0: zfsonfirecuda:vm-100-disk-2,size=128G
vmgenid: a04f4117-2219-4865-bed3-757ecc6bfb05

[acronisbackup]
bios: ovmf
boot: order=virtio0;net0;ide0;ide2
cores: 24
cpu: host
efidisk0: zfsonfirecuda:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:43:00,pcie=1,x-vga=1
hostpci1: 0000:44:00,pcie=1
hostpci2: 0000:65:00,pcie=1
ide2: local:iso/virtio-win-0.1.215.iso,media=cdrom,size=528322K
machine: pc-q35-6.1
memory: 22048
meta: creation-qemu=6.1.0,ctime=1651135806
name: BlackLionWP11Screens
net0: virtio=22:CE:C5:98:69:EA,bridge=vmbr0,firewall=1
net1: rtl8139=CE:A0:66:CF:54:D5,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
parent: audiosolvewithbluetooth
runningcpu: kvm64,enforce,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep
runningmachine: pc-q35-6.1+pve0
scsihw: virtio-scsi-pci
smbios1: uuid=2e349ac4-75bc-494f-aaea-a40cb31de45d
snaptime: 1651363383
sockets: 2
tpmstate0: zfsonfirecuda:vm-100-disk-1,size=4M,version=v2.0
usb0: host=04d8:eed2,usb3=1
usb1: host=17e9:4300,usb3=1
usb2: host=0bda:2550,usb3=1
usb3: host=145f:01ad,usb3=1
usb4: host=1235:8210,usb3=1
virtio0: zfsonfirecuda:vm-100-disk-2,size=128G
vmgenid: a04f4117-2219-4865-bed3-757ecc6bfb05
vmstate: zfsonfirecuda:vm-100-state-acronisbackup

[audiosolvewithbluetooth]
#beforeprogramsinstall
bios: ovmf
boot: order=virtio0;net0;ide0;ide2
cores: 24
cpu: host
efidisk0: zfsonfirecuda:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:43:00,pcie=1,x-vga=1
ide2: local:iso/virtio-win-0.1.215.iso,media=cdrom,size=528322K
machine: pc-q35-6.1
memory: 22048
meta: creation-qemu=6.1.0,ctime=1651135806
name: erdemwindowspro11
net0: virtio=22:CE:C5:98:69:EA,bridge=vmbr0,firewall=1
net1: rtl8139=CE:A0:66:CF:54:D5,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win11
parent: workswith5screens
runningcpu: kvm64,enforce,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep
runningmachine: pc-q35-6.1+pve0
scsihw: virtio-scsi-pci
smbios1: uuid=2e349ac4-75bc-494f-aaea-a40cb31de45d
snaptime: 1651347816
sockets: 2
tpmstate0: zfsonfirecuda:vm-100-disk-1,size=4M,version=v2.0
usb0: host=04d8:eed2,usb3=1
usb1: host=17e9:4300,usb3=1
usb2: host=0bda:2550,usb3=1
usb3: host=145f:01ad,usb3=1
virtio0: zfsonfirecuda:vm-100-disk-2,size=128G
vmgenid: a04f4117-2219-4865-bed3-757ecc6bfb05
vmstate: zfsonfirecuda:vm-100-state-audiosolvewithbluetooth

[realvncfearlicencewindows]
bios: ovmf
boot: order=virtio0;ide2;net0;ide0
cores: 24
efidisk0: zfsonfirecuda:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
ide0: local:iso/virtio-win-0.1.217.iso,media=cdrom,size=519096K
ide2: local:iso/Win11_EnglishInternational_x64v1.iso,media=cdrom
machine: pc-q35-6.1
memory: 22048
meta: creation-qemu=6.1.0,ctime=1651135806
name: erdemwindowspro11
net0: virtio=22:CE:C5:98:69:EA,bridge=vmbr0,firewall=1
net1: rtl8139=CE:A0:66:CF:54:D5,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win11
runningcpu: kvm64,enforce,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep
runningmachine: pc-q35-6.1+pve0
scsihw: virtio-scsi-pci
smbios1: uuid=2e349ac4-75bc-494f-aaea-a40cb31de45d
snaptime: 1651304210
sockets: 2
tpmstate0: zfsonfirecuda:vm-100-disk-1,size=4M,version=v2.0
usb1: host=17e9:4300,usb3=1
usb2: host=0bda:2550,usb3=1
usb3: host=04d8:eed2,usb3=1
virtio0: zfsonfirecuda:vm-100-disk-2,size=128G
vmgenid: a04f4117-2219-4865-bed3-757ecc6bfb05
vmstate: zfsonfirecuda:vm-100-state-realvncfearlicencewindows

[workswith5screens]
bios: ovmf
boot: order=virtio0;net0;ide0;ide2
cores: 24
cpu: host
efidisk0: zfsonfirecuda:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:43:00,pcie=1,x-vga=1
ide2: local:iso/virtio-win-0.1.215.iso,media=cdrom,size=528322K
machine: pc-q35-6.1
memory: 22048
meta: creation-qemu=6.1.0,ctime=1651135806
name: erdemwindowspro11
net0: virtio=22:CE:C5:98:69:EA,bridge=vmbr0,firewall=1
net1: rtl8139=CE:A0:66:CF:54:D5,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win11
parent: realvncfearlicencewindows
runningcpu: kvm64,enforce,hv_ipi,hv_relaxed,hv_reset,hv_runtime,hv_spinlocks=0x1fff,hv_stimer,hv_synic,hv_time,hv_vapic,hv_vpindex,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep
runningmachine: pc-q35-6.1+pve0
scsihw: virtio-scsi-pci
smbios1: uuid=2e349ac4-75bc-494f-aaea-a40cb31de45d
snaptime: 1651347364
sockets: 2
tpmstate0: zfsonfirecuda:vm-100-disk-1,size=4M,version=v2.0
usb0: host=04d8:eed2,usb3=1
usb1: host=17e9:4300,usb3=1
usb2: host=0bda:2550,usb3=1
usb3: host=145f:01ad,usb3=1
virtio0: zfsonfirecuda:vm-100-disk-2,size=128G
vmgenid: a04f4117-2219-4865-bed3-757ecc6bfb05
vmstate: zfsonfirecuda:vm-100-state-workswith5screens
root@workstation:/etc/pve/qemu-server#

Config File Radeon 7 ( GPU II )

root@workstation:/etc/pve/qemu-server# cat 111.conf
agent: 1
bios: ovmf
boot: order=sata0;net0;ide2
cores: 4
cpu: host
efidisk0: local-lvm:vm-111-disk-1,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:03:00,pcie=1,x-vga=1
ide2: local:iso/virtio-win-0.1.217.iso,media=cdrom,size=519096K
machine: pc-q35-6.1
memory: 12048
meta: creation-qemu=6.1.0,ctime=1651350125
name: BlueLionWindows11Games
net0: rtl8139=26:5E:27:F5:96:0C,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win11
sata0: local-lvm:vm-111-disk-0,cache=writeback,size=64G
scsihw: virtio-scsi-pci
smbios1: uuid=0e40f7ee-7dea-478c-a11b-542ec7491959
sockets: 4
startup: order=4
tpmstate0: local-lvm:vm-111-disk-2,size=4M,version=v2.0
vga: none
vmgenid: e37c155c-d84b-4d8d-af7c-dbd29ea50982
root@workstation:/etc/pve/qemu-server#

End Result
- 2 AMD GPU passed
- 6 monitors worked
- Audio Worked
- XSplit Recording worked

Concepts
IOMMU : The Input-Output Memory Management Unit (IOMMU)
vfio : The VFIO driver framework provides unified APIs for direct device access.
dmesg: dmesg is used to examine or control the kernel ring buffer. The default action is to display all messages from the kernel ring buffer.

Commitment
it took 1 week 16 hours

References
Amd Gpu Pass Concepts
https://matt.olan.me/proxmox-gpu-passthrough-guide/
Check IOMMU Code
https://3os.org/infrastructure/prox...vm/#proxmox-configuration-for-gpu-passthrough
Code For the Proxmox Configs/Commands
https://github.com/rifaterdemsahin/workstation/tree/master/proxmox
Memory Error
https://forums.unraid.net/topic/994...0-cant-reserve/?do=findComment&comment=923145
AMD Settings and Flags
https://tomschlick.com/defeating-proxmox-gpu-passthrough-fix/
Reset Bug implementation
https://www.nicksherlock.com/2020/11/working-around-the-amd-gpu-reset-bug-on-proxmox/
Remote Desktop Tool
https://www.realvnc.com
Motherboard
https://www.asus.com/Motherboards-Components/Motherboards/Workstation/Pro-WS-WRX80E-SAGE-SE-WIFI/
Case
https://www.bequiet.com/en/case/1472
Research
https://forum.proxmox.com/threads/gpu-passthrough-radeon-6800xt-and-beyond.86932/#post-467263
https://www.youtube.com/watch?v=5ce-CcYjqe8

Community Contributions
Leesketen
iommu_groups to see

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done

pcie_acs_override which invalidates the output of the IOMMU groups.

 cat /proc/cmdline

 

[Dunuin]

I personally really don't like VNC. Its quite laggy, got alot of artefacts and Sound isn`t always working. You might want to try Parsec or SPICE. Parsec got the best latency and graphics so far of the remote desktop protocols I've tested and SPICE is nice because you can use USB devices remotely.

 

[rifaterdemsahin]

I personally really don't like VNC. Its quite laggy, got alot of artefacts and Sound isn`t always working. You might want to try Parsec or SPICE. Parsec got the best latency and graphics so far of the remote desktop protocols I've tested and SPICE is nice because you can use USB devices remotely.

Thanks for the feedback.

I am curious about parsec and spice I would try them in the wework office as they are in the same network.

I need to use them from my home. RealVnc is 3 dollars which is ok
https://www.realvnc.com/en/connect/pricing

Plus: RealVNC is my next-door neighbour here.

 

[rifaterdemsahin]

@leesteken

root@workstation:~# for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
IOMMU group 0 60:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 10 60:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 11 60:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 12 60:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 13 60:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 14 61:00.0 Non-Volatile memory controller [0108]: Seagate Technology PLC FireCuda 530 SSD [1bb1:5018] (rev 01)
IOMMU group 15 62:00.0 Non-Volatile memory controller [0108]: Seagate Technology PLC FireCuda 530 SSD [1bb1:5018] (rev 01)
IOMMU group 16 63:00.0 Non-Volatile memory controller [0108]: Seagate Technology PLC FireCuda 530 SSD [1bb1:5018] (rev 01)
IOMMU group 17 64:00.0 Non-Volatile memory controller [0108]: Seagate Technology PLC FireCuda 530 SSD [1bb1:5018] (rev 01)
IOMMU group 18 65:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller [1b21:2142]
IOMMU group 19 66:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU group 1 60:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 20 67:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU group 21 40:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 22 40:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 23 40:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 24 40:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 25 40:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 26 40:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 27 40:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 28 40:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 29 40:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 2 60:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 30 40:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 31 40:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 32 41:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c0)
IOMMU group 33 42:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
IOMMU group 34 43:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c0)
IOMMU group 35 43:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]
IOMMU group 36 44:00.0 Multimedia controller [0480]: YUAN High-Tech Development Co., Ltd. Device [12ab:0710]
IOMMU group 37 45:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU group 38 46:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU group 39 20:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 3 60:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 40 20:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 41 20:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 42 20:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 43 20:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 44 20:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 45 20:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 46 20:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 47 20:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 48 20:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 49 20:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 4 60:01.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 50 20:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 51 21:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream [1022:57ad]
IOMMU group 52 22:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU group 53 22:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU group 54 22:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU group 55 22:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU group 56 22:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU group 57 22:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU group 58 22:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU group 59 22:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU group 5 60:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 60 22:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
IOMMU group 61 23:00.0 Non-Volatile memory controller [0108]: Intel Corporation Optane SSD 900P Series [8086:2700]
IOMMU group 62 24:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller [1b21:3242]
IOMMU group 63 25:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10G X550T [8086:1563] (rev 01)
IOMMU group 64 25:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10G X550T [8086:1563] (rev 01)
IOMMU group 65 26:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX200 [8086:2723] (rev 1a)
IOMMU group 66 27:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU group 67 28:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU group 68 29:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU group 69 29:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 6 60:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 70 29:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU group 71 2a:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU group 72 2b:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU group 73 2c:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU group 74 2d:00.0 Non-Volatile memory controller [0108]: Sandisk Corp WD PC SN810 / Black SN850 NVMe SSD [15b7:5011] (rev 01)
IOMMU group 75 2e:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU group 76 2f:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU group 77 2f:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU group 78 2f:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c]
IOMMU group 79 2f:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
IOMMU group 7 60:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 80 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 81 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 82 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 83 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU group 84 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 85 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 86 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 87 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 88 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 89 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU group 8 60:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU group 90 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
IOMMU group 90 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU group 91 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 0 [1022:1490]IOMMU group 91 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 1 [1022:1491]IOMMU group 91 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 2 [1022:1492]IOMMU group 91 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 3 [1022:1493]IOMMU group 91 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 4 [1022:1494]IOMMU group 91 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 5 [1022:1495]IOMMU group 91 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 6 [1022:1496]IOMMU group 91 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 7 [1022:1497]IOMMU group 92 01:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:14a0] (rev c1)
IOMMU group 93 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:14a1]
IOMMU group 94 03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 20 [Radeon VII] [1002:66af] (rev c1)
IOMMU group 95 03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 20 HDMI Audio [Radeon VII] [1002:ab20]
IOMMU group 96 04:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU group 9 60:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]

 

[rifaterdemsahin]

@leesteken

cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.15.30-2-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off video=efifb:off video=simplefb:off
root@workstation:~#

 

[rifaterdemsahin]

@leesteken current error

 

[rifaterdemsahin]

For some reason my result is all empty

 

[rifaterdemsahin]

https://pve.proxmox.com/wiki/PCI(e)_Passthrough
https://askubuntu.com/questions/1080217/how-to-blacklist-amdgpu-driver

 

[leesteken]

The current status of your setup is not clear to me. What problem(s) are you currently having?

 

[rifaterdemsahin]

https://twitter.com/x/status/1521036261343084544

https://twitter.com/rifaterdemsahin/status/1521036261343084544

Error

 

[leesteken]

Sorry, I don't have twitter. What is the output of cat /proc/iomem? Can you attach /etc/default/grub, /etc/modules and all files from the directory /etc/modprobe.d/?

 

[rifaterdemsahin]

Sorry, I don't have twitter. What is the output of cat /proc/iomem? Can you attach /etc/default/grub, /etc/modules and all files from the directory /etc/modprobe.d/?

Video of the Error > https://youtu.be/8pdLlRCHoZ0

root@workstation:~# cat /proc/iomem
00000000-00000fff : Reserved
00001000-0008ffff : System RAM
00090000-00090fff : Unknown E820 type
00091000-0009ffff : System RAM
000a0000-000fffff : Reserved
  00000000-00000000 : PCI Bus 0000:00
  000a0000-000dffff : PCI Bus 0000:00
  000f0000-000fffff : System ROM
00100000-03ffffff : System RAM
04000000-04013fff : ACPI Non-volatile Storage
04014000-09df1fff : System RAM
09df2000-09ffffff : Reserved
0a000000-b1c33fff : System RAM
b1c34000-b1c79fff : Reserved
b1c7a000-b3f16fff : System RAM
b3f17000-b3f17fff : Reserved
b3f18000-b506bfff : System RAM
b506c000-b89b1fff : Reserved
  b8950018-b8950019 : APEI ERST
  b895001c-b8950021 : APEI ERST
  b8950028-b8950039 : APEI ERST
  b8950040-b895004c : APEI ERST
  b8950050-b895204f : APEI ERST
b89b2000-b8d99fff : ACPI Tables
b8d9a000-b91dcfff : ACPI Non-volatile Storage
b91dd000-bb13ffff : Reserved
bb140000-bb1fefff : Unknown E820 type
bb1ff000-bbffffff : System RAM
bc000000-bfffffff : Reserved
c0000000-c2fffffe : PCI Bus 0000:20
  c0000000-c11fffff : PCI Bus 0000:21
    c0000000-c11fffff : PCI Bus 0000:22
      c0000000-c00fffff : PCI Bus 0000:23
        c0000000-c0003fff : 0000:23:00.0
          c0000000-c0003fff : nvme
      c0100000-c01fffff : PCI Bus 0000:24
        c0100000-c0107fff : 0000:24:00.0
          c0100000-c0107fff : xhci-hcd
      c0200000-c0afffff : PCI Bus 0000:25
        c0200000-c02fffff : 0000:25:00.0
        c0300000-c03fffff : 0000:25:00.0
        c0400000-c04fffff : 0000:25:00.1
        c0500000-c05fffff : 0000:25:00.1
      c0b00000-c0bfffff : PCI Bus 0000:26
        c0b00000-c0b03fff : 0000:26:00.0
          c0b00000-c0b03fff : iwlwifi
      c0c00000-c0cfffff : PCI Bus 0000:27
        c0c00000-c0c001ff : 0000:27:00.0
          c0c00000-c0c001ff : ahci
      c0d00000-c0dfffff : PCI Bus 0000:28
        c0d00000-c0d001ff : 0000:28:00.0
          c0d00000-c0d001ff : ahci
      c0e00000-c0ffffff : PCI Bus 0000:29
        c0e00000-c0efffff : 0000:29:00.1
          c0e00000-c0efffff : xhci-hcd
        c0f00000-c0ffffff : 0000:29:00.3
          c0f00000-c0ffffff : xhci-hcd
      c1000000-c10fffff : PCI Bus 0000:2a
        c1000000-c10007ff : 0000:2a:00.0
          c1000000-c10007ff : ahci
      c1100000-c11fffff : PCI Bus 0000:2b
        c1100000-c11007ff : 0000:2b:00.0
          c1100000-c11007ff : ahci
  c2a00000-c2cfffff : PCI Bus 0000:2f
    c2a00000-c2afffff : 0000:2f:00.3
      c2a00000-c2afffff : xhci-hcd
    c2b00000-c2bfffff : 0000:2f:00.1
      c2b00000-c2bfffff : ccp
    c2c00000-c2c07fff : 0000:2f:00.4
    c2c08000-c2c09fff : 0000:2f:00.1
      c2c08000-c2c09fff : ccp
  c2d00000-c2dfffff : PCI Bus 0000:2d
    c2d00000-c2d03fff : 0000:2d:00.0
      c2d00000-c2d03fff : nvme
  c2e00000-c2efffff : PCI Bus 0000:2c
    c2e00000-c2e03fff : 0000:2c:00.0
      c2e00000-c2e03fff : nvme
c4180000-c41fffff : amd_iommu
c4200000-c4200fff : Reserved
  c4200000-c42003ff : IOAPIC 3
c5100000-c517ffff : amd_iommu
c5180000-c5180fff : Reserved
  c5180000-c51803ff : IOAPIC 2
c6000000-c85ffffe : PCI Bus 0000:40
  c8000000-c82fffff : PCI Bus 0000:41
    c8000000-c81fffff : PCI Bus 0000:42
      c8000000-c81fffff : PCI Bus 0000:43
        c8000000-c80fffff : 0000:43:00.0
        c8100000-c811ffff : 0000:43:00.0
        c8120000-c8123fff : 0000:43:00.1
    c8200000-c8203fff : 0000:41:00.0
  c8300000-c84fffff : PCI Bus 0000:44
    c8300000-c83fffff : 0000:44:00.0
    c8400000-c840ffff : 0000:44:00.0
ca000000-cc5ffffe : PCI Bus 0000:60
  cc000000-cc0fffff : PCI Bus 0000:65
    cc000000-cc007fff : 0000:65:00.0
  cc100000-cc1fffff : PCI Bus 0000:64
    cc100000-cc103fff : 0000:64:00.0
      cc100000-cc103fff : nvme
  cc200000-cc2fffff : PCI Bus 0000:63
    cc200000-cc203fff : 0000:63:00.0
      cc200000-cc203fff : nvme
  cc300000-cc3fffff : PCI Bus 0000:62
    cc300000-cc303fff : 0000:62:00.0
      cc300000-cc303fff : nvme
  cc400000-cc4fffff : PCI Bus 0000:61
    cc400000-cc403fff : 0000:61:00.0
      cc400000-cc403fff : nvme
ce100000-ce17ffff : amd_iommu
ce200000-ce300fff : Reserved
  ce300000-ce3003ff : IOAPIC 1
ce400000-ce4fffff : Reserved
e0000000-efffffff : PCI MMCONFIG 0000 [bus 00-ff]
  e0000000-efffffff : pnp 00:01
f0000000-f22ffffe : PCI Bus 0000:00
  f2000000-f21fffff : PCI Bus 0000:01
    f2000000-f20fffff : PCI Bus 0000:02
      f2000000-f20fffff : PCI Bus 0000:03
        f2000000-f207ffff : 0000:03:00.0
          f2000000-f207ffff : vfio-pci
        f2080000-f209ffff : 0000:03:00.0
        f20a0000-f20a3fff : 0000:03:00.1
          f20a0000-f20a3fff : vfio-pci
    f2100000-f2103fff : 0000:01:00.0
f4100000-f41fffff : Reserved
  f4100000-f41fffff : pnp 00:00
f4200000-f427ffff : amd_iommu
f4280000-f4280fff : Reserved
  f4280000-f42803ff : IOAPIC 4
fea00000-feafffff : Reserved
fec00000-fec00fff : Reserved
  fec00000-fec003ff : IOAPIC 0
fec10000-fec10fff : Reserved
  fec10000-fec10fff : pnp 00:06
fed00000-fed00fff : Reserved
  fed00000-fed003ff : HPET 0
    fed00000-fed003ff : PNP0103:00
fed40000-fed44fff : Reserved
fed80000-fed8ffff : Reserved
  fed81500-fed818ff : AMDI0030:00
fedc0000-fedc0fff : pnp 00:06
fedc2000-fedcffff : Reserved
  fedc3000-fedc3fff : AMDI0010:01
    fedc3000-fedc3fff : AMDI0010:01 AMDI0010:01
fedd4000-fedd5fff : Reserved
fee00000-fee00fff : Local APIC
  fee00000-fee00fff : pnp 00:06
ff000000-ffffffff : Reserved
  ff000000-ffffffff : pnp 00:06
100000000-103fefffff : System RAM
103ff00000-103fffffff : Reserved
1040000000-183f0fffff : System RAM
  1119000000-111a0025bf : Kernel code
  111a200000-111abf0fff : Kernel rodata
  111ac00000-111b03e4bf : Kernel data
  111b336000-111b7fffff : Kernel bss
183f100000-183fffffff : Reserved
10021000000-14020ffffff : PCI Bus 0000:60
14021000000-18020ffffff : PCI Bus 0000:40
  18010000000-180201fffff : PCI Bus 0000:41
    18010000000-180201fffff : PCI Bus 0000:42
      18010000000-180201fffff : PCI Bus 0000:43
        18010000000-1801fffffff : 0000:43:00.0
          18010000000-180102fffff : BOOTFB
        18020000000-180201fffff : 0000:43:00.0
18081000000-1c080ffffff : PCI Bus 0000:20
  1c080400000-1c080cfffff : PCI Bus 0000:21
    1c080400000-1c080cfffff : PCI Bus 0000:22
      1c080400000-1c080cfffff : PCI Bus 0000:25
        1c080400000-1c0807fffff : 0000:25:00.1
          1c080400000-1c0807fffff : ixgbe
        1c080800000-1c080bfffff : 0000:25:00.0
          1c080800000-1c080bfffff : ixgbe
        1c080c00000-1c080c03fff : 0000:25:00.1
          1c080c00000-1c080c03fff : ixgbe
        1c080c04000-1c080c07fff : 0000:25:00.0
          1c080c04000-1c080c07fff : ixgbe
1c081000000-20080ffffff : PCI Bus 0000:00
  20070000000-200801fffff : PCI Bus 0000:01
    20070000000-200801fffff : PCI Bus 0000:02
      20070000000-200801fffff : PCI Bus 0000:03
        20070000000-2007fffffff : 0000:03:00.0
          20070000000-2007fffffff : vfio-pci
        20080000000-200801fffff : 0000:03:00.0
          20080000000-200801fffff : vfio-pci
root@workstation:~#

root@workstation:~# cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
#GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off"
#GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off video=efifb:off"
#GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vga video=efifb:off"
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off video=efifb:off video=simplefb:off"
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entrie
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"
root@workstation:~#

root@workstation:~# cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
vendor-reset
root@workstation:~#

root@workstation:~# ls /etc/modprobe.d
blacklist.conf  dkms.conf  iommu_unsafe_interrupts.conf  kvm.conf  pve-blacklist.conf  vfio.conf
root@workstation:~#

 

[leesteken]

14021000000-18020ffffff : PCI Bus 0000:40 18010000000-180201fffff : PCI Bus 0000:41 18010000000-180201fffff : PCI Bus 0000:42 18010000000-180201fffff : PCI Bus 0000:43 18010000000-1801fffffff : 0000:43:00.0 18010000000-180102fffff : BOOTFB 18020000000-180201fffff : 0000:43:00.0

The boot display has not released the memory. If all the video=... don't work then remove them all and don't blacklist amdgpu because (in my experience) amdgpu can take over from BOOTFB and release it properly to vfio_pci.

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off video=efifb:off video=simplefb:off"

Have you tried GRUB_CMDLINE_LINUX_DEFAULT="iommu=pt pcie_acs_override=downstream,multifunction" and un-blacklisting amdgpu instead? In my experience this works better (with recent PVE kernels) and allows you to see Proxmox boot messages before starting the VM.

root@workstation:~# ls /etc/modprobe.d blacklist.conf dkms.conf iommu_unsafe_interrupts.conf kvm.conf pve-blacklist.conf vfio.conf

Please attach the files in that directory. What is the contents of blacklist.conf, dkms.conf, iommu_unsafe_interrupts.conf, kvm.conf, pve-blacklist.conf and vfio.conf? Is there blacklisting of amdgpu one or more times? Are there early bindings to vfio_pci? Ths is important to know for my suggestion above.

 

[rifaterdemsahin]

The boot display has not released the memory. If all the video=... don't work then remove them all and don't blacklist amdgpu because (in my experience) amdgpu can take over from BOOTFB and release it properly to vfio_pci.

Have you tried GRUB_CMDLINE_LINUX_DEFAULT="iommu=pt pcie_acs_override=downstream,multifunction" and un-blacklisting amdgpu instead? In my experience this works better (with recent PVE kernels) and allows you to see Proxmox boot messages before starting the VM.

Please attach the files in that directory. What is the contents of blacklist.conf, dkms.conf, iommu_unsafe_interrupts.conf, kvm.conf, pve-blacklist.conf and vfio.conf? Is there blacklisting of amdgpu one or more times? Are there early bindings to vfio_pci? Ths is important to know for my suggestion above.

Should I remove the blacklisting AMD and reapply it ?
Or just remove the AMD black listing?
I am open to try

root@workstation:/etc/modprobe.d# ls
blacklist.conf  dkms.conf  iommu_unsafe_interrupts.conf  kvm.conf  pve-blacklist.conf  vfio.conf
root@workstation:/etc/modprobe.d# cat blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia
blacklist nvidiafb
root@workstation:/etc/modprobe.d# cat dkms.conf
# modprobe information used for DKMS modules
#
# This is a stub file, should be edited when needed,
# used by default by DKMS.
root@workstation:/etc/modprobe.d# cat iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1
root@workstation:/etc/modprobe.d# cat kvm.conf
options kvm ignore_msrs=1
root@workstation:/etc/modprobe.d# cat pve-blacklist.conf
# This file contains a list of modules which are not supported by Proxmox VE

# nidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
root@workstation:/etc/modprobe.d# cat vfio.conf
options kvm ignore_msrs=1
options vfio-pci ids=1002:66af,1002:ab20,1002:73bf,1002:ab28 disable_vga=1
softdep amdgpu pre: vfio vfio_pci
root@workstation:/etc/modprobe.d#

 

[leesteken]

root@workstation:/etc/modprobe.d# cat vfio.conf
options kvm ignore_msrs=1
options vfio-pci ids=1002:66af,1002:ab20,1002:73bf,1002:ab28 disable_vga=1
softdep amdgpu pre: vfio vfio_pci

If you want to try my suggestion of using amdgpu, please don't bind the 43:00.0 device of vfio-pci. I don't know which of those four numbers it is, but you can check with lspci -nns 43:00.0 and remove it (and don't forget update-initframfs -u).

Note that your have options kvm ignore_msrs=1 twice, so you can safely remove /etc/modprobe.d/kvm.conf. And you have blacklist nvidiafb twice, which you don't even need, so you can safely remove /etc/modprobe.d/blacklist.conf.

 

[leesteken]

Have you tried GRUB_CMDLINE_LINUX_DEFAULT="iommu=pt pcie_acs_override=downstream,multifunction" and un-blacklisting amdgpu instead? In my experience this works better (with recent PVE kernels) and allows you to see Proxmox boot messages before starting the VM.

You are using AMD Threadripper (which has many PCIe lanes comming from the CPU) and I think you really don't need pcie_acs_override=downstream,multifunction at all. Can you remove this and show the IOMMU groups? This only leaves iommu=pt and that is probably no necessary either...

 

[rifaterdemsahin]

Mate thanks for your comments. I am new to proxmox and appreciate the help
here is the 6900 XT

root@workstation:/etc/modprobe.d# lspci -nns 43:00.0
43:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c0)
root@workstation:/etc/modprobe.d# lspci -nns 43:00.1
43:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]
root@workstation:/etc/modprobe.d#

updated now vfio.conf

root@workstation:/etc/modprobe.d# update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-5.15.30-2-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-5.13.19-2-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
root@workstation:/etc/modprobe.d# cat vfio.conf
options kvm ignore_msrs=1
options vfio-pci ids=1002:66af,1002:ab20 disable_vga=1
softdep amdgpu pre: vfio vfio_pci
root@workstation:/etc/modprobe.d#

removed kvm.conf

root@workstation:/etc/modprobe.d# rm kvm.conf
root@workstation:/etc/modprobe.d# ls
blacklist.conf  dkms.conf  iommu_unsafe_interrupts.conf  pve-blacklist.conf  vfio.conf
root@workstation:/etc/modprobe.d#

removed the blacklist

root@workstation:/etc/modprobe.d# rm blacklist.conf
root@workstation:/etc/modprobe.d# ls
dkms.conf  iommu_unsafe_interrupts.conf  pve-blacklist.conf  vfio.conf
root@workstation:/etc/modprobe.d#

updated the grub

GRUB_CMDLINE_LINUX_DEFAULT="iommu=pt pcie_acs_override=downstream,multifunction"

update-grub

update-initramfs -u -k all

Rebooted and waiting for it to turn on....( I would update with results )

 

[rifaterdemsahin]

@leesteken wooooow mate

THANKS, A MILLION

feel free to add me at Linkedin https://www.linkedin.com/in/rifaterdemsahin/

Got zillion questions but now enjoying it.
I got to update the top documentation for the dual gpu,
I have gaps in the FIRST PRINCIPLES

 

[rifaterdemsahin]

 

[leesteken]

I would suggest not using Primary GPU (x-vga=1) with AMD GPUs. You might want to enable the virt-ssbd and hv-tlbflush CPU flags for Windows. Also, replace emulated devices such as IDE, SATA and RTL8139 with virtio devices and install all Windows VirtIO drivers version 0.1.215 (0.1.217 is broken). Also, VirtIO SCSI disks supports more features then VirIO disks. As far as I know, there is no reason to use more than 1 socket for your VMs or does Threadripper expose NUMA (in which case you can to enable NUMA in the VM and use as many sockets as actual NUMA partitions on the host). (And I find the configuration files much easier to read than screenshots or photographs).
Anyway, what was your question in your last post?