Trouble with GPU Passthrough on 6.1

[lnumbp]

Hello,

I'm struggling to get my first ever Proxmox system setup. I can get to the point where a windows 10 guest sees the 1070 ti, but I cannot get rid of the code 43 error. I've tried all sorts of troubleshooting including args, passing the rom through, uninstalling and reinstalling drivers, etc. I have also tried swapping around hardware and pcie lanes. My system hardware is:

CPU: Threadripper 2950x
Mobo: Gigabyte Aorus X399 Pro
GPU (host): MSI 560 ti
GPU (guest/pthru): ASUS ROG 1070 ti
HDD: Tried NVME and SATA SSD
RAM: 32gb Kingston ECC

My setup process is:
1. Install proxmox
2. Attach CIFS (from another NAS for ISO files)
3. Setup VM as follows:

agent: 1
bios: ovmf
bootdisk: scsi0
cores: 4
cpu: host
efidisk0: local-lvm:vm-100-disk-0,size=128K
machine: q35
memory: 8192
name: WindowsVM
net0: virtio=_____,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
sata0: FREENAS:iso/virtio-win-0.1.171.iso,media=crom,size=363030K
scsi0; local-lvm:vm-100-disk-1,cache=writeback,size=40G
scsihw: virtio-scsi-single
smbios1: uuid=288dlda9-f2e9-4701-8e82-87cd4141f89e
sockets: 1
vga: default

4. Then I run through win10 home edition setup and install all missing virtio drivers.
5. Shutdown the VM,, attach

hostpci0: 41:00,pcie=1,rombar=0,x-vga=1

6. Follow the wiki here: https://pve.proxmox.com/wiki/Pci_passthrough#GPU_PASSTHROUGH
Notes: IOMMU groups look great and isolated.
Tried with and without setting unsafe interupts option in modprobe.d/.
Sure to add both the VGA device and audio vendor ID's in vfio.conf.
Tried with and without disable_vga=1 in vfio.conf.

7. Start the VM and install the latest NVIDIA driver using clean install option.

At this point the 1070 ti shows error 43 every time, without fail. I'm on the ropes about this whole project and would like a little bit of direction if at all possible.

Thank you!

 

[lnumbp]

I figured it out. I needed to turn ROMBAR on. The guide was otherwise followed to a T, no interupt required either. I'm very excited right now!

 

[lnumbp]

Okay now I'm having a very weird issue. Same hardware as above.

I cannot pass through a PCIe device if an M.2 drive is installed in a slot that shares the same cpu die. Refer to the image in the last post of this thread for more info:

https://forum.gigabyte.us/thread/2747/x399-aorus-gaming-chipset-lanes

Upon starting the VM I immediately get an error that reads:
nvme nvme0: failed to set APST feature (-19)
Buffer I/O error on device dm-1 logical block ________
Aborting journal on device dm-1-8
...
EXT4-fs (dm-1): I/O error while writing superblock
EXT4-fs error (device dm-1): ext4_journal_check_start:61: Detected aborted journal
EXT4-fs (dm-1): Remounting filesystem read-only

The system also completely locks and the webUI becomes inaccessible.

I tried messing around with 'nvme_core.default_ps_max_latency_us=__' and pcie_aspm = off in the grub, but I don't think the issue really is to do with nvme power states.

This is a pretty big bummer because it either prevents me from using m.2 drives altogether or cuts my board in half. Not ideal!

 

[lnumbp]

Holy craaaaap! Enabling "ACS Enable" in the bios fixed the issue. I took a look at my IOMMU groupings using

find /sys/kernel/iommu_groups/ -type l

before and it was a total mess. Enabling ACS broke everything out into its own group.

This project is such a roller coaster.