[SOLVED] GPU Passthrough problem with driver installation in Windows 10

[teclab]

Hi everyone,
I tried to solve this problem over weeks and weeks, performed so many Windows 10 installations, even a complete proxmox reinstall but nothin helped.
What I already did:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on" in
/etc/default/grub and update-grub
after reboot dmesg | grep -e DMAR -e IOMMU says:
[ 0.013181] ACPI: DMAR 0x000000003E2D4000 000070 (v01 INTEL EDK2 00000002 01000013) [ 0.013209] ACPI: Reserving DMAR table memory at [mem 0x3e2d4000-0x3e2d406f] [ 0.050362] DMAR: IOMMU enabled [ 0.118456] DMAR: Host address width 39 [ 0.118457] DMAR: DRHD base: 0x000000fed91000 flags: 0x1 [ 0.118460] DMAR: dmar0: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da [ 0.118462] DMAR: RMRR base: 0x0000003e8d9000 end: 0x0000003eb22fff [ 0.118464] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 0 [ 0.118465] DMAR-IR: HPET id 0 under DRHD base 0xfed91000 [ 0.118466] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 0.119836] DMAR-IR: Enabled IRQ remapping in x2apic mode [ 2.062440] DMAR: No ATSR found [ 2.062441] DMAR: dmar0: Using Queued invalidation [ 2.063211] DMAR: Intel(R) Virtualization Technology for Directed I/O
so I think thats good so far.
I added vfio vfio_iommu_type1 vfio_pci vfio_virqfd to /etc/modules

the IOMMU Isolation should work too:
find /sys/kernel/iommu_groups/ -type l /sys/kernel/iommu_groups/7/devices/0000:00:1b.4 /sys/kernel/iommu_groups/5/devices/0000:00:17.0 /sys/kernel/iommu_groups/3/devices/0000:00:14.2 /sys/kernel/iommu_groups/3/devices/0000:00:14.0 /sys/kernel/iommu_groups/11/devices/0000:03:00.0 /sys/kernel/iommu_groups/1/devices/0000:00:01.0 /sys/kernel/iommu_groups/1/devices/0000:01:00.0 /sys/kernel/iommu_groups/1/devices/0000:01:00.1 /sys/kernel/iommu_groups/8/devices/0000:00:1c.0 /sys/kernel/iommu_groups/6/devices/0000:00:1b.0 /sys/kernel/iommu_groups/4/devices/0000:00:16.0 /sys/kernel/iommu_groups/2/devices/0000:00:12.0 /sys/kernel/iommu_groups/10/devices/0000:00:1f.0 /sys/kernel/iommu_groups/10/devices/0000:00:1f.5 /sys/kernel/iommu_groups/10/devices/0000:00:1f.3 /sys/kernel/iommu_groups/10/devices/0000:00:1f.6 /sys/kernel/iommu_groups/10/devices/0000:00:1f.4 /sys/kernel/iommu_groups/0/devices/0000:00:00.0 /sys/kernel/iommu_groups/9/devices/0000:00:1d.4 /sys/kernel/iommu_groups/9/devices/0000:00:1d.0

I found this

and did that + backlisting: root@gamingrig:~# lspci -n -s 01:00 01:00.0 0300: 10de:1b06 (rev a1) 01:00.1 0403: 10de:10ef (rev a1) root@gamingrig:~# echo "options vfio-pci ids=10de:1b06,10de:10ef disable_vga=1" > /etc/modprobe.d/vfio.conf root@gamingrig:~# echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] [10de:1b06] (rev a1) Subsystem: Gigabyte Technology Co., Ltd GP102 [GeForce GTX 1080 Ti] [1458:376a] Kernel driver in use: vfio-pci Kernel modules: nvidiafb, nouveau


Installing Windows 10 an the virtio drivers etc.with this config works fine:
agent: 1 balloon: 4096 bios: ovmf boot: order=scsi0;net0;ide0;hostpci0 cores: 8 cpu: host,hidden=1 efidisk0: fast:vm-100-disk-1,size=4M hostpci0: 0000:01:00,pcie=1 machine: pc-q35-6.0 memory: 16384 name: Windows10 net0: virtio=8E:AC:5E:40:13:E7,bridge=vmbr0,firewall=1 numa: 0 ostype: win10 scsi0: fast:vm-100-disk-0,discard=on,iothread=1,size=250G,ssd=1 scsihw: virtio-scsi-single smbios1: uuid=38fc5f11-490f-446e-9d30-fcec465ea6c3 sockets: 1 vmgenid: 0bf0005e-3aa3-4988-8b72-d753c0da21bb
System says error 34, when I look for the card in Windows Device Manager, which is ok, because I didnt use the x-vga=1.
At this time I didnt install any other driver, just enabled the remote access, virtio drivers etc. (I also tried to install the nvidia driver at thtis point --> system freeze while installing with the nvidia driver and with driver installation out of geforce experience. also tried older driver, but same behavior system freeze and no boot after that freeze)
After this I edited the VM to start with GPU:
agent: 1 balloon: 4096 bios: ovmf boot: order=scsi0;net0;ide0;hostpci0 cores: 8 cpu: host,hidden=1 efidisk0: fast:vm-100-disk-1,size=4M hostpci0: 0000:01:00,pcie=1,x-vga=1, machine: pc-q35-6.0 memory: 16384 name: Windows10 net0: virtio=8E:AC:5E:40:13:E7,bridge=vmbr0,firewall=1 numa: 0 ostype: win10 scsi0: fast:vm-100-disk-0,discard=on,iothread=1,size=250G,ssd=1 scsihw: virtio-scsi-single smbios1: uuid=38fc5f11-490f-446e-9d30-fcec465ea6c3 sockets: 1 vmgenid: 0bf0005e-3aa3-4988-8b72-d753c0da21bb
System wont boot, dont know why, nothing happens, no error in Proxmox UI, just nothin happens. Plugged a Monitor to the card, I can see the some fragments of Gigabyte Boot Logo but nothin else.



I also tried with a rom for my card from TechPowerUp with "EFI", added romfile=bios.rom to the hostpci0 but nothin changes. The monitor keeps black.
I need help what else can I do to get this work?

 

[goldsteal]

For future reference: What is the Proxmox version?

 

[teclab]

Proxmox 7.0-13
Fixed by myself with adding video=efifb:off to the kernel.
Closed

 

[ieronymous]

Proxmox 7.0-13
Fixed by myself with adding video=efifb:off to the kernel.
Closed

.... to the kernel like which path and conf file you edited? Did that line had to go at the end, besides another one?

 

[teclab]

.... to the kernel like which path and conf file you edited? Did that line had to go at the end, besides another one?

nano /etc/default/grub
added it to the line
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on video=efifb:off"
then update-grub
& reboot and everything was fine

 

[ieronymous]

nano /etc/default/grub
added it to the line
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on video=efifb:off"
then update-grub
& reboot and everything was fine

So you are using bios mode instead of uefi I suppose by the line you had to edit,

 

[teclab]

So you are using bios mode instead of uefi I suppose by the line you had to edit,

I'm using bios: ovmf in my VM with efidesk etc. dont know what u mean with "bios mode".
My System/VM etc. is configured like in my first post ,just added the efifbff because a memory error in the logs.
Found it here: PCI Passtrough

Troubleshooting​

BAR 3: can't reserve [mem] error​

If you have this error when you try to use the card for a VM:

vfio-pci 0000:04:00.0: BAR 3: can't reserve [mem 0xca000000-0xcbffffff 64bit]


you can try to add the following kernel commandline option:

video=efifbff

 

[ieronymous]

dont know what u mean with "bios mode"

I meant during installation of proxmox if the drive that is going to be installed on has a gpt partition instead of mbr and in the physical machine itself you have setup uefi boot instead of legacy, then the changes you made had to be done in another path not nano /etc/default/grub but /etc/kernel/cmdline instead.
After that you have to run pve-efiboot-tool refresh even though running update-initramfs –u will update pve-efiboot-tool too.

 

[bvrulez]

I meant during installation of proxmox if the drive that is going to be installed on has a gpt partition instead of mbr and in the physical machine itself you have setup uefi boot instead of legacy, then the changes you made had to be done in another path not nano /etc/default/grub but /etc/kernel/cmdline instead.
After that you have to run pve-efiboot-tool refresh even though running update-initramfs –u will update pve-efiboot-tool too.

I am not sure if this is true. I am booting my proxmox in UEFI (the motherboard BIOS setup to exclusively boot in UEFI) and my /etc/kernel/cmdline is empty. Also, adding all the stuff to grub seems to do fine. But, I have to say, I am unable to passthrough my GTX 560Ti yet.