[SOLVED] AMD Vega passthrough on Threadripper 2950X

[Chais]

Hi everyone,

hope you had nice holidays. I'm back to fiddling around with my new box and the essential piece of the puzzle is getting passthrough to work.
I've gone through the setup in the wiki and the pinned post here, so here's the current situation.

# pveversion
pve-manager/5.3-6/37b3c8df (running kernel: 4.15.18-9-pve)

# qm config 101
balloon: 0
bios: ovmf
bootdisk: scsi0
cores: 8
cpu: EPYC
efidisk0: local-lvm:vm-101-disk-1,size=128K
hostpci0: 0b:00,x-vga=1,romfile=undefined,pcie=1,rombar=0
ide2: local:iso/Win10_1809Oct_English_x64.iso,media=cdrom
machine: q35
memory: 8192
name: test
net0: virtio=1E:6F:DD:31:84:3B,bridge=vmbr0,link_down=1
net1: virtio=9A:83:59:8A:F0:7B,bridge=vmbr1
numa: 0
ostype: win10
scsi0: local-lvm:vm-101-disk-0,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=092727a5-6292-44be-ac53-8848cc7121db
sockets: 1
vmgenid: d7e2080b-e984-4853-bb12-9bbdef5e99bd

# cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=1002:687f,1002:aaf8 disable_vga=1

# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.15.18-9-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on video=efifb:off

# cat /etc/modules
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/17/devices/0000:0b:00.1
/sys/kernel/iommu_groups/35/devices/0000:43:00.0
/sys/kernel/iommu_groups/7/devices/0000:00:07.1
/sys/kernel/iommu_groups/25/devices/0000:40:02.0
/sys/kernel/iommu_groups/15/devices/0000:0a:00.0
/sys/kernel/iommu_groups/33/devices/0000:41:00.0
/sys/kernel/iommu_groups/5/devices/0000:00:04.0
/sys/kernel/iommu_groups/23/devices/0000:0d:00.3
/sys/kernel/iommu_groups/13/devices/0000:02:00.0
/sys/kernel/iommu_groups/13/devices/0000:08:00.0
/sys/kernel/iommu_groups/13/devices/0000:01:00.2
/sys/kernel/iommu_groups/13/devices/0000:02:03.0
/sys/kernel/iommu_groups/13/devices/0000:01:00.0
/sys/kernel/iommu_groups/13/devices/0000:02:09.0
/sys/kernel/iommu_groups/13/devices/0000:02:02.0
/sys/kernel/iommu_groups/13/devices/0000:05:00.0
/sys/kernel/iommu_groups/13/devices/0000:01:00.1
/sys/kernel/iommu_groups/13/devices/0000:02:01.0
/sys/kernel/iommu_groups/13/devices/0000:02:04.0
/sys/kernel/iommu_groups/41/devices/0000:45:00.2
/sys/kernel/iommu_groups/31/devices/0000:40:08.0
/sys/kernel/iommu_groups/3/devices/0000:00:03.0
/sys/kernel/iommu_groups/21/devices/0000:0d:00.0
/sys/kernel/iommu_groups/11/devices/0000:00:18.3
/sys/kernel/iommu_groups/11/devices/0000:00:18.1
/sys/kernel/iommu_groups/11/devices/0000:00:18.6
/sys/kernel/iommu_groups/11/devices/0000:00:18.4
/sys/kernel/iommu_groups/11/devices/0000:00:18.2
/sys/kernel/iommu_groups/11/devices/0000:00:18.0
/sys/kernel/iommu_groups/11/devices/0000:00:18.7
/sys/kernel/iommu_groups/11/devices/0000:00:18.5
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/38/devices/0000:44:00.2
/sys/kernel/iommu_groups/28/devices/0000:40:04.0
/sys/kernel/iommu_groups/18/devices/0000:0c:00.0
/sys/kernel/iommu_groups/36/devices/0000:43:00.1
/sys/kernel/iommu_groups/8/devices/0000:00:08.0
/sys/kernel/iommu_groups/26/devices/0000:40:03.0
/sys/kernel/iommu_groups/16/devices/0000:0b:00.0
/sys/kernel/iommu_groups/34/devices/0000:42:00.0
/sys/kernel/iommu_groups/6/devices/0000:00:07.0
/sys/kernel/iommu_groups/24/devices/0000:40:01.0
/sys/kernel/iommu_groups/14/devices/0000:09:00.0
/sys/kernel/iommu_groups/32/devices/0000:40:08.1
/sys/kernel/iommu_groups/4/devices/0000:00:03.1
/sys/kernel/iommu_groups/22/devices/0000:0d:00.2
/sys/kernel/iommu_groups/12/devices/0000:00:19.6
/sys/kernel/iommu_groups/12/devices/0000:00:19.4
/sys/kernel/iommu_groups/12/devices/0000:00:19.2
/sys/kernel/iommu_groups/12/devices/0000:00:19.0
/sys/kernel/iommu_groups/12/devices/0000:00:19.7
/sys/kernel/iommu_groups/12/devices/0000:00:19.5
/sys/kernel/iommu_groups/12/devices/0000:00:19.3
/sys/kernel/iommu_groups/12/devices/0000:00:19.1
/sys/kernel/iommu_groups/40/devices/0000:45:00.0
/sys/kernel/iommu_groups/30/devices/0000:40:07.1
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/20/devices/0000:0c:00.3
/sys/kernel/iommu_groups/10/devices/0000:00:14.3
/sys/kernel/iommu_groups/10/devices/0000:00:14.0
/sys/kernel/iommu_groups/39/devices/0000:44:00.3
/sys/kernel/iommu_groups/29/devices/0000:40:07.0
/sys/kernel/iommu_groups/0/devices/0000:00:01.0
/sys/kernel/iommu_groups/19/devices/0000:0c:00.2
/sys/kernel/iommu_groups/37/devices/0000:44:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:08.1
/sys/kernel/iommu_groups/27/devices/0000:40:03.1

The VM will start with ROMBAR disabled using either the whole device or passing both sub-devices. It will not, however, display anything on the attached monitor. The VM starts and works fine if I don't pass the card through.
Is there a way for me to get more information as to why things fail?
Am I missing something?
As you can see I'm using the latest Win10 install medium from microsoft.com. Should I try a different version?
A little off-topic: is there a way to get hyper-threading support for the Threadripper?
Thanks in advance for any suggestions.

 

[AlexLup]

Use the Proxmox PCI Passthrough guide to dump the ROM of the card, that's what I did to get my Nvidias working

 

[Chais]

I've done what you suggested with minuscule success. With the romfile supplied the output of the card is at least blanked when the VM is started. But that's it.
I've tried PCIE on and off (along with q35, obviously) and using the complete device or passing subdevices separately. It's always the same outcome. The VM starts but doesn't display anything.
I tried running it with SeaBIOS with and without the romfile supplied, which resulted in the following error:

failed getting region info for VGA region index 8: Invalid argument
device does not support requested feature x-vga

If I remove the x-vga flag the VM starts but again doesn't display anything. Neither on the attached display nor through PVE.
Any other ideas? Still trying to get debug output.

 

[AlexLup]

Try adding
iommu=pt pcie_acs_override=downstream

to
/etc/default/grub ->
GRUB_CMDLINE_LINUX_DEFAULT

 

[Chais]

Nope, that didn't do anything. I think I've tried all combinations. But honestly, I'd have been surprised if this had worked. I already have IOMMU groups than I care to count. The card has separate groups from the sub-devices. Updated original post with groups.

 

[elurex]

Nope, that didn't do anything. I think I've tried all combinations. But honestly, I'd have been surprised if this had worked. I already have IOMMU groups than I care to count. The card has separate groups from the sub-devices. Updated original post with groups.

I am able to pci-passthrough amd vega 64 with following config

agent: 1
bios: ovmf
boot: c
bootdisk: scsi0
cores: 8
cpu: host
efidisk0: local-zfs:vm-905-disk-0,size=1M
hostpci0: 06:00
machine: q35
memory: 16384
name: amdgpu
net0: virtio=EA:8F:88:AB:6D:50,bridge=vmbr0
numa: 1
ostype: l26
scsi0: local-zfs:vm-905-disk-1,cache=writeback
scsihw: virtio-scsi-pci
smbios1: uuid=363350cd-6bca-4738-bc43-9c29934b7730
sockets: 1
vmgenid: a9ad60a1-8f4f-47b6-8cb8-04dd8487a2d5

 

[Chais]

Nope. But it's good to know that cpu: host now works.
Any idea what I should feed qemu's -d at to get more information?

 

[Chais]

In case someone stumble across this:
The solution was using the latest firmware for my Gigabyte AMD RX Vega 64. Version F2 as of writing this.