Trouble with AMD 6800XT and ubuntu 22.04.3

[Luthr0n]

I have attempted to look and try everything I could find to get this card to operate in linux but am just not finding a solution.

The issue I am having is when running clinfo the card is not displaying. When I run lspci on VM it is showing up and is using the AMDGPU kernel. I also install radeon-profile just to see if it would detect the card but it says there is no AMD graphics card.

I guess I am looking for some clarification if this would be a driver issue or a passthrough issue. Is there another test or way I can isolate this? In bios I have applied the settings to turn off rom and 4g decoding. From what I could tell if this was causing the issue the VM wouldn't have booted up.

 

[leesteken]

I would expect a 68000XT to reset and passthrough without any problems on the recent Proxmox kernels. Can you share the VM configuration file (qm config ) and the output of cat /proc/cmdline?
Did you disable Resizable BAR in the motherboard BIOS? Sometimes it's called SMA (Smart Access Memory) or CMA, but it is not supported (yet) with virtualization.

 

[Luthr0n]

qm config

qm config <vmid> [OPTIONS]
root@pve2:~# qm config 101
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 2
cpu: x86-64-v2-AES
efidisk0: local-lvm:vm-101-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:10:00,rombar=0
ide2: local:iso/ubuntu-22.04.3-desktop-amd64.iso,media=cdrom,size=4919592K
machine: q35
memory: 8000
meta: creation-qemu=8.0.2,ctime=1692305625
name: vm-101
net0: virtio=5A:75:71:E5:75:2D,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: local-lvm:vm-101-disk-1,discard=on,iothread=1,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=d5c9c3cf-6fda-41ca-bde8-e1c1ef62b6cf
sockets: 1
vmgenid: b29a5130-9b03-4115-892f-2d58f44b3580

cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-6.2.16-8-pve root=/dev/mapper/pve-root ro quiet initcall_blacklist=sysfb_init

dmesg | grep 'remapping'

[ 0.688942] x2apic: IRQ remapping doesn't support X2APIC mode
[ 0.976541] AMD-Vi: Interrupt remapping enabled

Resizable Bar is disabled as well as 4g decoding.

I can see the device in the VM listed under lspci

06:10.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] (rev c1) (prog-if 00 [VGA controller]
Subsystem: ASRock Incorporation Navi 21 [Radeon RX 6800/6800 XT / 6900 XT]
Physical Slot: 16-2
Flags: bus master, fast devsel, latency 0, IRQ 10
Memory at c0e0000000 (64-bit, prefetchable) [size=256M]
Memory at c0f0000000 (64-bit, prefetchable) [size=2M]
I/O ports at 9000
Memory at c1600000 (32-bit, non-prefetchable) [size=1M]
Capabilities: <access denied>
Kernel modules: amdgpu

But under clinfo I see this

NULL platform behavior
clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...) No platform
clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...) No platform
clCreateContext(NULL, ...) [default] No platform
clCreateContext(NULL, ...) [other] Success [POCL]
clCreateContextFromType(NULL, CL_DEVICE_TYPE_DEFAULT) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL) No devices found in platform

 

[leesteken]

Why the rombar=0? Why initcall_blacklist=sysfb_init? I have a 6950XT that works without those by just letting amdgpu load during Proxmox boot and then it reset properly, as a 6800XT should also. Did you early bind the device to vfio-pci? Did you blacklist amdgpu? With kernel 6.2, all those things are no necessary for AMD 6000 series, I think, when you just let amdgpu do its thing.
Also, set Display to None (vga=none) to make sure the VM uses the GPU. I strongly advise against automatically starting it at boot (onboot: 1) until you get passthrough fully working.
Without early binding, blacklisting and initcall_blacklist=sysfb_init[, you should see boot messages and have a host console until you start the VM. One thing I do before starting the VM is echo 0 | tee /sys/class/vtconsole/vtcon*/bind to close all host consoles before amdgpu passed the GPU to vfio-pci.

 

[Luthr0n]

leesteken I appreciate the help. With the new config and setting vga=none I can't run the vm. below is current config

bios: ovmf
boot: order=scsi0;ide2;net0
cores: 2
cpu: x86-64-v2-AES
efidisk0: local-lvm:vm-101-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
ide2: local:iso/ubuntu-22.04.3-desktop-amd64.iso,media=cdrom,size=4919592K
machine: q35
memory: 8000
meta: creation-qemu=8.0.2,ctime=1692305625
name: vm-101
net0: virtio=5A:75:71:E5:75:2D,bridge=vmbr0,firewall=1
numa: 0
onboot: 0
ostype: l26
scsi0: local-lvm:vm-101-disk-1,discard=on,iothread=1,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=d5c9c3cf-6fda-41ca-bde8-e1c1ef62b6cf
sockets: 1
vga: none
vmgenid: b29a5130-9b03-4115-892f-2d58f44b358

here is the error:

VM 101 qmp command 'set_password' failed - Could not set password
TASK ERROR: Failed to run vncproxy.

When I set display to default I am able to view from console

I have also removed the grub entries and update so this is current

BOOT_IMAGE=/boot/vmlinuz-6.2.16-8-pve root=/dev/mapper/pve-root ro quiet

I also took out the info in vfio and removed the black list entries for radeon which was the only one there that shouldn't have mattered.


Still clinfo is not showing a card but lspci still have the radeon listed.

Can you confirm that you get info back from clinfo?

 

[leesteken]

leesteken I appreciate the help. With the new config and setting vga=none I can't run the vm. below is current config


here is the error:

That's normal because there is no virtual display. Do you seen output on a physical display connected to the GPU?

When I set display to default I am able to view from console

But then Ubuntu might not use the GPU and only use the virtual display.

Can you confirm that you get info back from clinfo?

My GPU works fine and runs games very well but clinfo tells me Number of platforms 0. But I know nothing about clinfo and had to install it first.

 

[leesteken]

Can you confirm that you get info back from clinfo?

I needed to install mesa-opencl-icd (sudo apt install mesa-opencl-icd clinfo ) to get info from clinfo:

Number of platforms                               1
  Platform Name                                   Clover
  Platform Vendor                                 Mesa
  Platform Version                                OpenCL 1.1 Mesa 23.0.4-0ubuntu1~22.04.1
  Platform Profile                                FULL_PROFILE
  Platform Extensions                             cl_khr_icd
  Platform Extensions function suffix             MESA