Hey all, would appreciate any insight as I've spent a great deal of time on this and am no longer making progress. I have a single GTX 1650 GPU and a PCIE Coral TPU passed through to a VM (not a container) that I would like to use as a media library using Jellyfin and continuous recorder for security cameras via Shinobi. I'm nearly certain I have the GPU passed through correctly (and the TPU, although I'm not concerned about that in this post right now), but for some reason I cannot get hardware acceleration to work with anything - I can't get the GPU to "work" (i.e. processor % and power in `nvtop`) for *anything* with the exception of GravityMark running in Vulkan mode. Running GravityMark with OpenGL is an immediate seg fault. Doesn't this mean that the GPU is passed through successfully if I can see it getting used for the GravityMark Vulkan test? `vainfo` shows a driver error also:
Clearly there is a driver issue, but I have tried at least 4 versions of the Nvidia driver, including the one that installs with this version of Ubuntu 22.04 with no luck. There's no video hardware acceleration happening anywhere including with VLC, Chrome, Thorium browser, jellyfin, Kodi, Shinobi, etc. I've put the info below that is usually posted in the GPU passthrough posts, but honestly I think I've got it passed through correctly. I haven't seen anyone talk about this specific setup, but if I can get it to use the integrated GPU as the primary and just offload the the Nvidia for video decoding that would be ideal, although I have tried disabling the display, hooking up a monitor and RDP'ing into it and hardware acceleration didn't work there either.
If this post is more of a Ubuntu driver issue and doesn't belong here I understand, but I'm pretty certain if that's all it was I'd have figured it out by now. Thanks for your help.
$vainfo (on guest)libva info: VA-API version 1.14.0libva error: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null)vaInitialize failed with error code -1 (unknown libva error),exitClearly there is a driver issue, but I have tried at least 4 versions of the Nvidia driver, including the one that installs with this version of Ubuntu 22.04 with no luck. There's no video hardware acceleration happening anywhere including with VLC, Chrome, Thorium browser, jellyfin, Kodi, Shinobi, etc. I've put the info below that is usually posted in the GPU passthrough posts, but honestly I think I've got it passed through correctly. I haven't seen anyone talk about this specific setup, but if I can get it to use the integrated GPU as the primary and just offload the the Nvidia for video decoding that would be ideal, although I have tried disabling the display, hooking up a monitor and RDP'ing into it and hardware acceleration didn't work there either.
If this post is more of a Ubuntu driver issue and doesn't belong here I understand, but I'm pretty certain if that's all it was I'd have figured it out by now. Thanks for your help.
Code:
**uname -a and cat /etc/os-release on guest VM**
```
Linux shinobi.milloy.arpa 6.5.0-41-generic #41~22.04.2-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun 3 11:32:55 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
```
================================
pveversion -v
================================
```
proxmox-ve: 8.2.0 (running kernel: 6.8.8-2-pve)
pve-manager: 8.2.4 (running version: 8.2.4/faa83925c9641325)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.8-2
proxmox-kernel-6.8.8-2-pve-signed: 6.8.8-2
proxmox-kernel-6.8.4-3-pve-signed: 6.8.4-3
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
proxmox-kernel-6.5.13-5-pve-signed: 6.5.13-5
proxmox-kernel-6.5: 6.5.13-5
proxmox-kernel-6.5.13-1-pve-signed: 6.5.13-1
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
proxmox-kernel-6.5.11-7-pve-signed: 6.5.11-7
proxmox-kernel-6.5.11-4-pve-signed: 6.5.11-4
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.3
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.9
libpve-storage-perl: 8.2.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.4-1
proxmox-backup-file-restore: 3.2.4-1
proxmox-firewall: 0.4.2
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.1.12
pve-docs: 8.2.2
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.1
pve-firewall: 5.0.7
pve-firmware: 3.12-1
pve-ha-manager: 4.0.5
pve-i18n: 3.2.2
pve-qemu-kvm: 9.0.0-3
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.4-pve1
```
================================
cat /etc/default/grub
================================
```
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction"
GRUB_CMDLINE_LINUX=""
# https://www.reddit.com/r/homeassistant/comments/xzajo0/coral_tpu_proxmox_frigate/
pcie_aspm=off
```
================================
dmesg | grep -e DMAR -e IOMMU
================================
```
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[ 0.010480] ACPI: DMAR 0x000000006E0C6000 000088 (v01 INTEL EDK2 00000002 01000013)
[ 0.010505] ACPI: Reserving DMAR table memory at [mem 0x6e0c6000-0x6e0c6087]
[ 0.153718] DMAR: IOMMU enabled
[ 0.338991] DMAR: Host address width 39
[ 0.338992] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.338996] DMAR: dmar0: reg_base_addr fed90000 ver 4:0 cap 1c0000c40660462 ecap 29a00f0505e
[ 0.338997] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.339000] DMAR: dmar1: reg_base_addr fed91000 ver 5:0 cap d2008c40660462 ecap f050da
[ 0.339001] DMAR: RMRR base: 0x00000076000000 end: 0x000000807fffff
[ 0.339004] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.339005] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ 0.339005] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.340632] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.544523] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
[ 0.628552] DMAR: No ATSR found
[ 0.628553] DMAR: No SATC found
[ 0.628554] DMAR: IOMMU feature fl1gp_support inconsistent
[ 0.628555] DMAR: IOMMU feature pgsel_inv inconsistent
[ 0.628555] DMAR: IOMMU feature nwfs inconsistent
[ 0.628556] DMAR: IOMMU feature dit inconsistent
[ 0.628557] DMAR: IOMMU feature sc_support inconsistent
[ 0.628557] DMAR: IOMMU feature dev_iotlb_support inconsistent
[ 0.628558] DMAR: dmar0: Using Queued invalidation
[ 0.628560] DMAR: dmar1: Using Queued invalidation
[ 0.629651] DMAR: Intel(R) Virtualization Technology for Directed I/O
```
================================
/etc/modprobe.d/*
================================
--------------------------------
### /etc/modprobe.d/coral-tpu.conf contents:
```
# https://www.reddit.com/r/homeassistant/comments/xzajo0/coral_tpu_proxmox_frigate/
blacklist gasket
blacklist apex
options vfio-pci ids=1ac1:089a
```
--------------------------------
### /etc/modprobe.d/dkms.conf contents:
empty
--------------------------------
### /etc/modprobe.d/nvidia_gtx_1650_passthrough.conf contents:
```
# passthrough nvidia gpu gtx 1650 to vm's
# https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_host_device_passthrough
options vfio-pci ids=10de:1f0a,10de:10f9 disable_vga=1
blacklist nvidiafb nouveau snd_hda_intel
#softdep nvidiafb pre: vfio-pci
#softdep snd_hda_intel pre: vfio-pci
```
--------------------------------
### /etc/modprobe.d/nvidia-installer-disable-nouveau.conf.disabled contents:
```
# generated by nvidia-installer - disabled by rob
# blacklist nouveau
# options nouveau modeset=0
```
--------------------------------
### /etc/modprobe.d/pve-blacklist.conf contents:
```
# This file contains a list of modules which are not supported by Proxmox VE
# nvidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
```
--------------------------------
### /etc/modprobe.d/zfs.conf contents:
```
options zfs zfs_arc_max=6717177856
```
--------------------------------
================================
/etc/pve/qemu-server/*
================================
--------------------------------
### /etc/pve/qemu-server/107.conf contents:
```
#virtio1%3A spare%3Avm-107-disk-0,backup=0,cache=directsync,iothread=1,size=3500G
#virtio1%3A spare%3Avm-107-disk-0,backup=0,iothread=1,size=3500G
agent: 1
audio0: device=ich9-intel-hda,driver=spice
balloon: 16000
bios: ovmf
boot: order=virtio0;ide2;net0
cores: 4
cpu: host
efidisk0: local-zfs:vm-107-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:01:00,pcie=1,x-vga=1
hostpci1: 0000:07:00,pcie=1
ide2: none,media=cdrom
machine: q35
memory: 32768
meta: creation-qemu=8.1.5,ctime=1708573643
name: shinobi
net0: virtio=BC:24:11:B8:28:9E,bridge=vmbr0,firewall=1
numa: 1
onboot: 1
ostype: l26
protection: 1
scsihw: virtio-scsi-single
smbios1: uuid=04405862-94be-4d67-a09c-ee4648d2e8cd
sockets: 2
spice_enhancements: foldersharing=1,videostreaming=all
tablet: 1
vga: qxl2
virtio0: local-zfs:vm-107-disk-1,iothread=1,size=100G
virtio1: spare:vm-107-disk-0,iothread=1,size=3500G
vmgenid: f76c68fc-17e1-4531-9d5d-cb8d43582bf9
vmstatestorage: local-zfs
```
======================================
#lshw -C display (on proxmox host)
=====================================
```
*-display
description: VGA compatible controller
product: TU106 [GeForce GTX 1650]
vendor: NVIDIA Corporation
physical id: 0
bus info: pci@0000:01:00.0
version: a1
width: 64 bits
clock: 33MHz
capabilities: pm msi pciexpress vga_controller bus_master cap_list rom
configuration: driver=vfio-pci latency=0
resources: iomemory:600-5ff iomemory:600-5ff irq:282 memory:81000000-81ffffff memory:60e0000000-60efffffff memory:60f0000000-60f1ffffff ioport:8000(size=128) memory:82000000-8207ffff
*-display
description: Display controller
product: AlderLake-S GT1
vendor: Intel Corporation
physical id: 2
bus info: pci@0000:00:02.0
version: 0c
width: 64 bits
clock: 33MHz
capabilities: pciexpress msi pm bus_master cap_list
configuration: driver=i915 latency=0
resources: iomemory:600-5ff iomemory:400-3ff irq:249 memory:60fa000000-60faffffff memory:4000000000-400fffffff ioport:9000(size=64) memory:60f3000000-60f9ffffff memory:6000000000-60dfffffff
```
===============================
Output from Ubuntu 22.04 VM
===============================
$glxinfo -B
```
name of display: :1
display: :1 screen: 0
direct rendering: Yes
Extended renderer info (GLX_MESA_query_renderer):
Vendor: Mesa (0xffffffff)
Device: llvmpipe (LLVM 15.0.7, 256 bits) (0xffffffff)
Version: 23.2.1
Accelerated: no
Video memory: 32097MB
Unified memory: yes
Preferred profile: core (0x1)
Max core profile version: 4.5
Max compat profile version: 4.5
Max GLES1 profile version: 1.1
Max GLES[23] profile version: 3.2
Memory info (GL_ATI_meminfo):
VBO free memory - total: 31 MB, largest block: 31 MB
VBO free aux. memory - total: 15408 MB, largest block: 15408 MB
Texture free memory - total: 31 MB, largest block: 31 MB
Texture free aux. memory - total: 15408 MB, largest block: 15408 MB
Renderbuffer free memory - total: 31 MB, largest block: 31 MB
Renderbuffer free aux. memory - total: 15408 MB, largest block: 15408 MB
Memory info (GL_NVX_gpu_memory_info):
Dedicated video memory: 4293974383 MB
Total available memory: 4294006481 MB
Currently available dedicated video memory: 31 MB
OpenGL vendor string: Mesa
OpenGL renderer string: llvmpipe (LLVM 15.0.7, 256 bits)
OpenGL core profile version string: 4.5 (Core Profile) Mesa 23.2.1-1ubuntu3.1~22.04.2
OpenGL core profile shading language version string: 4.50
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile
OpenGL version string: 4.5 (Compatibility Profile) Mesa 23.2.1-1ubuntu3.1~22.04.2
OpenGL shading language version string: 4.50
OpenGL context flags: (none)
OpenGL profile mask: compatibility profile
OpenGL ES profile version string: OpenGL ES 3.2 Mesa 23.2.1-1ubuntu3.1~22.04.2
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20
```
$nvidia-smi
```
Thu Jul 4 12:58:11 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07 Driver Version: 550.90.07 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce GTX 1650 Off | 00000000:01:00.0 Off | N/A |
| 40% 27C P8 5W / 90W | 8MiB / 4096MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 12353 G /usr/lib/xorg/Xorg 4MiB |
+-----------------------------------------------------------------------------------------+
```