Poor Transcode Performance with Quadro P400 Passthrough

J

jppars

New Member
Jul 6, 2025
5
0
1
Hi! I've been struggling with GPU passthrough of my Quadro P400 into a Debian 12 VM.

I want to use this card for video transcoding, and I'd seen online that it should be good enough to handle multiple 1080p streams. However, when it tries to transcode just one stream, it maxes out the GPU usage, and eventually crashes. Transcoding is done by Jellyfin running inside a Docker container.

I believe it's been passed through to the VM successfully (the correct device appears in an "lspci" on the guest), and I downloaded the proprietary drivers via apt. I feel like there must be an issue in my setup, but I can't seem to find it. Any suggestions or insight would be appreciated!


Proxmox Host PC
CPU: Ryzen 5 4650G
Mobo: ASUS B550M-C
GPU: Quadro P400


GRUB config
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt video=vesafb:off,efifb:off"


/etc/vfio.conf
options vfio-pci ids=10de:1cb3,10de:0fb9 disable_vga=1


/etc/blacklist.conf
Code: 
blacklist nouveau
blacklist nvidia
blacklist nvidiafb


/etc/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1


Guest VM Config
Code: 
agent: enabled=1
bios: ovmf
boot: order=scsi0;ide2
cores: 4
efidisk0: cache:201/vm-201-disk-0.qcow2,size=4M
hostpci0: mapping=GPU,pcie=1
# ide2: local:iso/debian-12.10.0-amd64-netinst.iso,media=cdrom,size=633M
localtime: 1
machine: q35
memory: 8192
meta: creation-qemu=9.2.0,ctime=1751445104
name: docker
net0: virtio=02:BA:A1:0F:33:49,bridge=vmbr0
onboot: 1
ostype: l26
scsi0: cache:201/vm-201-disk-1.qcow2,size=20G
scsi1: cache:201/vm-201-disk-2.qcow2,size=200G
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=ce5e42fc-5116-4246-890c-3d039af74aba
tablet: 0
tags: community-script
vmgenid: 7a357433-7066-4ef9-bf66-4ff66b19746f


lspci on guest VM
Code: 
01:00.0 VGA compatible controller: NVIDIA Corporation GP107GL [Quadro P400] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Dell GP107GL [Quadro P400]
        Physical Slot: 0
        Flags: bus master, fast devsel, latency 0, IRQ 42
        Memory at c1000000 (32-bit, non-prefetchable) [size=16M]
        Memory at c000000000 (64-bit, prefetchable) [size=256M]
        Memory at c010000000 (64-bit, prefetchable) [size=32M]
        I/O ports at d000 [size=128]
        Expansion ROM at c2080000 [virtual] [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [250] Latency Tolerance Reporting
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [420] Advanced Error Reporting
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Kernel driver in use: nvidia
        Kernel modules: nvidia

01:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1)
        Subsystem: Dell GP107GL High Definition Audio Controller
        Physical Slot: 0
        Flags: bus master, fast devsel, latency 0, IRQ 17
        Memory at c2000000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel


(Guest VM) dmesg | grep "01:00\.0"
Code: 
[    0.635226] pci 0000:01:00.0: [10de:1cb3] type 00 class 0x030000
[    0.668329] pci 0000:01:00.0: reg 0x10: [mem 0xc1000000-0xc1ffffff]
[    0.700330] pci 0000:01:00.0: reg 0x14: [mem 0xc000000000-0xc00fffffff 64bit pref]
[    0.732332] pci 0000:01:00.0: reg 0x1c: [mem 0xc010000000-0xc011ffffff 64bit pref]
[    0.764332] pci 0000:01:00.0: reg 0x24: [io  0xd000-0xd07f]
[    0.800331] pci 0000:01:00.0: reg 0x30: [mem 0xfff80000-0xffffffff pref]
[    0.801083] pci 0000:01:00.0: Max Payload Size set to 128 (was 256, max 256)
[    0.802471] pci 0000:01:00.0: 32.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x16 link at 0000:00:1c.0 (capable of 126.016 Gb/s with 8.0 GT/s PCIe x16 link)
[    1.084334] pci 0000:01:00.0: vgaarb: bridge control possible
[    1.084913] pci 0000:01:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[    1.114737] pci 0000:01:00.0: can't claim BAR 6 [mem 0xfff80000-0xffffffff pref]: no compatible bridge window
[    1.116745] pci 0000:01:00.0: BAR 6: assigned [mem 0xc2080000-0xc20fffff pref]
[    1.236866] pci 0000:01:00.1: D0 power state depends on 0000:01:00.0
[    3.642627] nvidia 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[    5.225281] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:01:00.0 on minor 1


Output from nvidia-smi when transcoding
https://pastebin.com/z3m9VNsW

I installed the NVIDIA and CUDA drivers through apt following this guide:
https://wiki.debian.org/NvidiaGraphicsDrivers
I also installed and configured the nvidia-container-toolkit for docker following this guide:
https://docs.nvidia.com/datacenter/cloud...guide.html

Thanks!
 
Last edited:
Have you tried cpu: host for that VM?

P.S. I don't do any passthrough myself.
 
gfngfn256 said:
Have you tried cpu: host for that VM?

P.S. I don't do any passthrough myself.
Click to expand...

I just added that line to the guest config and tried again with the same result. I can see it's updated in the web UI, but how would I expect this to affect the VM?
 
Dubyah said:
You can use jellyfin inside a container with HW encoding instead of a VM
Click to expand...

Hmm, I think I'd ultimately like it to be in the VM ( I have an immich container, and it would be nice if both could share the GPU resources. I also have my media files on an NFS share that I found easier to mount in a VM )

Is there a performance difference between LXC and VM significant enough that I should reconsider?
 
At the least you will need to stop & start that VM. Possibly a complete node shutdown & reboot would be appropriate. You may also want to increase the RAM of that guest from 8GB to 16GB to see what effect this has, as transcoding can sometimes be heavy depending on the stream.

I note from the general VM config that you setup the VM using debian-12.11.0-amd64-netinst.iso, but then I see tags: community-script, which leads me to thinking you used this, but that would be odd with that attached ISO. Maybe you are referring to the Jellyfin install in docker? IDK.
 
Last edited:
gfngfn256 said:
At the least you will need to stop & start that VM. Possibly a complete node shutdown & reboot would be appropriate. You may also want to increase the RAM of that guest from 8GB to 16GB to see what effect this has, as transcoding can sometimes be heavy depending on the stream.

I note from the general VM config that you setup the VM using debian-12.11.0-amd64-netinst.iso, but then I see tags: community-script, which leads me to thinking you used this, but that would be odd with that attached ISO. Maybe you are referring to the Jellyfin install in docker? IDK.
Click to expand...

Sorry, after I made that change, I did a full host reboot without any luck. I just increased the RAM to 12GB, rebooted the VM, and tried again with no luck. The Proxmox web UI is showing ~7GB of free RAM when performing transcoding, so it seems like RAM isn't really an issue.

As for the debian ISO. You're correct that I used the linked script for the initial creation, but I broke something while trying to alter my root partition. I added that ISO to recover my files. Sorry for the confusion!
 
Running any internet scripts (blindly or not) can unknowingly have adverse effects both on the host node & the emerging VM/LXC.

As I've said already above, I don't do any GPU passthrough - so take my advice with this limitation in mind.

If I were you - I'd probably try starting fresh (at least to test the GPU) by doing the following:

1. Create my own fresh Debian VM (no scripts - just use an original Debian ISO). Update & Upgrade.
2. Enable the GPU passthrough to the VM & test.
3. Install Jellyfin (maybe directly without docker in the VM - as here ?) & test.

Good luck.
 
gfngfn256 said:
Running any internet scripts (blindly or not) can unknowingly have adverse effects both on the host node & the emerging VM/LXC.

As I've said already above, I don't do any GPU passthrough - so take my advice with this limitation in mind.

If I were you - I'd probably try starting fresh (at least to test the GPU) by doing the following:

1. Create my own fresh Debian VM (no scripts - just use an original Debian ISO). Update & Upgrade.
2. Enable the GPU passthrough to the VM & test.
3. Install Jellyfin (maybe directly without docker in the VM - as here ?) & test.

Good luck.
Click to expand...

That makes sense, thanks for your help!
 
You must log in or register to reply here.
Top Bottom