GPU Nvidia GeForce GTX 1660 Ti passthrough AMD Ryzen 9 5900X VM Ubuntu Server 20.04

stepanov1975

New Member
Oct 22, 2021
7
2
3
46
Hi,

I needed to passthrough GPU Nvidia GeForce GTX 1660 Ti to VM to use it for Plex server hardware acceleration.

Took me a lot of time to find a working configuration so I want to share. May be will help someone
I used the following sources:
Plex on ProxMox Tutorial WITH nVidia Hardware Encoding
The Ultimate Beginner's Guide to GPU Passthrough
GPU Nvidia 1660 passthrough

103.conf
Code:
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 4
cpu: host,hidden=1,flags=+pcid
efidisk0: proxthin:vm-103-disk-1,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:08:00,pcie=1,x-vga=1
ide2: local:iso/ubuntu-20.04.3-live-server-amd64.iso,media=cdrom
machine: q35
memory: 5024
name: plex
net0: virtio=96:4C:80:5C:68:D0,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
parent: drivers_downloaded
scsi0: proxthin:vm-103-disk-0,size=96G
scsihw: virtio-scsi-pci
smbios1: uuid=3f65ef9d-1434-4938-ba99-3e66103674f2
snaptime: 1634907300
sockets: 1
vga: virtio
vmgenid: 43467307-a9d1-4a06-9bcb-795afc011219

/etc/default/grub
Code:
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt video=vesafb:off video=efifb:off"
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

Code:
update-grub

/etc/modprobe.d/blacklist.conf
Code:
blacklist nvidia
blacklist nouveau
blacklist radeon
blacklist i2c_nvidia_gpu
blacklist nvidiafb

lspci -k
Code:
08:00.0 VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1660 Ti] (rev a1)
        Subsystem: PNY TU116 [GeForce GTX 1660 Ti]
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau
08:00.1 Audio device: NVIDIA Corporation TU116 High Definition Audio Controller (rev a1)
        Subsystem: PNY TU116 High Definition Audio Controller
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
08:00.2 USB controller: NVIDIA Corporation TU116 USB 3.1 Host Controller (rev a1)
        Subsystem: PNY TU116 USB 3.1 Host Controller
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci
08:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 USB Type-C UCSI Controller (rev a1)
        Subsystem: PNY TU116 USB Type-C UCSI Controller
        Kernel driver in use: vfio-pci
        Kernel modules: i2c_nvidia_gpu

lspci -nns 08:00
Code:
08:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU116 [GeForce GTX 1660 Ti] [10de:2182] (rev a1)
08:00.1 Audio device [0403]: NVIDIA Corporation TU116 High Definition Audio Controller [10de:1aeb] (rev a1)
08:00.2 USB controller [0c03]: NVIDIA Corporation TU116 USB 3.1 Host Controller [10de:1aec] (rev a1)
08:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 USB Type-C UCSI Controller [10de:1aed] (rev a1)

/etc/modprobe.d/vfio.conf
Code:
options vfio-pci ids=10de:2182,10de:1aeb,10de:1aec,10de:1aed disable_vga=1


On the VM (Ubuntu server 20.04)
/etc/modprobe.d/blacklist-nvidia-nouveau.conf
Code:
blacklist nouveau
options nouveau modeset=0
sudo update-initramfs -u

lspci -k
Code:
01:00.0 VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1660 Ti] (rev a1)
        Subsystem: PNY TU116 [GeForce GTX 1660 Ti]
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
01:00.1 Audio device: NVIDIA Corporation TU116 High Definition Audio Controller (rev a1)
        Subsystem: PNY TU116 High Definition Audio Controller
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
01:00.2 USB controller: NVIDIA Corporation TU116 USB 3.1 Host Controller (rev a1)
        Subsystem: PNY Device 1313
        Kernel driver in use: xhci_hcd
01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] (rev a1)
        Subsystem: PNY TU116 [GeForce GTX 1650 SUPER]
        Kernel driver in use: nvidia-gpu
        Kernel modules: i2c_nvidia_gpu

lspci -nns 01:00
Code:
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU116 [GeForce GTX 1660 Ti] [10de:2182] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation TU116 High Definition Audio Controller [10de:1aeb] (rev a1)
01:00.2 USB controller [0c03]: NVIDIA Corporation TU116 USB 3.1 Host Controller [10de:1aec] (rev a1)
01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] [10de:1aed] (rev a1)
 
Last edited:
  • Like
Reactions: Dunuin

leesteken

Famous Member
May 31, 2020
1,943
409
88
Thanks for posting a working configuration. Just a few minor things:
amd_iommu=on is unnecessary as it is always on unless you turn it off; actually on is not even a valid option.
Did you mean to write modeset=0 or is options nouveau modset=0 correct? If you blacklist the module, the options should have no effect, so you might as well leave them out.
Why do you blacklist radeon when you are passing through an NVidia GPU and not an AMD one?
The vga: virtio is ignored when using x-vga=1, so it might as well be set to vga: none or removed. Or add a serial console?
Does your GPU consist of four devices, as implied by the four sets of vfio-pci.ids, instead of the usual two? Maybe you can add the output of lspci -nns 08:00 to explain those vfio-pci.ids?

EDIT: Please don't be offended by my remars. I'm really happy that you got it working and especially that you took the trouble to report it for others to learn from. I'm just dotting the i's and crossing the t's, so to speak.
 
Last edited:
  • Like
Reactions: Dunuin

stepanov1975

New Member
Oct 22, 2021
7
2
3
46
Thanks for posting a working configuration. Just a few minor things:
amd_iommu=on is unnecessary as it is always on unless you turn it off; actually on is not even a valid option.
Did you mean to write modeset=0 or is options nouveau modset=0 correct? If you blacklist the module, the options should have no effect, so you might as well leave them out.
Why do you blacklist radeon when you are passing through an NVidia GPU and not an AMD one?
The vga: virtio is ignored when using x-vga=1, so it might as well be set to vga: none or removed. Or add a serial console?
Does your GPU consist of three devices, as implied by the three sets of vfio-pci.ids, instead of the usual two? Maybe you can add the output of lspci -nns 08:00 to explain those vfio-pci.ids?
@avw
Thank you for your comments. I don't understand a thing from all configurations above. I only looked for instructions and tried what worked

amd_iommu=on is unnecessary as it is always on unless you turn it off; actually on is not even a valid option.
Option amd_iommu=on recommended here: https://pve.proxmox.com/wiki/Pci_passthrough

Did you mean to write modeset=0 or is options nouveau modset=0 correct?

options nouveau modset=0 taken from tutorial. I checked and it seems there is a typo and it should be modeset. I fixed it in the original post and in my configuration. Seems to work fine.
Why do you blacklist radeon when you are passing through an NVidia GPU and not an AMD one?
Just in case. Anyway, that is what was in the tutorial :)

The vga: virtio is ignored when using x-vga=1, so it might as well be set to vga: none or removed. Or add a serial console?
With vga: none noVNC not working

Does your GPU consist of three devices, as implied by the three sets of vfio-pci.ids, instead of the usual two? Maybe you can add the output of lspci -nns 08:00 to explain those vfio-pci.ids?

As far as I understand all devices are part of GPU so I am passing all as I don't know what is important. I added output of lspci -nns 08:00 to the original post
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!