GPU Nvidia GeForce GTX 1660 Ti passthrough AMD Ryzen 9 5900X VM Ubuntu Server 20.04

[stepanov1975]

Hi,

I needed to passthrough GPU Nvidia GeForce GTX 1660 Ti to VM to use it for Plex server hardware acceleration.

Took me a lot of time to find a working configuration so I want to share. May be will help someone
I used the following sources:
Plex on ProxMox Tutorial WITH nVidia Hardware Encoding
The Ultimate Beginner's Guide to GPU Passthrough
GPU Nvidia 1660 passthrough

103.conf

bios: ovmf
boot: order=scsi0;ide2;net0
cores: 4
cpu: host,hidden=1,flags=+pcid
efidisk0: proxthin:vm-103-disk-1,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:08:00,pcie=1,x-vga=1
ide2: local:iso/ubuntu-20.04.3-live-server-amd64.iso,media=cdrom
machine: q35
memory: 5024
name: plex
net0: virtio=96:4C:80:5C:68:D0,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
parent: drivers_downloaded
scsi0: proxthin:vm-103-disk-0,size=96G
scsihw: virtio-scsi-pci
smbios1: uuid=3f65ef9d-1434-4938-ba99-3e66103674f2
snaptime: 1634907300
sockets: 1
vga: virtio
vmgenid: 43467307-a9d1-4a06-9bcb-795afc011219

/etc/default/grub

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt video=vesafb:off video=efifb:off"
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

update-grub

/etc/modprobe.d/blacklist.conf

blacklist nvidia
blacklist nouveau
blacklist radeon
blacklist i2c_nvidia_gpu
blacklist nvidiafb

lspci -k

08:00.0 VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1660 Ti] (rev a1)
        Subsystem: PNY TU116 [GeForce GTX 1660 Ti]
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau
08:00.1 Audio device: NVIDIA Corporation TU116 High Definition Audio Controller (rev a1)
        Subsystem: PNY TU116 High Definition Audio Controller
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
08:00.2 USB controller: NVIDIA Corporation TU116 USB 3.1 Host Controller (rev a1)
        Subsystem: PNY TU116 USB 3.1 Host Controller
        Kernel driver in use: vfio-pci
        Kernel modules: xhci_pci
08:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 USB Type-C UCSI Controller (rev a1)
        Subsystem: PNY TU116 USB Type-C UCSI Controller
        Kernel driver in use: vfio-pci
        Kernel modules: i2c_nvidia_gpu

lspci -nns 08:00

08:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU116 [GeForce GTX 1660 Ti] [10de:2182] (rev a1)
08:00.1 Audio device [0403]: NVIDIA Corporation TU116 High Definition Audio Controller [10de:1aeb] (rev a1)
08:00.2 USB controller [0c03]: NVIDIA Corporation TU116 USB 3.1 Host Controller [10de:1aec] (rev a1)
08:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 USB Type-C UCSI Controller [10de:1aed] (rev a1)

/etc/modprobe.d/vfio.conf

options vfio-pci ids=10de:2182,10de:1aeb,10de:1aec,10de:1aed disable_vga=1

On the VM (Ubuntu server 20.04)
/etc/modprobe.d/blacklist-nvidia-nouveau.conf

blacklist nouveau
options nouveau modeset=0

sudo update-initramfs -u

lspci -k

01:00.0 VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1660 Ti] (rev a1)
        Subsystem: PNY TU116 [GeForce GTX 1660 Ti]
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
01:00.1 Audio device: NVIDIA Corporation TU116 High Definition Audio Controller (rev a1)
        Subsystem: PNY TU116 High Definition Audio Controller
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
01:00.2 USB controller: NVIDIA Corporation TU116 USB 3.1 Host Controller (rev a1)
        Subsystem: PNY Device 1313
        Kernel driver in use: xhci_hcd
01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] (rev a1)
        Subsystem: PNY TU116 [GeForce GTX 1650 SUPER]
        Kernel driver in use: nvidia-gpu
        Kernel modules: i2c_nvidia_gpu

lspci -nns 01:00

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU116 [GeForce GTX 1660 Ti] [10de:2182] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation TU116 High Definition Audio Controller [10de:1aeb] (rev a1)
01:00.2 USB controller [0c03]: NVIDIA Corporation TU116 USB 3.1 Host Controller [10de:1aec] (rev a1)
01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] [10de:1aed] (rev a1)

 

[leesteken]

Thanks for posting a working configuration. Just a few minor things:
amd_iommu=on is unnecessary as it is always on unless you turn it off; actually on is not even a valid option.
Did you mean to write modeset=0 or is options nouveau modset=0 correct? If you blacklist the module, the options should have no effect, so you might as well leave them out.
Why do you blacklist radeon when you are passing through an NVidia GPU and not an AMD one?
The vga: virtio is ignored when using x-vga=1, so it might as well be set to vga: none or removed. Or add a serial console?
Does your GPU consist of four devices, as implied by the four sets of vfio-pci.ids, instead of the usual two? Maybe you can add the output of lspci -nns 08:00 to explain those vfio-pci.ids?

EDIT: Please don't be offended by my remars. I'm really happy that you got it working and especially that you took the trouble to report it for others to learn from. I'm just dotting the i's and crossing the t's, so to speak.

 

[stepanov1975]

Thanks for posting a working configuration. Just a few minor things:
amd_iommu=on is unnecessary as it is always on unless you turn it off; actually on is not even a valid option.
Did you mean to write modeset=0 or is options nouveau modset=0 correct? If you blacklist the module, the options should have no effect, so you might as well leave them out.
Why do you blacklist radeon when you are passing through an NVidia GPU and not an AMD one?
The vga: virtio is ignored when using x-vga=1, so it might as well be set to vga: none or removed. Or add a serial console?
Does your GPU consist of three devices, as implied by the three sets of vfio-pci.ids, instead of the usual two? Maybe you can add the output of lspci -nns 08:00 to explain those vfio-pci.ids?

@avw
Thank you for your comments. I don't understand a thing from all configurations above. I only looked for instructions and tried what worked

amd_iommu=on is unnecessary as it is always on unless you turn it off; actually on is not even a valid option.

Option amd_iommu=on recommended here: https://pve.proxmox.com/wiki/Pci_passthrough

Did you mean to write modeset=0 or is options nouveau modset=0 correct?

options nouveau modset=0 taken from tutorial. I checked and it seems there is a typo and it should be modeset. I fixed it in the original post and in my configuration. Seems to work fine.

Why do you blacklist radeon when you are passing through an NVidia GPU and not an AMD one?

Just in case. Anyway, that is what was in the tutorial

The vga: virtio is ignored when using x-vga=1, so it might as well be set to vga: none or removed. Or add a serial console?

With vga: none noVNC not working

Does your GPU consist of three devices, as implied by the three sets of vfio-pci.ids, instead of the usual two? Maybe you can add the output of lspci -nns 08:00 to explain those vfio-pci.ids?

As far as I understand all devices are part of GPU so I am passing all as I don't know what is important. I added output of lspci -nns 08:00 to the original post

 

[atari]

Is it possible to use 1660 super whit v-gpu patch?
I have Ryzen 9 5950x whit 1660super runing proxmox:
pveversion -v
proxmox-ve: 7.3-1 (running kernel: 5.19.17-1-pve)
pve-manager: 7.3-3 (running version: 7.3-3/c3928077)
pve-kernel-5.15: 7.2-14
pve-kernel-5.19: 7.2-14
pve-kernel-helper: 7.2-14
pve-kernel-5.19.17-1-pve: 5.19.17-1
pve-kernel-5.15.74-1-pve: 5.15.74-1
pve-kernel-5.15.64-1-pve: 5.15.64-1
pve-kernel-5.15.60-2-pve: 5.15.60-2
ceph-fuse: 15.2.14-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-1
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.3-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.3.1-1
proxmox-backup-file-restore: 2.3.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.0-1
proxmox-widget-toolkit: 3.5.3
pve-cluster: 7.3-1
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.5-6
pve-ha-manager: 3.5.1
pve-i18n: 2.8-1
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-1
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1


passthrough works but starting pached driver get error:
./NVIDIA-Linux-x86_64-525.60.12-vgpu-kvm-custom.run
Verifying archive integrity... OK
Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 525.60.12....................................................................................................................................................................................................................................................

ERROR: Unable to load the kernel module 'nvidia.ko'.

dmesg | grep -e DMAR -e IOMMU
[ 0.810231] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.811112] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 0.822894] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
root@pve:~# lspci |grep -i nvidia
08:00.0 VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1660 SUPER] (rev a1)
08:00.1 Audio device: NVIDIA Corporation TU116 High Definition Audio Controller (rev a1)
08:00.2 USB controller: NVIDIA Corporation TU116 USB 3.1 Host Controller (rev a1)
08:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 USB Type-C UCSI Controller (rev a1)
root@pve:~# lsmod | grep 'nvidia\|vfio'
vfio_pci 16384 4
vfio_pci_core 73728 1 vfio_pci
vfio_virqfd 16384 1 vfio_pci_core
vfio_iommu_type1 40960 1
vfio 45056 8 vfio_pci_core,vfio_iommu_type1
irqbypass 16384 161 vfio_pci_core,kvm
find /usr/lib/modules -name "*.ko" | grep -i nvidia
/usr/lib/modules/5.15.60-2-pve/kernel/drivers/platform/x86/nvidia-wmi-ec-backlight.ko
/usr/lib/modules/5.15.60-2-pve/kernel/drivers/usb/typec/altmodes/typec_nvidia.ko
/usr/lib/modules/5.15.60-2-pve/kernel/drivers/i2c/busses/i2c-nvidia-gpu.ko
/usr/lib/modules/5.15.60-2-pve/kernel/drivers/video/fbdev/nvidia/nvidiafb.ko
/usr/lib/modules/5.15.60-2-pve/kernel/drivers/net/ethernet/nvidia/forcedeth.ko
/usr/lib/modules/5.19.17-1-pve/kernel/drivers/platform/x86/nvidia-wmi-ec-backlight.ko
/usr/lib/modules/5.19.17-1-pve/kernel/drivers/usb/typec/altmodes/typec_nvidia.ko
/usr/lib/modules/5.19.17-1-pve/kernel/drivers/i2c/busses/i2c-nvidia-gpu.ko
/usr/lib/modules/5.19.17-1-pve/kernel/drivers/video/fbdev/nvidia/nvidiafb.ko
/usr/lib/modules/5.19.17-1-pve/kernel/drivers/net/ethernet/nvidia/forcedeth.ko
/usr/lib/modules/5.15.74-1-pve/updates/dkms/nvidia-drm.ko
/usr/lib/modules/5.15.74-1-pve/updates/dkms/nvidia.ko
/usr/lib/modules/5.15.74-1-pve/updates/dkms/nvidia-vgpu-vfio.ko
/usr/lib/modules/5.15.74-1-pve/updates/dkms/nvidia-uvm.ko
/usr/lib/modules/5.15.74-1-pve/updates/dkms/nvidia-modeset.ko
/usr/lib/modules/5.15.74-1-pve/kernel/drivers/platform/x86/nvidia-wmi-ec-backlight.ko
/usr/lib/modules/5.15.74-1-pve/kernel/drivers/usb/typec/altmodes/typec_nvidia.ko
/usr/lib/modules/5.15.74-1-pve/kernel/drivers/i2c/busses/i2c-nvidia-gpu.ko
/usr/lib/modules/5.15.74-1-pve/kernel/drivers/video/fbdev/nvidia/nvidiafb.ko
/usr/lib/modules/5.15.74-1-pve/kernel/drivers/net/ethernet/nvidia/forcedeth.ko