RTX A6000 vGPU Passthrough not working

asparkoffire · Jun 9, 2022

Hello,

I'm trying to use vGPU Passthrough on RTX A6000 but when I start the VM after mdevctl Configuration, it fails with the following error.

Code:

()
kvm: -device vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:1b:00.4/00000000-0000-0000-0000-000000000104,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0: vfio 00000000-0000-0000-0000-000000000104: failed to get region 1 info: Input/output error
TASK ERROR: start failed: QEMU exited with code 1

Thanks

asparkoffire · Jun 13, 2022

Anyone?

dcsapak · Jun 13, 2022

sadly i have no vgpu capable nvidia card here to test, but can you post the vm config and the journal ? maybe we can see something there...

yayster · Jun 13, 2022

I have been working hard to pass through an RTX A6000 to a Windows 10 pro guest system.

Screen Shot 2022-06-13 at 8.03.40 AM.png

The GPU is added:

Screen Shot 2022-06-13 at 8.05.45 AM.png

My problem is not the guest system but the host proxmox system. The following log files on the host system would begin to balloon in size:

/var/log/syslog
/var/log/kern.log
/var/log/messages
/var/log/user.log
/var/log/journal/*

When all said and done, and the VM started, the host OS starts filling up various log files with messages detailing complaints such as:

Screen Shot 2022-06-13 at 8.26.12 AM.png

The thought is that some simple configuration change in /etc/default/grub would allow everything to work correctly. However, though many hours – no, days – were spend reading, researching, and testing various permutations nothing proved to be the key to make things work.

HARDWARE CONFIGURATION:

DELL Precision 5820 Tower X-Series
MOTHERBOARD: Base board Dell 0XNJY Version A00
CPU: Intel Core i9-10900X @3.70 GHz
MEMORY: 32GB
GPU: NVIDIA RTX A6000

SOFTWARE CONFIGURATION:

HOST: Proxmox version 7.2, Wed 04 May 2022 07:30:00 AM CEST
GUEST: Windows 10 Pro

GRUB CONFIGURATION:

/etc/default/grub was modified with the following line:

Code:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off video=efifb:off video=simplefb:off"

Afterwards, the update-grub command was run.

VFIO MODULES:

/etc/modules was updated with the following:

Code:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

IOMMU interrupt remapping, Blacklisting Drivers, and adding GPU to VFIO

In the /etc/modprobe.d directory, several things were done:

Code:

root@proxmox01:~# ls -l /etc/modprobe.d
total 20
-rw-r--r-- 1 root root  71 Jun 12 12:17 blacklist.conf
-rw-r--r-- 1 root root  51 Jun  7 10:52 iommu_unsafe_interrupts.conf
-rw-r--r-- 1 root root  26 Jun  7 10:53 kvm.conf
-rw-r--r-- 1 root root 171 May  3 23:46 pve-blacklist.conf
-rw-r--r-- 1 root root  55 Jun  7 11:03 vfio.conf
root@proxmox01:~# cat /etc/modprobe.d/blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia
blacklist nvidiafb
root@proxmox01:~# cat /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1
root@proxmox01:~# cat /etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1
root@proxmox01:~# cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:2230,10de:1aef disable_vga=1
root@proxmox01:~# cat /etc/modprobe.d/pve-blacklist.conf
# This file contains a list of modules which are not supported by Proxmox VE
# nidiafb see bugreport 701 -- kernel 4.1 does not boot on nvidiafb: unable to setup MTRR blacklist nvidiafb
root@proxmox01:~#

Afterwards, update-initramfs -u followed by a reboot.

leesteken · Jun 13, 2022

yayster said:
I have been working hard to pass through an RTX A6000 to a Windows 10 pro guest system.

Check if BOOTFB is in cat /proc/iomem after booting the Proxmox host and before starting the VM. If it is there, then you are having the same problem (BAR 1 cannot reserve) as other people doing PCIe passthrough with NVidia GPUs and pve-kernel 5.15. Maybe you can find the work-around of virtually disconnecting the GPU and rescanning the PCI bus in one of those threads.

@asparkoffire, who originally started this thread, was not trying to do PCIe passthrough but had an issue with mediated GPU passthrough instead.

yayster · Jun 13, 2022

leesteken said:
Check if BOOTFB is in cat /proc/iomem after booting the Proxmox host and before starting the VM. If it is there, then you are having the same problem (BAR 1 cannot reserve) as other people doing PCIe passthrough with NVidia GPUs and pve-kernel 5.15. Maybe you can find the work-around of virtually disconnecting the GPU and rescanning the PCI bus in one of those threads.

@asparkoffire, who originally started this thread, was not trying to do PCIe passthrough but had an issue with mediated GPU passthrough instead.

Maybe you can find the work-around of virtually disconnecting the GPU and rescanning the PCI bus in one of those threads.

This is the correct answer! Thank you very much!

https://forum.proxmox.com/threads/problem-with-gpu-passthrough.55918/#post-257462 gives said solution

punjprateek · Jun 14, 2022

yayster said:
I have been working hard to pass through an RTX A6000 to a Windows 10 pro guest system.

View attachment 37926

The GPU is added:
View attachment 37927

My problem is not the guest system but the host proxmox system. The following log files on the host system would begin to balloon in size:

/var/log/syslog
/var/log/kern.log
/var/log/messages
/var/log/user.log
/var/log/journal/*

When all said and done, and the VM started, the host OS starts filling up various log files with messages detailing complaints such as:

View attachment 37928

The thought is that some simple configuration change in /etc/default/grub would allow everything to work correctly. However, though many hours – no, days – were spend reading, researching, and testing various permutations nothing proved to be the key to make things work.

HARDWARE CONFIGURATION:
DELL Precision 5820 Tower X-Series
MOTHERBOARD: Base board Dell 0XNJY Version A00
CPU: Intel Core i9-10900X @3.70 GHz
MEMORY: 32GB
GPU: NVIDIA RTX A6000

SOFTWARE CONFIGURATION:
HOST: Proxmox version 7.2, Wed 04 May 2022 07:30:00 AM CEST
GUEST: Windows 10 Pro

GRUB CONFIGURATION:
/etc/default/grub was modified with the following line:

Code:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off video=efifb:off video=simplefb:off"

Afterwards, the update-grub command was run.

VFIO MODULES:
/etc/modules was updated with the following:

Code:

vfio vfio_iommu_type1 vfio_pci vfio_virqfd

IOMMU interrupt remapping, Blacklisting Drivers, and adding GPU to VFIO
In the /etc/modprobe.d directory, several things were done:

Code:

root@proxmox01:~# ls -l /etc/modprobe.d total 20 -rw-r--r-- 1 root root 71 Jun 12 12:17 blacklist.conf -rw-r--r-- 1 root root 51 Jun 7 10:52 iommu_unsafe_interrupts.conf -rw-r--r-- 1 root root 26 Jun 7 10:53 kvm.conf -rw-r--r-- 1 root root 171 May 3 23:46 pve-blacklist.conf -rw-r--r-- 1 root root 55 Jun 7 11:03 vfio.conf root@proxmox01:~# cat /etc/modprobe.d/blacklist.conf blacklist radeon blacklist nouveau blacklist nvidia blacklist nvidiafb root@proxmox01:~# cat /etc/modprobe.d/iommu_unsafe_interrupts.conf options vfio_iommu_type1 allow_unsafe_interrupts=1 root@proxmox01:~# cat /etc/modprobe.d/kvm.conf options kvm ignore_msrs=1 root@proxmox01:~# cat /etc/modprobe.d/vfio.conf options vfio-pci ids=10de:2230,10de:1aef disable_vga=1 root@proxmox01:~# cat /etc/modprobe.d/pve-blacklist.conf # This file contains a list of modules which are not supported by Proxmox VE # nidiafb see bugreport 701 -- kernel 4.1 does not boot on nvidiafb: unable to setup MTRR blacklist nvidiafb root@proxmox01:~#

Afterwards, update-initramfs -u followed by a reboot.

GPU Passthrough is not a problem but unable to use VGPU after changing the mode to display less as nvidia suggest and creating mdev profiles which we are unable to make it work with A6000

yayster · Jun 18, 2022

punjprateek said:
GPU Passthrough is not a problem but unable to use VGPU after changing the mode to display less as nvidia suggest and creating mdev profiles which we are unable to make it work with A6000

I have mine working. Happy to help to see if there are any differences in setup.

punjprateek · Jun 20, 2022

yayster said:
I have mine working. Happy to help to see if there are any differences in setup.

Have you successfully tested vGPU Profiles with mdev ?

yayster · Jun 20, 2022

punjprateek said:
Have you successfully tested vGPU Profiles with mdev ?

I cannot say that I have -- but I am interested in learning more.

The system is now being stressed by other people in our organization using an application called vmix.

Republicus · Jun 20, 2022

vGPU works on my end with little troubles. I switched to Rust based vGPU-unlock which simplified the process and works well

punjprateek · Jun 21, 2022

Republicus said:
vGPU works on my end with little troubles. I switched to Rust based vGPU-unlock which simplified the process and works well

Do you have RTX A6000 ??, if not then you are commenting on wrong thread bro

Docop2 · Jun 22, 2022

I do have the whole Quadro lineup and the A4000. While vgpu can act as a Grid card.. a simple update of prox will kill it. We already have a buggy version with 7.2 ... And installing the nvidia drv directly, we then loose the possibility to pass the entire card or it still possible ? As usualy the module need to load the ingnore of the nvidia...

Republicus · Jun 23, 2022

punjprateek said:
Do you have RTX A6000 ??, if not then you are commenting on wrong thread bro

Have you yet tried vgpu-unlock-rs and had the same results?

@Docop2 would a vgpu profile override using the entirety of the card's resources work?

punjprateek · Jun 23, 2022

Docop2 said:
I do have the whole Quadro lineup and the A4000. While vgpu can act as a Grid card.. a simple update of prox will kill it. We already have a buggy version with 7.2 ... And installing the nvidia drv directly, we then loose the possibility to pass the entire card or it still possible ? As usualy the module need to load the ingnore of the nvidia...

GPU Passthrough just works fine for A4000/A6000 on proxmox 7.2 also

RTX A6000 vGPU Passthrough not working

New Member

New Member

Proxmox Staff Member

New Member

HARDWARE CONFIGURATION:​

SOFTWARE CONFIGURATION:​

GRUB CONFIGURATION:​

VFIO MODULES:​

IOMMU interrupt remapping, Blacklisting Drivers, and adding GPU to VFIO​

Distinguished Member

New Member

Member

HARDWARE CONFIGURATION:​

SOFTWARE CONFIGURATION:​

GRUB CONFIGURATION:​

VFIO MODULES:​

IOMMU interrupt remapping, Blacklisting Drivers, and adding GPU to VFIO​

New Member

Member

New Member

Well-Known Member

Member

Member

Well-Known Member

Member

HARDWARE CONFIGURATION:

SOFTWARE CONFIGURATION:

GRUB CONFIGURATION:

VFIO MODULES:

IOMMU interrupt remapping, Blacklisting Drivers, and adding GPU to VFIO

HARDWARE CONFIGURATION:

SOFTWARE CONFIGURATION:

GRUB CONFIGURATION:

VFIO MODULES:

IOMMU interrupt remapping, Blacklisting Drivers, and adding GPU to VFIO