Hello everyone,
I would like to set up a GPU passthrough on a Proxmox server with Windows VMs, following the proxmox wiki (https://pve.proxmox.com/wiki/Pci_passthrough#GPU_Passthrough). Two graphic cards (Titan RTX A40) are availables and will be used inside two different VM.
Everything seems to work properly and I am able to see the graphic card inside the VM but with the error "This device cannot find enough free that it can use (Code 12)".
I configured the Proxmox server as follow :
/etc/default/grub : (I tried a lot of different configurations)
I have updated grub and rebooted the server.
IOMMU is enabled (not auto) on BIOS. Strange thing, when amd_iommu=on is set, I don't have "DMAR: IOMMU enabled" as you can see above but when I try intel_iommu=on, IOMMU is enabled in dmesg. The CPU is an AMD EPYC, so I don't understand why amd_iommu parameter doesn't show IOMMU enabled and intel_iommu does.
Modules are added :
Interrupt remapping is enabled :
The two graphic cards are isolated in IOMMU groups :
/etc/modprobe.d/vfio.conf
Drivers are blacklisted :
/etc/modprobe.d/blacklist.conf
The Windows VM is configured as follow :
When I try to check if the graphic card is OVMF compatible, echo 1 > rom returns Permission denied (like in this post https://forum.proxmox.com/threads/unable-to-dump-my-video-card-vbios.100041/)
A monitor is connected to each graphic card and I can connect to the VM via RDP.
I also tried with a Linux VM and obtained this error message with the nvidia-smi command : "NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running."
I don't understand why I can't use the GPU while it is seen into the VM. If someone can help me, I would be grateful.
I would like to set up a GPU passthrough on a Proxmox server with Windows VMs, following the proxmox wiki (https://pve.proxmox.com/wiki/Pci_passthrough#GPU_Passthrough). Two graphic cards (Titan RTX A40) are availables and will be used inside two different VM.
Everything seems to work properly and I am able to see the graphic card inside the VM but with the error "This device cannot find enough free that it can use (Code 12)".
I configured the Proxmox server as follow :
/etc/default/grub : (I tried a lot of different configurations)
Code:
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt video=efifb:off video=vesa:off"
Code:
dmesg | grep -e DMAR -e IOMMU
[ 6.064564] pci 0000:60:00.2: AMD-Vi: IOMMU performance counters supported
[ 6.064617] pci 0000:40:00.2: AMD-Vi: IOMMU performance counters supported
[ 6.064647] pci 0000:20:00.2: AMD-Vi: IOMMU performance counters supported
[ 6.064679] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 6.064719] pci 0000:e0:00.2: AMD-Vi: IOMMU performance counters supported
[ 6.064760] pci 0000:c0:00.2: AMD-Vi: IOMMU performance counters supported
[ 6.064804] pci 0000:a0:00.2: AMD-Vi: IOMMU performance counters supported
[ 6.064851] pci 0000:80:00.2: AMD-Vi: IOMMU performance counters supported
[ 6.082871] pci 0000:60:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 6.082883] pci 0000:40:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 6.082890] pci 0000:20:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 6.082897] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 6.082903] pci 0000:e0:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 6.082909] pci 0000:c0:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 6.082916] pci 0000:a0:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 6.082922] pci 0000:80:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 6.120179] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[ 6.120269] perf/amd_iommu: Detected AMD IOMMU #1 (2 banks, 4 counters/bank).
[ 6.120361] perf/amd_iommu: Detected AMD IOMMU #2 (2 banks, 4 counters/bank).
[ 6.120455] perf/amd_iommu: Detected AMD IOMMU #3 (2 banks, 4 counters/bank).
[ 6.120550] perf/amd_iommu: Detected AMD IOMMU #4 (2 banks, 4 counters/bank).
[ 6.120649] perf/amd_iommu: Detected AMD IOMMU #5 (2 banks, 4 counters/bank).
[ 6.120750] perf/amd_iommu: Detected AMD IOMMU #6 (2 banks, 4 counters/bank).
[ 6.120834] perf/amd_iommu: Detected AMD IOMMU #7 (2 banks, 4 counters/bank).
IOMMU is enabled (not auto) on BIOS. Strange thing, when amd_iommu=on is set, I don't have "DMAR: IOMMU enabled" as you can see above but when I try intel_iommu=on, IOMMU is enabled in dmesg. The CPU is an AMD EPYC, so I don't understand why amd_iommu parameter doesn't show IOMMU enabled and intel_iommu does.
Modules are added :
Code:
lsmod | grep ^vfio
vfio_pci 57344 1
vfio_virqfd 16384 1 vfio_pci
vfio_iommu_type1 40960 1
vfio 36864 5 vfio_iommu_type1,vfio_pci
Interrupt remapping is enabled :
Code:
dmesg | grep 'remapping'
[ 6.082928] AMD-Vi: Interrupt remapping enabled
The two graphic cards are isolated in IOMMU groups :
Code:
IOMMU Group 24:
41:00.0 3D controller [0302]: NVIDIA Corporation GA102GL [RTX A40] [10de:2235] (rev a1)
IOMMU Group 108:
a1:00.0 3D controller [0302]: NVIDIA Corporation GA102GL [RTX A40] [10de:2235] (rev a1)
/etc/modprobe.d/vfio.conf
Code:
options vfio-pci ids=10de:2235 disable_vga=1
Drivers are blacklisted :
/etc/modprobe.d/blacklist.conf
Code:
blacklist nouveau
blacklist nvidia
blacklist nvidiafb
blacklist nvidia-gpu
The Windows VM is configured as follow :
Code:
agent: 1
bios: ovmf
boot: order=scsi0;net0
cores: 8
efidisk0: tank:vm-101-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:41:00,pcie=1,x-vga=1
machine: pc-q35-6.1
memory: 16384
meta: creation-qemu=6.1.0,ctime=1653398297
name: gpu
net0: virtio=C6:69:19:96:9B:BF,bridge=vmbr0
numa: 0
ostype: win10
scsi0: local-lvm:vm-101-disk-0,cache=writeback,discard=on,size=200G
scsihw: virtio-scsi-pci
smbios1: uuid=e47c6ba2-9135-409b-8ead-2e659b31ee47
sockets: 1
vga: none
vmgenid: c0314fed-30cc-4d87-a3ad-973feac05d4d
When I try to check if the graphic card is OVMF compatible, echo 1 > rom returns Permission denied (like in this post https://forum.proxmox.com/threads/unable-to-dump-my-video-card-vbios.100041/)
A monitor is connected to each graphic card and I can connect to the VM via RDP.
I also tried with a Linux VM and obtained this error message with the nvidia-smi command : "NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running."
I don't understand why I can't use the GPU while it is seen into the VM. If someone can help me, I would be grateful.