Hello,
I am having a lot of problems getting GPU Pass Through working on a HP Proliant DL 380 G8 with Vega 64 cards. I've got two installed and they work on VMWare ESXi 7.0. I am getting incredible benchmarks with them on Windows 10 VM's, but slower than I would expect FPS's in games like Fortnite for example. I can't really get higher than 60 FPS (It usually averages 40 - 50) and I would expect 120+ with these cards. I've tuned everything the best I can, but I want to get a little better performance out of this as I want to use this as a gaming server. Monitors are directly attached. I am trying Proxmox, but I am having a terrible time trying to get GPU Passthrough working. VMWare was relatively easy to get this working, but I've followed tons of guides and still I am not getting there with Proxmox which I thought would have been easier.
I really followed this guide:
https://www.reddit.com/r/homelab/comments/iw5cew/proxmox_i_created_a_script_to_fix_gpu_pass/
As that contains links to:
https://pve.proxmox.com/wiki/Pci_passthrough
It gives you steps to compile a new kernel to get around the "Device is ineligible for IOMMU domain attach due to platform RMRR requirement" RMRR errors you get on an HP Bios with non-supported Vega Cards.
However when I start the VM I get:
kvm: -device vfio-pci,host=0000:26:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1: VFIO_MAP_DMA failed: Invalid argument
kvm: -device vfio-pci,host=0000:26:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1: vfio 0000:26:00.1: failed to setup container for group 91: memory listener initialization failed: Region pc.ram: vfio_dma_map(0x7fa31e067740, 0x0, 0x80000000, 0x7f9f1c200000) = -22 (Invalid argument)
TASK ERROR: start failed: QEMU exited with code 1
As for some of my config files:
root@hpprxsrv01:/etc/pve/qemu-server# cat vmid.conf
hostpci0: 26:00.0;26:00.1
hostpci1: 09:00.0;09:00.1
root@hpprxsrv01:/etc/pve/qemu-server# cat /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1
root@hpprxsrv01:/etc/pve/qemu-server# cat /etc/modprobe.d/vfio.conf
softdep radeon pre: vfio-pci
softdep amdgpu pre: vfio-pci
softdep nouveau pre: vfio-pci
softdep drm pre: vfio-pci
options vfio-pci ids=1002:687f,1002:aaf8 disable_vga=1
root@hpprxsrv01:/etc/pve/qemu-server# cat 100.conf
bootdisk: ide0
cores: 4
hostpci0: 26:00,pcie=1,romfile=Powercolor.RXVega64.8192.180108.rom
ide0: local-lvm:vm-100-disk-0,size=512G
ide2: local:iso/virtio-win-0.1.189.iso,media=cdrom,size=488766K
machine: q35
memory: 16384
name: proxgamevm01
net0: e1000=xx:xx:xx:xx:x:x,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=a3174a62-264d-45a9-98b5-753826698d18
sockets: 2
usb0: host=0a5c:21e8,usb3=1
vmgenid: b8d70450-f823-4825-8e3e-316f31b8be8b
root@hpprxsrv01:~# cat /etc/modprobe.d/pve-blacklist.conf
# This file contains a list of modules which are not supported by Proxmox VE
# nidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
blacklist nouveau
blacklist nvidia
blacklist radeon
blacklist amdgpu
blacklist snd_hda_intel
root@hpprxsrv01:~# cat /etc/modprobe.d/blacklist.conf
blacklist nouveau
blacklist nvidia
blacklist radeon
blacklist amdgpu
blacklist snd_hda_intel
Startup GRUB Commad Line:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on intremap=no_x2apic_optout"
Excerpts of my syslog:
Oct 30 17:24:52 hpprxsrv01 systemd[1]: 100.scope: Succeeded.
Oct 30 17:24:52 hpprxsrv01 pvedaemon[26974]: start failed: QEMU exited with code 1
Oct 30 17:24:52 hpprxsrv01 pvedaemon[23206]: <root@pam> end task UPID:hpprxsrv01:0000695E:001067D5:5F9C849B:qmstart:100:root@pam: start failed: QEMU exited with code 1
Oct 30 17:24:53 hpprxsrv01 qmeventd[1037]: Starting cleanup for 100
Oct 30 17:24:53 hpprxsrv01 qmeventd[1037]: Finished cleanup for 100
Oct 30 17:24:58 hpprxsrv01 pveproxy[16259]: worker exit
Oct 30 17:24:58 hpprxsrv01 pveproxy[1437]: worker 16259 finished
Oct 30 17:24:58 hpprxsrv01 pveproxy[1437]: starting 1 worker(s)
Oct 30 17:24:58 hpprxsrv01 pveproxy[1437]: worker 27116 started
I'd be happy to provide any additional files if you feel it's worth while. Been struggling with this for a while and I do not seem to make any progress. Any help would be greatly appreciated!
I am having a lot of problems getting GPU Pass Through working on a HP Proliant DL 380 G8 with Vega 64 cards. I've got two installed and they work on VMWare ESXi 7.0. I am getting incredible benchmarks with them on Windows 10 VM's, but slower than I would expect FPS's in games like Fortnite for example. I can't really get higher than 60 FPS (It usually averages 40 - 50) and I would expect 120+ with these cards. I've tuned everything the best I can, but I want to get a little better performance out of this as I want to use this as a gaming server. Monitors are directly attached. I am trying Proxmox, but I am having a terrible time trying to get GPU Passthrough working. VMWare was relatively easy to get this working, but I've followed tons of guides and still I am not getting there with Proxmox which I thought would have been easier.
I really followed this guide:
https://www.reddit.com/r/homelab/comments/iw5cew/proxmox_i_created_a_script_to_fix_gpu_pass/
As that contains links to:
https://pve.proxmox.com/wiki/Pci_passthrough
It gives you steps to compile a new kernel to get around the "Device is ineligible for IOMMU domain attach due to platform RMRR requirement" RMRR errors you get on an HP Bios with non-supported Vega Cards.
However when I start the VM I get:
kvm: -device vfio-pci,host=0000:26:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1: VFIO_MAP_DMA failed: Invalid argument
kvm: -device vfio-pci,host=0000:26:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1: vfio 0000:26:00.1: failed to setup container for group 91: memory listener initialization failed: Region pc.ram: vfio_dma_map(0x7fa31e067740, 0x0, 0x80000000, 0x7f9f1c200000) = -22 (Invalid argument)
TASK ERROR: start failed: QEMU exited with code 1
As for some of my config files:
root@hpprxsrv01:/etc/pve/qemu-server# cat vmid.conf
hostpci0: 26:00.0;26:00.1
hostpci1: 09:00.0;09:00.1
root@hpprxsrv01:/etc/pve/qemu-server# cat /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1
root@hpprxsrv01:/etc/pve/qemu-server# cat /etc/modprobe.d/vfio.conf
softdep radeon pre: vfio-pci
softdep amdgpu pre: vfio-pci
softdep nouveau pre: vfio-pci
softdep drm pre: vfio-pci
options vfio-pci ids=1002:687f,1002:aaf8 disable_vga=1
root@hpprxsrv01:/etc/pve/qemu-server# cat 100.conf
bootdisk: ide0
cores: 4
hostpci0: 26:00,pcie=1,romfile=Powercolor.RXVega64.8192.180108.rom
ide0: local-lvm:vm-100-disk-0,size=512G
ide2: local:iso/virtio-win-0.1.189.iso,media=cdrom,size=488766K
machine: q35
memory: 16384
name: proxgamevm01
net0: e1000=xx:xx:xx:xx:x:x,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=a3174a62-264d-45a9-98b5-753826698d18
sockets: 2
usb0: host=0a5c:21e8,usb3=1
vmgenid: b8d70450-f823-4825-8e3e-316f31b8be8b
root@hpprxsrv01:~# cat /etc/modprobe.d/pve-blacklist.conf
# This file contains a list of modules which are not supported by Proxmox VE
# nidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
blacklist nouveau
blacklist nvidia
blacklist radeon
blacklist amdgpu
blacklist snd_hda_intel
root@hpprxsrv01:~# cat /etc/modprobe.d/blacklist.conf
blacklist nouveau
blacklist nvidia
blacklist radeon
blacklist amdgpu
blacklist snd_hda_intel
Startup GRUB Commad Line:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on intremap=no_x2apic_optout"
Excerpts of my syslog:
Oct 30 17:24:52 hpprxsrv01 systemd[1]: 100.scope: Succeeded.
Oct 30 17:24:52 hpprxsrv01 pvedaemon[26974]: start failed: QEMU exited with code 1
Oct 30 17:24:52 hpprxsrv01 pvedaemon[23206]: <root@pam> end task UPID:hpprxsrv01:0000695E:001067D5:5F9C849B:qmstart:100:root@pam: start failed: QEMU exited with code 1
Oct 30 17:24:53 hpprxsrv01 qmeventd[1037]: Starting cleanup for 100
Oct 30 17:24:53 hpprxsrv01 qmeventd[1037]: Finished cleanup for 100
Oct 30 17:24:58 hpprxsrv01 pveproxy[16259]: worker exit
Oct 30 17:24:58 hpprxsrv01 pveproxy[1437]: worker 16259 finished
Oct 30 17:24:58 hpprxsrv01 pveproxy[1437]: starting 1 worker(s)
Oct 30 17:24:58 hpprxsrv01 pveproxy[1437]: worker 27116 started
I'd be happy to provide any additional files if you feel it's worth while. Been struggling with this for a while and I do not seem to make any progress. Any help would be greatly appreciated!