I am trying to pass a Nvidia Quadro 2000 card to a Windows 10 virtual machine but am experiencing a few issues. First the system details:
Proxmox Version: 4.1-13/cfb599fb
Using pve-no-subscription repo
System last updated on Feb 16th 2016
uname -r
4.2.8-1-pve
cat /etc/pve/qemu-server/105.conf
agent: 1
bios: ovmf
bootdisk: virtio0
cores: 2
memory: 8192
name: Windows10-EFI
net0: virtio=36:66:33:39:34:32,bridge=vmbr4
numa: 0
ostype: win8
sata0: none,media=cdrom
scsihw: virtio-scsi-pci
smbios1: uuid=15abfaf2-77a6-4da5-9e20-38151e42c0ee
sockets: 1
virtio0: local-raid10:105/vm-105-disk-1.raw,cache=writeback,iothread=1,size=100G
virtio1: local-raid10:105/vm-105-disk-2.raw,cache=writeback,iothread=1,size=500G
usb0: host=1-3.1
hostpci0: 03:00,pcie=1,x-vga=on
lspci | grep -i Nvidia
03:00.0 VGA compatible controller: NVIDIA Corporation GF106GL [Quadro 2000] (rev a1)
03:00.1 Audio device: NVIDIA Corporation GF106 High Definition Audio Controller (rev a1)
cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:0dd8,10de:0be9
cat /etc/modprobe.d/pve-blacklist.conf
# This file contains a list of modules which are not supported by Proxmox VE
# nidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
blacklist nouveau
/etc/default/grub - relavent line:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on nouveau.modeset=0"
find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/2/devices/0000:00:03.0 # Graphics card I think!
/sys/kernel/iommu_groups/3/devices/0000:00:07.0
/sys/kernel/iommu_groups/4/devices/0000:00:09.0
/sys/kernel/iommu_groups/5/devices/0000:00:13.0
/sys/kernel/iommu_groups/6/devices/0000:00:14.0
/sys/kernel/iommu_groups/6/devices/0000:00:14.1
/sys/kernel/iommu_groups/6/devices/0000:00:14.2
/sys/kernel/iommu_groups/6/devices/0000:00:14.3
/sys/kernel/iommu_groups/7/devices/0000:00:16.0
/sys/kernel/iommu_groups/7/devices/0000:00:16.1
/sys/kernel/iommu_groups/7/devices/0000:00:16.2
/sys/kernel/iommu_groups/7/devices/0000:00:16.3
/sys/kernel/iommu_groups/7/devices/0000:00:16.4
/sys/kernel/iommu_groups/7/devices/0000:00:16.5
/sys/kernel/iommu_groups/7/devices/0000:00:16.6
/sys/kernel/iommu_groups/7/devices/0000:00:16.7
/sys/kernel/iommu_groups/8/devices/0000:00:1a.0
/sys/kernel/iommu_groups/8/devices/0000:00:1a.1
/sys/kernel/iommu_groups/8/devices/0000:00:1a.2
/sys/kernel/iommu_groups/8/devices/0000:00:1a.7
/sys/kernel/iommu_groups/9/devices/0000:00:1c.0
/sys/kernel/iommu_groups/9/devices/0000:00:1c.4
/sys/kernel/iommu_groups/9/devices/0000:00:1c.5
/sys/kernel/iommu_groups/9/devices/0000:05:00.0
/sys/kernel/iommu_groups/9/devices/0000:06:00.0
/sys/kernel/iommu_groups/9/devices/0000:07:00.0
/sys/kernel/iommu_groups/10/devices/0000:00:1d.0
/sys/kernel/iommu_groups/10/devices/0000:00:1d.1
/sys/kernel/iommu_groups/10/devices/0000:00:1d.2
/sys/kernel/iommu_groups/10/devices/0000:00:1d.7
/sys/kernel/iommu_groups/11/devices/0000:00:1e.0
/sys/kernel/iommu_groups/11/devices/0000:08:01.0
/sys/kernel/iommu_groups/11/devices/0000:08:03.0
/sys/kernel/iommu_groups/11/devices/0000:08:04.0
/sys/kernel/iommu_groups/12/devices/0000:00:1f.0
/sys/kernel/iommu_groups/12/devices/0000:00:1f.2
/sys/kernel/iommu_groups/12/devices/0000:00:1f.3
/sys/kernel/iommu_groups/13/devices/0000:01:00.0
/sys/kernel/iommu_groups/14/devices/0000:03:00.0
/sys/kernel/iommu_groups/14/devices/0000:03:00.1
/sys/kernel/iommu_groups/15/devices/0000:04:00.0
/sys/kernel/iommu_groups/16/devices/0000:fe:00.0
/sys/kernel/iommu_groups/16/devices/0000:fe:00.1
/sys/kernel/iommu_groups/17/devices/0000:fe:02.0
/sys/kernel/iommu_groups/17/devices/0000:fe:02.1
/sys/kernel/iommu_groups/17/devices/0000:fe:02.2
/sys/kernel/iommu_groups/17/devices/0000:fe:02.3
/sys/kernel/iommu_groups/17/devices/0000:fe:02.4
/sys/kernel/iommu_groups/17/devices/0000:fe:02.5
/sys/kernel/iommu_groups/18/devices/0000:fe:03.0
/sys/kernel/iommu_groups/18/devices/0000:fe:03.1
/sys/kernel/iommu_groups/18/devices/0000:fe:03.2
/sys/kernel/iommu_groups/18/devices/0000:fe:03.4
/sys/kernel/iommu_groups/19/devices/0000:fe:04.0
/sys/kernel/iommu_groups/19/devices/0000:fe:04.1
/sys/kernel/iommu_groups/19/devices/0000:fe:04.2
/sys/kernel/iommu_groups/19/devices/0000:fe:04.3
/sys/kernel/iommu_groups/20/devices/0000:fe:05.0
/sys/kernel/iommu_groups/20/devices/0000:fe:05.1
/sys/kernel/iommu_groups/20/devices/0000:fe:05.2
/sys/kernel/iommu_groups/20/devices/0000:fe:05.3
/sys/kernel/iommu_groups/21/devices/0000:fe:06.0
/sys/kernel/iommu_groups/21/devices/0000:fe:06.1
/sys/kernel/iommu_groups/21/devices/0000:fe:06.2
/sys/kernel/iommu_groups/21/devices/0000:fe:06.3
/sys/kernel/iommu_groups/22/devices/0000:ff:00.0
/sys/kernel/iommu_groups/22/devices/0000:ff:00.1
/sys/kernel/iommu_groups/23/devices/0000:ff:02.0
/sys/kernel/iommu_groups/23/devices/0000:ff:02.1
/sys/kernel/iommu_groups/23/devices/0000:ff:02.2
/sys/kernel/iommu_groups/23/devices/0000:ff:02.3
/sys/kernel/iommu_groups/23/devices/0000:ff:02.4
/sys/kernel/iommu_groups/23/devices/0000:ff:02.5
/sys/kernel/iommu_groups/24/devices/0000:ff:03.0
/sys/kernel/iommu_groups/24/devices/0000:ff:03.1
/sys/kernel/iommu_groups/24/devices/0000:ff:03.2
/sys/kernel/iommu_groups/24/devices/0000:ff:03.4
/sys/kernel/iommu_groups/25/devices/0000:ff:04.0
/sys/kernel/iommu_groups/25/devices/0000:ff:04.1
/sys/kernel/iommu_groups/25/devices/0000:ff:04.2
/sys/kernel/iommu_groups/25/devices/0000:ff:04.3
/sys/kernel/iommu_groups/26/devices/0000:ff:05.0
/sys/kernel/iommu_groups/26/devices/0000:ff:05.1
/sys/kernel/iommu_groups/26/devices/0000:ff:05.2
/sys/kernel/iommu_groups/26/devices/0000:ff:05.3
/sys/kernel/iommu_groups/27/devices/0000:ff:06.0
/sys/kernel/iommu_groups/27/devices/0000:ff:06.1
/sys/kernel/iommu_groups/27/devices/0000:ff:06.2
/sys/kernel/iommu_groups/27/devices/0000:ff:06.3
The issue:
If I start the Proxmox server up from a clean boot and load the Windows 10 virtual machine I am able to successfully use the GPU. However once the Windows virtual machine is rebooted or power cycled the GPU no longer works and I just get a black screen on the monitor. It takes a full Proxmox server restart in order to bring the GPU back to a working state. I have duplicated this issue on a Windows 10 VM using Seabios as well as a Windows 7 VM using Seabios. The current testing system is using OVMF as can be seen from the config.
Proxmox Version: 4.1-13/cfb599fb
Using pve-no-subscription repo
System last updated on Feb 16th 2016
uname -r
4.2.8-1-pve
cat /etc/pve/qemu-server/105.conf
agent: 1
bios: ovmf
bootdisk: virtio0
cores: 2
memory: 8192
name: Windows10-EFI
net0: virtio=36:66:33:39:34:32,bridge=vmbr4
numa: 0
ostype: win8
sata0: none,media=cdrom
scsihw: virtio-scsi-pci
smbios1: uuid=15abfaf2-77a6-4da5-9e20-38151e42c0ee
sockets: 1
virtio0: local-raid10:105/vm-105-disk-1.raw,cache=writeback,iothread=1,size=100G
virtio1: local-raid10:105/vm-105-disk-2.raw,cache=writeback,iothread=1,size=500G
usb0: host=1-3.1
hostpci0: 03:00,pcie=1,x-vga=on
lspci | grep -i Nvidia
03:00.0 VGA compatible controller: NVIDIA Corporation GF106GL [Quadro 2000] (rev a1)
03:00.1 Audio device: NVIDIA Corporation GF106 High Definition Audio Controller (rev a1)
cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:0dd8,10de:0be9
cat /etc/modprobe.d/pve-blacklist.conf
# This file contains a list of modules which are not supported by Proxmox VE
# nidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
blacklist nouveau
/etc/default/grub - relavent line:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on nouveau.modeset=0"
find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/2/devices/0000:00:03.0 # Graphics card I think!
/sys/kernel/iommu_groups/3/devices/0000:00:07.0
/sys/kernel/iommu_groups/4/devices/0000:00:09.0
/sys/kernel/iommu_groups/5/devices/0000:00:13.0
/sys/kernel/iommu_groups/6/devices/0000:00:14.0
/sys/kernel/iommu_groups/6/devices/0000:00:14.1
/sys/kernel/iommu_groups/6/devices/0000:00:14.2
/sys/kernel/iommu_groups/6/devices/0000:00:14.3
/sys/kernel/iommu_groups/7/devices/0000:00:16.0
/sys/kernel/iommu_groups/7/devices/0000:00:16.1
/sys/kernel/iommu_groups/7/devices/0000:00:16.2
/sys/kernel/iommu_groups/7/devices/0000:00:16.3
/sys/kernel/iommu_groups/7/devices/0000:00:16.4
/sys/kernel/iommu_groups/7/devices/0000:00:16.5
/sys/kernel/iommu_groups/7/devices/0000:00:16.6
/sys/kernel/iommu_groups/7/devices/0000:00:16.7
/sys/kernel/iommu_groups/8/devices/0000:00:1a.0
/sys/kernel/iommu_groups/8/devices/0000:00:1a.1
/sys/kernel/iommu_groups/8/devices/0000:00:1a.2
/sys/kernel/iommu_groups/8/devices/0000:00:1a.7
/sys/kernel/iommu_groups/9/devices/0000:00:1c.0
/sys/kernel/iommu_groups/9/devices/0000:00:1c.4
/sys/kernel/iommu_groups/9/devices/0000:00:1c.5
/sys/kernel/iommu_groups/9/devices/0000:05:00.0
/sys/kernel/iommu_groups/9/devices/0000:06:00.0
/sys/kernel/iommu_groups/9/devices/0000:07:00.0
/sys/kernel/iommu_groups/10/devices/0000:00:1d.0
/sys/kernel/iommu_groups/10/devices/0000:00:1d.1
/sys/kernel/iommu_groups/10/devices/0000:00:1d.2
/sys/kernel/iommu_groups/10/devices/0000:00:1d.7
/sys/kernel/iommu_groups/11/devices/0000:00:1e.0
/sys/kernel/iommu_groups/11/devices/0000:08:01.0
/sys/kernel/iommu_groups/11/devices/0000:08:03.0
/sys/kernel/iommu_groups/11/devices/0000:08:04.0
/sys/kernel/iommu_groups/12/devices/0000:00:1f.0
/sys/kernel/iommu_groups/12/devices/0000:00:1f.2
/sys/kernel/iommu_groups/12/devices/0000:00:1f.3
/sys/kernel/iommu_groups/13/devices/0000:01:00.0
/sys/kernel/iommu_groups/14/devices/0000:03:00.0
/sys/kernel/iommu_groups/14/devices/0000:03:00.1
/sys/kernel/iommu_groups/15/devices/0000:04:00.0
/sys/kernel/iommu_groups/16/devices/0000:fe:00.0
/sys/kernel/iommu_groups/16/devices/0000:fe:00.1
/sys/kernel/iommu_groups/17/devices/0000:fe:02.0
/sys/kernel/iommu_groups/17/devices/0000:fe:02.1
/sys/kernel/iommu_groups/17/devices/0000:fe:02.2
/sys/kernel/iommu_groups/17/devices/0000:fe:02.3
/sys/kernel/iommu_groups/17/devices/0000:fe:02.4
/sys/kernel/iommu_groups/17/devices/0000:fe:02.5
/sys/kernel/iommu_groups/18/devices/0000:fe:03.0
/sys/kernel/iommu_groups/18/devices/0000:fe:03.1
/sys/kernel/iommu_groups/18/devices/0000:fe:03.2
/sys/kernel/iommu_groups/18/devices/0000:fe:03.4
/sys/kernel/iommu_groups/19/devices/0000:fe:04.0
/sys/kernel/iommu_groups/19/devices/0000:fe:04.1
/sys/kernel/iommu_groups/19/devices/0000:fe:04.2
/sys/kernel/iommu_groups/19/devices/0000:fe:04.3
/sys/kernel/iommu_groups/20/devices/0000:fe:05.0
/sys/kernel/iommu_groups/20/devices/0000:fe:05.1
/sys/kernel/iommu_groups/20/devices/0000:fe:05.2
/sys/kernel/iommu_groups/20/devices/0000:fe:05.3
/sys/kernel/iommu_groups/21/devices/0000:fe:06.0
/sys/kernel/iommu_groups/21/devices/0000:fe:06.1
/sys/kernel/iommu_groups/21/devices/0000:fe:06.2
/sys/kernel/iommu_groups/21/devices/0000:fe:06.3
/sys/kernel/iommu_groups/22/devices/0000:ff:00.0
/sys/kernel/iommu_groups/22/devices/0000:ff:00.1
/sys/kernel/iommu_groups/23/devices/0000:ff:02.0
/sys/kernel/iommu_groups/23/devices/0000:ff:02.1
/sys/kernel/iommu_groups/23/devices/0000:ff:02.2
/sys/kernel/iommu_groups/23/devices/0000:ff:02.3
/sys/kernel/iommu_groups/23/devices/0000:ff:02.4
/sys/kernel/iommu_groups/23/devices/0000:ff:02.5
/sys/kernel/iommu_groups/24/devices/0000:ff:03.0
/sys/kernel/iommu_groups/24/devices/0000:ff:03.1
/sys/kernel/iommu_groups/24/devices/0000:ff:03.2
/sys/kernel/iommu_groups/24/devices/0000:ff:03.4
/sys/kernel/iommu_groups/25/devices/0000:ff:04.0
/sys/kernel/iommu_groups/25/devices/0000:ff:04.1
/sys/kernel/iommu_groups/25/devices/0000:ff:04.2
/sys/kernel/iommu_groups/25/devices/0000:ff:04.3
/sys/kernel/iommu_groups/26/devices/0000:ff:05.0
/sys/kernel/iommu_groups/26/devices/0000:ff:05.1
/sys/kernel/iommu_groups/26/devices/0000:ff:05.2
/sys/kernel/iommu_groups/26/devices/0000:ff:05.3
/sys/kernel/iommu_groups/27/devices/0000:ff:06.0
/sys/kernel/iommu_groups/27/devices/0000:ff:06.1
/sys/kernel/iommu_groups/27/devices/0000:ff:06.2
/sys/kernel/iommu_groups/27/devices/0000:ff:06.3
The issue:
If I start the Proxmox server up from a clean boot and load the Windows 10 virtual machine I am able to successfully use the GPU. However once the Windows virtual machine is rebooted or power cycled the GPU no longer works and I just get a black screen on the monitor. It takes a full Proxmox server restart in order to bring the GPU back to a working state. I have duplicated this issue on a Windows 10 VM using Seabios as well as a Windows 7 VM using Seabios. The current testing system is using OVMF as can be seen from the config.
