Hi,
I know there are a lot of posts about (not working) PCie passthrough, and I am pretty sure I have read all of them. However, I do not seem to get it to work on my Quadro P400.
System spec:
Asus Crosshair V Formula
AMD FX-8350
Quadro P400
16 GB Ram
Standard SATA SSDs
The Quadro is the only GPU in the system. (I have read multiple times that this might be an issue, others say it isn't...)
Proxmox version:
Since the Crosshair V Formular (or in generall many boards with this chipset) is known to have a buggy bios, a hack has to be applied.
Here is my
Most people seem to use
This is the content of my
And the content of my
This is the output of
Which.... seems fine?
The output of
Which seems to confirm that 1) the GPU is found correctly and B) the vfio-pci driver is in use.
At this point, I think that hardware/host wise, I am done? And that everything is working? Is there anything I could do to further check if the issue is related to my hardware or proxmox config?
I have set up a Windows 10 Pro (N) 21H1 VM with the following config:
The setup went smoothly, I installed nvidia driver 450.x and got a bluescreen related to
Booting with "primary GPU" unchecked works, but then I get a Code 43 in the device Manager for the Quadro P400.
Strangely, in the device manager I also see a PCI Device (bus 6, device 3) with a Code 28, and I have no Idea what that could be.
I would greatly appreciate any suggestions you might have
I know there are a lot of posts about (not working) PCie passthrough, and I am pretty sure I have read all of them. However, I do not seem to get it to work on my Quadro P400.
System spec:
Asus Crosshair V Formula
AMD FX-8350
Quadro P400
16 GB Ram
Standard SATA SSDs
The Quadro is the only GPU in the system. (I have read multiple times that this might be an issue, others say it isn't...)
Proxmox version:
Code:
pve-manager/7.0-8/b1dbf562 (running kernel: 5.11.22-1-pve)Since the Crosshair V Formular (or in generall many boards with this chipset) is known to have a buggy bios, a hack has to be applied.
Here is my
etc/default/grub:
Code:
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=fullflush iommu=pt pcie_acs_override=downstream ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1 video=efifb:off"Most people seem to use
amd_iommu=on, but this doesnt seem to be valid according to the doc.This is the content of my
/etc/modprobe.d/blacklist.conf file:
Code:
blacklist nouveau
blacklist nvidia
blacklist snd_hda_intelAnd the content of my
/etc/modules file:
Code:
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfdThis is the output of
dmesg | grep "AMD-Vi":
Code:
root@proxmox001:~# dmesg | grep "AMD-Vi"
[ 2.050945] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 2.050947] AMD-Vi: Interrupt remapping enabled
[ 2.051047] AMD-Vi: IO/TLB flush on unmap enabled
root@proxmox001:~#Which.... seems fine?
The output of
lspci -v: (Removed other details for shortness:
Code:
01:00.0 VGA compatible controller: NVIDIA Corporation GP107GL [Quadro P400] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation GP107GL [Quadro P400]
Flags: bus master, fast devsel, latency 0, IRQ 65, NUMA node 0, IOMMU group 16
Memory at fd000000 (32-bit, non-prefetchable) [size=16M]
Memory at c0000000 (64-bit, prefetchable) [size=256M]
Memory at d0000000 (64-bit, prefetchable) [size=32M]
I/O ports at e000 [size=128]
Expansion ROM at 000c0000 [disabled] [size=128K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Legacy Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [250] Latency Tolerance Reporting
Capabilities: [128] Power Budgeting <?>
Capabilities: [420] Advanced Error Reporting
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900] Secondary PCI Express
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau
01:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1)
Subsystem: NVIDIA Corporation GP107GL High Definition Audio Controller
Flags: bus master, fast devsel, latency 0, IRQ 66, NUMA node 0, IOMMU group 16
Memory at fe080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intelWhich seems to confirm that 1) the GPU is found correctly and B) the vfio-pci driver is in use.
At this point, I think that hardware/host wise, I am done? And that everything is working? Is there anything I could do to further check if the issue is related to my hardware or proxmox config?
I have set up a Windows 10 Pro (N) 21H1 VM with the following config:
Code:
bios: ovmf
boot: order=ide0;ide2;net0
cores: 2
cpu: host
efidisk0: SSD1_1TB:101/vm-101-disk-1.qcow2,size=128K
hostpci0: 0000:01:00,pcie=1,x-vga=1
ide0: SSD1_1TB:101/vm-101-disk-0.qcow2,size=32G
ide2: none,media=cdrom
machine: pc-q35-6.0
memory: 2048
name: GPUtest
net0: e1000=7A:F0:1D:85:D4:E0,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=304d1841-932d-41ad-b7fa-9f6517ecb5ac
sockets: 1
vga: virtio
vmgenid: 6fb8131a-7df9-425b-af5d-77f1d95f8760The setup went smoothly, I installed nvidia driver 450.x and got a bluescreen related to
nvlddmkm.sys (System Service Exception). I have read that this might be an issue with MSI support here but this is already enabled on my VM.Booting with "primary GPU" unchecked works, but then I get a Code 43 in the device Manager for the Quadro P400.
Strangely, in the device manager I also see a PCI Device (bus 6, device 3) with a Code 28, and I have no Idea what that could be.
I would greatly appreciate any suggestions you might have
