Yes another pci pass through trouble

[francoisH]

Hello everyone, I open a thread for a pci pass through.

Symptom, everything work great exact, in game (with parsec or moonlight) or GPU benchmark (benchmark heaven), i have no more than 15 fps, I try to reduce or increase graphical in game but not change.



Hardware:

GPU: GTX 1070 TI funder edition + hdmi dummy

CPU: Intel Xeon E5-2697 v3

Motherboard: x99 Chinese motherboard

(https://fr.aliexpress.com/item/1005...st_main.9.21ef5e5be3qxFg&gatewayAdapt=glo2fra)

RAM: 32 Gbits 2600 ECC DDR4.

OS: Proxmox 7.1-7



VM configuration (windows 10 22h2)

root@pve:~# less /etc/pve/nodes/pve/qemu-server/104.conf
agent: 1
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'
bios: ovmf
boot: order=sata0;ide2
cores: 16
cpu: host,hidden=1,flags=+pcid,hv-vendor-id=proxmox
efidisk0: NVME:vm-104-disk-1,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:03:00,pcie=1
ide2: none,media=cdrom
machine: pc-q35-6.1
memory: 16384
meta: creation-qemu=6.1.0,ctime=1663166037
name: win-gpu
net0: e1000=66:5A:84:04:20:95,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
parent: snap
sata0: NVME:vm-104-disk-0,size=60G
scsi1: NVME:vm-104-disk-2,size=500G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=38e149a7-a7db-4aab-b560-b4204643cdef
sockets: 1
startup: order=10
vga: none
vmgenid: 13096c8a-eff1-4b99-b8aa-f7e8aabb09b4

root@pve:~# dmesg | grep -e DMAR -e IOMMU -e 3:00 [ 0.009819] ACPI: DMAR 0x000000007AB7CD28 0000E4 (v01 ALASKA A M I 00000001 INTL 20091013) [ 0.009838] ACPI: Reserving DMAR table memory at [mem 0x7ab7cd28-0x7ab7ce0b] [ 0.069855] DMAR: IOMMU enabled [ 0.184704] DMAR: Host address width 46 [ 0.184705] DMAR: DRHD base: 0x000000fbffd000 flags: 0x0 [ 0.184710] DMAR: dmar0: reg_base_addr fbffd000 ver 1:0 cap d2008c10ef0466 ecap f0205b [ 0.184713] DMAR: DRHD base: 0x000000fbffc000 flags: 0x1 [ 0.184716] DMAR: dmar1: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020df [ 0.184718] DMAR: RMRR base: 0x0000007b404000 end: 0x0000007b414fff [ 0.184719] DMAR: ATSR flags: 0x0 [ 0.184720] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x0 [ 0.184722] DMAR-IR: IOAPIC id 1 under DRHD base 0xfbffc000 IOMMU 1 [ 0.184724] DMAR-IR: IOAPIC id 2 under DRHD base 0xfbffc000 IOMMU 1 [ 0.184725] DMAR-IR: HPET id 0 under DRHD base 0xfbffc000 [ 0.184726] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 0.185195] DMAR-IR: Enabled IRQ remapping in x2apic mode [ 2.495157] pci 0000:03:00.0: [10de:1b82] type 00 class 0x030000 [ 2.495169] pci 0000:03:00.0: reg 0x10: [mem 0xfa000000-0xfaffffff] [ 2.495179] pci 0000:03:00.0: reg 0x14: [mem 0xe0000000-0xefffffff 64bit pref] [ 2.495189] pci 0000:03:00.0: reg 0x1c: [mem 0xf0000000-0xf1ffffff 64bit pref] [ 2.495196] pci 0000:03:00.0: reg 0x24: [io 0xe000-0xe07f] [ 2.495202] pci 0000:03:00.0: reg 0x30: [mem 0xfb000000-0xfb07ffff pref] [ 2.495208] pci 0000:03:00.0: enabling Extended Tags [ 2.495229] pci 0000:03:00.0: BAR 1: assigned to efifb [ 2.495305] pci 0000:03:00.0: 32.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x16 link at 0000:00:02.0 (capable of 126.016 Gb/s with 8.0 GT/s PCIe x16 link) [ 2.495394] pci 0000:03:00.1: [10de:10f0] type 00 class 0x040300 [ 2.495406] pci 0000:03:00.1: reg 0x10: [mem 0xfb080000-0xfb083fff] [ 2.495442] pci 0000:03:00.1: enabling Extended Tags [ 2.498755] pci 0000:03:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none [ 2.498755] pci 0000:03:00.0: vgaarb: bridge control possible [ 2.498755] pci 0000:03:00.0: vgaarb: setting as boot device [ 2.576901] pci 0000:03:00.0: CLS mismatch (64 != 32), using 64 bytes [ 2.576975] pci 0000:03:00.1: D0 power state depends on 0000:03:00.0 [ 2.577316] DMAR: [Firmware Bug]: RMRR entry for device 06:00.0 is broken - applying workaround [ 2.577320] DMAR: No SATC found [ 2.577322] DMAR: IOMMU feature sc_support inconsistent [ 2.577324] DMAR: IOMMU feature dev_iotlb_support inconsistent [ 2.577325] DMAR: dmar0: Using Queued invalidation [ 2.577329] DMAR: dmar1: Using Queued invalidation [ 2.580277] pci 0000:03:00.0: Adding to iommu group 31 [ 2.580313] pci 0000:03:00.1: Adding to iommu group 31 [ 2.580446] DMAR: Intel(R) Virtualization Technology for Directed I/O [ 6.839330] vfio-pci 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none

PCI device :
Device: 0000:03:00.0
All functions: YES
Primary GPU: NO
ROM-Bar: YES
PCI-Express: YES

root@pve:~# lspci -n -s 03:00 03:00.0 0300: 10de:1b82 (rev a1) 03:00.1 0403: 10de:10f0 (rev a1)

root@pve:~# cat /etc/modprobe.d/vfio.conf options vfio-pci ids=10de:1b82,10de:10f0 disable_vga=1


root@pve:~# cat /etc/default/grub # remove some # GRUB_DEFAULT=0 GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian` GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt quiet splash pcie_aspm=off video=efifb:off" GRUB_CMDLINE_LINUX=""

root@pve:~# dmesg | grep 'remapping' [ 0.184726] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 0.185195] DMAR-IR: Enabled IRQ remapping in x2apic mode


root@pve:~# cat /etc/kernel/cmdline #quiet intel_iommu=on #pci=noaer #pcie_aspm=off root=ZFS=rpool/ROOT/pve-1 boot=zfs i915.enable_gvt=1 kvm.ignore_msrs=1 intel_iommu=on iommu=pt


root@pve:~# cat /etc/modprobe.d/blacklist.conf blacklist radeon blacklist nouveau blacklist nvidia




[GPU]

For the graphic card for working i need to flash it with the last firwmare with rom-parser and nvflash:

467 ./nvflash --save orinal.rom
468 ./nvflash ../rom-parser/NVIDIA.GTX1070Ti.8192.170923.rom


If needed, i could provide any log.

I think about that but i'm not sure at all: my xeon processor doesn't have any transcoding capability, what do you think about ?
Thanks in adavanced for your help.

Regards,
Francois

 

[noel.]

While benchmarking in the VM, have you tried monitoring inside the Windows VM (for example with HWinfo [1]) to see where the bootlenecks are? E.G. Temperatures, CPU utilization, ram, GPU is not fully utilized...

One bottleneck might be "Graphic card slot : 3*PCI-E X16(PCIE X16_1, PCIE X16_2 support X16 GEN3 speed, PCIE X16_3 only supports x4 GEN3 speed) / 2*PCI-E 1X" (taken from the link you sent). The PCI-E lane only supports x4 Gen3 speed (=3.938 GB/s) instead of full x16 speeds (15.754 GB/s) but I doubt that that would lead to such a significant bottleneck...

---

[1]: https://www.hwinfo.com/

 

[francoisH]

Hello Noel, thanks you're answer,

I check some information provide by hwinfo.
1. general information about CG
2.Sensors without any load
3.Sensorrs after 5min load
4.Sensors after 30 min load

As you can see in top left we can see the average FPS and its very low for fury max.

Thanks

 

[noel.]

Hmmm, looks to me like the card is being fully detected and utilized. But why it's running at 97% with only 20fps, I am not quite sure. Some 10-20% perfomance loss in a vm seems plausible but this is too much. Perhaps it is a hardware issue? Can you stick the card in a different machine and benchmark it there? Or benchmark it in the same machine but not in a VM? You could also set up a windows 10 VM and see if the benchmarks are as bad there?

 

[Neobin]

Looks to me, that the GPU is stuck at idle(?) clocks of 139 MHz respectively 658 MHz.
Under load, a 1070 Ti should clock with at least 1600 MHz: [1].
Why this is and how to troubleshoot/fix it, I do not know.

[1] https://www.techpowerup.com/gpu-specs/geforce-gtx-1070-ti.c3010

 

[leesteken]

Have you tried it with a physical display attached (instead of the HDMI-plug and streaming)? Maybe there is a lot of overhead in encoding/copying the rendered screen?

 

[francoisH]

Hello @noel. , thanks i will try the easy one just changing the cg in another pc.
Hello @Neobin, yeah i compare with another pc with gtx960 and it's more, i will compare when a will change the pc
Hello @leesteken , yes i already test that and test without dumy and work "normally".

I will not be able to perform some test into near futur.
I will update as soon as possible.

regards,
Francois

 

[francoisH]

Hello @noel. , @Neobin ,@leesteken .

So i finally able to perform some new test with normal windows and .... still stuck at 139 Mhz and smal FPS i try different drivers and re flash the card but still the same .... i try different things found in google but i think the conclusion is the card is fuc*** ... lost 100 bucks..
thanks every one for your helps.

 

[francoisH]

thread could be closed

 

[leesteken]

thread could be closed

Please edit the first post of this thread and set it to Solved. Sorry to hear your card was broken.