VM's on 18 core datacenter stuck at 2-3% due to passthrough?

[futiless]

I have searched for my answer high and low. Essentially I am booting a (SINGLE) SSD on ACHI from UEFI to load Proxmox. From Proxmox I have made a zpool for nvme0n1 and nvme1n1. Then attached a ZFS via the UEFI interface to the newly created zpool. Next I passed through my GPU as specified in the PCI Passthrough guide from 2017. (my hardware is fully supported and slots are set correctly for proper grouping.

This actually passes through everything except the CPU is stuck at 2.85% ?!!?!?!?!?! WHY !? It's taking 2 hours (normally 2-3 minutes) to get through the Windows 10 Setup first screen. I have tried with and with out VirtIO, messing with grub, and so on ... I really am not sure where this happened and I have rebuilt my system from fresh install 3 times now. I am sure my methods are exact to the guide but I am getting weird issues.

Please help; and YES I WILL SUPPORT this community and its authors. BUT FIRST allow me to see it function. ATM it's far to slow to pay for ;(

 

[futiless]

After reflecting on the matter I have isolated this to Proxmox itself and passthrough. And that is really as I want to be a long term member of this community.

The CPU will work if I use KVM + OVMF + ZFS and NO passthrough. If I then use the same VM, and attempt passthrough I am stuck with a slow motion / (unstable and even crashes itself) or seeming 2.85 throttled CPU...

Is this VirtIO ? Is this network related? Having followed the guide I would love to know it is a hardware limitation of otherwise. As I would love to invest my life quite literally toward the expansion of this medium.

For it is a marvel to me to see such a light weight and quite literally a discovery THAT COULD change my LIFE forever.

Mark my words I want to be a Platinum member but it is ESSENTIAL to my testing to at least SEE the proper passthrough of my graphics. Otherwise sadly Proxmox is not what it appears to be short of this QUITE literally MAKE or BREAK function on my system.

So without further a due; my most private parts (hehe of my server that is : )

............................................................

Used : Zpool create NVMeStripe nvme0n1 nvme1n1

Then made standard ZFS item in Proxmox called NVMeStripeID




------------------------------------------------------------------------------

/etc/default/grub

---------------------



# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
GRUB_CMDLINE_LINUX=""

# Disable os-prober, it might add menu entries for each guest
GRUB_DISABLE_OS_PROBER=true

......

-----------------------------------------------------------------------------

/etc/modules

----------------------



vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

-----------------------------------------------------------------------------

/etc/modprobe.d/vfio.conf

----------------------



options vfio-pci ids=10de:1b80,10de:10f0 disable_vga=1


-----------------------------------------------------------------------------

/etc/modprobe.d/blacklist.conf

---------------------

blacklist nouveau
blacklist nvidia
blacklist radeon


-----------------------------------------------------------------------------

/etc/pve/nodes/SS1/qemu-server/100.conf

---------------------

agent: 1
balloon: 0
bios: ovmf
bootdisk: scsi0
scsi0: .....
cores: 36
efidisk0: NVMeStipeID:vm-100-disk-2,size=128K
hostpci0: 0b:00,pcie=1,x-vga=on
ide0: NVMeStipeID:vm-100-disk-1,size=256G
ide2: local:iso/Windows.iso,media=cdrom,size=6362368K
kvm: 0
machine: q35
memory: 65535
name: 4KHT10SS
net0: e1000=3E:E2:04:92:51:6A,bridge=vmbr0
numa: 1
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=e57e49b0-6a20-4573-b584-9b1171b439a2
sockets: 1
usb0: host=3-11.4
usb1: host=3-14.4



---------------------------------------------------------------------------------------------------------------


For what it's worth; here are a few points I have noticed along the way as my novice spidey senses are tingling.

1) Adding the line of KVM:0 from nothing was freezing my first start attempts after passthrough.
2) I always need to remind myself that UEFI/OVMF is in fact WORKING on ZFS, with initial KVM session. It was after modprobe.d/vfio.conf + /etc/default/grub/ changes that the CPU seemed to just fall off.
3)bios/uefi defaults + (same settings as ESXi) with Proxmox before (windows boot loader) in the boot sequence.
4!) Initially after clean Proxmox install the grub command prompt and everything after USB went into a initial boot (my first impression was a glitch or otherwise) !garbled graphics artifacts during the first boot of Proxmox. GRUB interface was entirely illegible unitil I used FTP (winSCP /etc/default/grub/) and made changes to

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
GRUB_CMDLINE_LINUX=""

HMMMmm or come to think of it it could be /etc/modprobe.d/blacklist.conf, OR /etc/modprobe.d/vfio.conf changes that caused it. Heck maybe it's /etc/modules changes. I simply know it is NOT 100.conf changes or at least I am convinced this way because I have witnessed the function on KVM - passthrough with whatever settings.

Please help me prove to myself that Proxmox is what I have always been missing.

Then I am ready to jump on board, Sincerely!

 

[brwainer]

1/36 = 0.0277 or 2.77% - so it looks like your VM is only able to access a single CPU core, or whatever task it is working on is only capable of using a single CPU thread. Can you see greater CPU usage in this VM if you remove the passthrough-ed GPU and use the standard remote console? How about changing your number of sockets to something higher, e.g. 2 or 4, unless you actually have an 18-core CPU with hyper threading? Or try limiting the VM to just 8 cores? Also try setting the CPU type to "host"?

 

[futiless]

Thank you so much for trying to help I really appreciate it!

This is an E5-2699 v3 @ 18 cores ; 36 with hyperthread. That is why I set it so. In ESXi I set it the same and it worked.
When considering the point I made earlier :

"
2) I always need to remind myself that UEFI/OVMF is in fact WORKING on ZFS, with initial KVM session. It was after modprobe.d/vfio.conf + /etc/default/grub/ changes that the CPU seemed to just fall off."

I can't be certain but it really seemed to happen when I ran update grub and restarted.

---------------

An aside not; when using zfs; OVMF running a passthrough should I loading drivers before launching the Win 10 ISO media, in on IDE2 ? Perhaps providing essential drivers from the VMware-VIM-all-6.5.0-4944578 media?

Thanks for you help !!!!!

 

[futiless]

1/36 = 0.0277 or 2.77% - so it looks like your VM is only able to access a single CPU core, or whatever task it is working on is only capable of using a single CPU thread. Can you see greater CPU usage in this VM if you remove the passthrough-ed GPU and use the standard remote console? How about changing your number of sockets to something higher, e.g. 2 or 4, unless you actually have an 18-core CPU with hyper threading? Or try limiting the VM to just 8 cores? Also try setting the CPU type to "host"?

GOOD point though on the 1 core, that must be right, and sadly your suggestion regarding the use of the host cpu flag although seeming hopeful prevented the VM from starting with an error to enable KVM and as I stated earlier it crashed when I tried passing through graphics AND using KVM. (no CPU use, even the odd time stuck from being able to STOP the machine ) ?!? that part makes no sense but its true... just another FYI

and again Thank You to infinity

 

[brwainer]

Oh! I had missed the fact that you had disabled kvm. That makes perfect sense then - if kvm is disabled, you are not using virtualization, but emulation. That is why you are getting stuck into a single CPU thread. You'll have to figure out how to get GPU passthrough working with KVM enabled. I have it working well on my systems, but they are different platforms from yours (one is a dual Opteron 6220, the other is an E3-1230v3) so my experience isn't going to be 100% useful. I can't go in depth roght now on helping you troubleshoot this, but I can later today.

 

[futiless]

Oh! I had missed the fact that you had disabled kvm. That makes perfect sense then - if kvm is disabled, you are not using virtualization, but emulation. That is why you are getting stuck into a single CPU thread. You'll have to figure out how to get GPU passthrough working with KVM enabled. I have it working well on my systems, but they are different platforms from yours (one is a dual Opteron 6220, the other is an E3-1230v3) so my experience isn't going to be 100% useful. I can't go in depth roght now on helping you troubleshoot this, but I can later today.

Wow that would be great atm the moment I have a very expensive paperweight,..... I guess in a way your my only hope; or if you are exhausted with this you can just send me whatever you got to read over. Presently I can only seem to find a flag to implement here and then one there. No master list or anything. Is it Apic or MMO or what... im not sure ;(

 

[futiless]

For me it's just endless stabbing in the dark at this point. I wish someone could simply point me in the right direction for how to Troubleshoot the sleeping monster.

 

[TheKingOfKats]

For me it's just endless stabbing in the dark at this point. I wish someone could simply point me in the right direction for how to Troubleshoot the sleeping monster.

Try removing the

kvm : 0

line and add

args: -cpu host,kvm=off
vga: none

Also you can't add every core to your virtual machine, you need to leave some dedicated to the host. Try using 30 cores instead. Likewise with memory, you need to leave some for the host; try 32GB.

While the VM is running, could you post the results of

ps -aux|grep <vmid>

(run from the host)

Also any particular reason you are using IDE for your NVME drives?

 

[futiless]

Try removing the line and add



Also you can't add every core to your virtual machine, you need to leave some dedicated to the host. Try using 30 cores instead. Likewise with memory, you need to leave some for the host; try 32GB.

Also any particular reason you are using IDE for your NVME drives?

As requested here is your requested output :

ps -aux|grep 100

root 100 0.0 0.0 0 0 ? S 02:35 0:00 [ksoftirqd/15]
root 1000 0.0 0.0 0 0 ? S 02:35 0:00 [arc_prune]
root 1001 0.0 0.0 0 0 ? S 02:35 0:00 [arc_prune]
root 1002 0.0 0.0 0 0 ? S 02:35 0:00 [arc_prune]
root 1003 0.0 0.0 0 0 ? S 02:35 0:00 [arc_prune]
root 1004 0.0 0.0 0 0 ? S 02:35 0:00 [arc_prune]
root 1005 0.0 0.0 0 0 ? S 02:35 0:00 [arc_prune]
root 1006 0.0 0.0 0 0 ? S 02:35 0:00 [arc_prune]
root 1007 0.0 0.0 0 0 ? S 02:35 0:00 [arc_prune]
root 1008 0.0 0.0 0 0 ? S 02:35 0:00 [arc_reclaim]
root 1009 0.0 0.0 0 0 ? S 02:35 0:00 [arc_user_evict s]
root 2100 0.0 0.0 0 0 ? S< 02:35 0:00 [z_wr_iss_h]
root 2980 0.0 0.0 533468 100072 ? S 02:35 0:00 pvedaemon worke r
root 2981 0.0 0.0 533480 100068 ? S 02:35 0:00 pvedaemon worke r
root 2982 0.0 0.0 539928 100780 ? S 02:35 0:00 pvedaemon worke r
root 4138 0.0 0.0 12788 1000 pts/0 S+ 02:47 0:00 grep 100

I am using the IDE setting as the real drive is a ZFS stripe 0 of the Samsung 950's. I guess your right though I just don't understand how it would affect the result. It's also really important to consider that it functions if I remove pass-through and reboot the Datacenter. It DID work with ESXi and I this is very familiar to the same issue resolved.

Also I have 2 of the SAME graphics card attached with a sli HB bridge,... It shouldn't matter I realize we aren't using this, but in ESXi it would launch to the console and use both for separate pass through vm's.

But the system has 128 GB and 18 xeon cores on 2699 v3; I have an efi unlock on it so it can go very fast but. So just to give your point a shot I attempted both 10 cores and left the 64 gb alone cause the server gets 64 then.

 

[futiless]

WAIT ;

no after a reboot and going over my settings carefully I noticed that the freeze CPU flag was enabled (keeps reenableing on its own too ?.. Ohh well after doing this I have PASSTHROUGH or ... is it, I mean this is my output from now :

root 100 0.0 0.0 0 0 ? S 03:31 0:00 [ksoftirqd/15]
root 1000 0.0 0.0 0 0 ? S 03:31 0:00 [arc_prune]
root 1001 0.0 0.0 0 0 ? S 03:31 0:00 [arc_prune]
root 1002 0.0 0.0 0 0 ? S 03:31 0:00 [arc_prune]
root 1003 0.0 0.0 0 0 ? S 03:31 0:00 [arc_prune]
root 1004 0.0 0.0 0 0 ? S 03:31 0:00 [arc_reclaim]
root 1005 0.0 0.0 0 0 ? S 03:31 0:00 [arc_user_evicts]
root 1006 0.0 0.0 0 0 ? S 03:31 0:00 [l2arc_feed]
root 1007 0.0 0.0 0 0 ? S 03:31 0:00 [z_unmount]
root 2100 0.0 0.0 0 0 ? S< 03:31 0:00 [z_wr_int_0]
root 2979 0.0 0.0 533444 100204 ? S 03:31 0:00 pvedaemon worker
root 2980 0.0 0.0 533580 100744 ? S 03:31 0:00 pvedaemon worker
root 2981 0.0 0.0 533496 100528 ? S 03:31 0:00 pvedaemon worker
root 3927 152 50.8 68034912 67146332 ? SLl 03:41 9:59 /usr/bin/kvm -id 100 -chardev socket,id=qmp,path=/v ar/run/qemu-server/100.qmp,server,nowait -mon chardev=qmp,mode=control -pidfile /var/run/qemu-server/100.pid -daemon ize -smbios type=1,uuid=a6cf4de7-db01-4f68-bceb-565cf872fa0e -drive if=pflash,unit=0,format=raw,readonly,file=/usr/s hare/kvm/OVMF_CODE-pure-efi.fd -drive if=pflash,unit=1,id=drive-efidisk0,format=raw,file=/dev/zvol/NVMeStripe/vm-100 -disk-2 -name 4KHT10SS -smp 10,sockets=1,cores=10,maxcpus=10 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000 ,splash=/usr/share/qemu-server/bootsplash.jpg -vga none -nographic -no-hpet -cpu host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_ vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,kvm=off -m 65535 -o bject memory-backend-ram,id=ram-node0,size=65535M -numa node,nodeid=0,cpus=0-9,memdev=ram-node0 -k en-us -cpu host,k vm=off -readconfig /usr/share/qemu-server/pve-q35.cfg -device usb-tablet,id=tablet,bus=ehci.0,port=1 -device vfio-pc i,host=0b:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on -device vfio-pci,host=0b:00.1,id=hostp ci0.1,bus=ich9-pcie-port-1,addr=0x0.1 -device usb-host,hostbus=3,hostport=11.4,id=usb0 -device usb-host,hostbus=3,ho stport=14.4,id=usb1 -iscsi initiator-name=iqn.1993-08.org.debian:01:65b0cb64ee12 -drive file=/dev/zvol/NVMeStripe/vm -100-disk-1,if=none,id=drive-ide0,discard=on,format=raw,cache=none,aio=native,detect-zeroes=unmap -device ide-hd,bus =ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100 -drive file=/var/lib/vz/template/iso/Windows.iso,if=none,id=dri ve-ide2,media=cdrom,aio=threads -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -rtc driftfix =slew,base=localtime -machine type=q35 -global kvm-pit.lost_tick_policy=discard
root 4587 0.0 0.0 12788 980 pts/0 S+ 03:48 0:00 grep 100

The only passthrough is back to where grub fell off; the lack of a frozen VM and well wow, we must be close ?

 

[futiless]

It's WORKING : THANK YOU SOOO VERY MUCH ; can I buy you a beer


Why can't I ever use more than 14 of 36 virtual processors like I could in ESX ? is this because of host mode passing them over entirely ? I would like to better understand this function

 

[TheKingOfKats]

It's WORKING : THANK YOU SOOO VERY MUCH ; can I buy you a beer

I'm not really following but if it is working then fantastic!

When you enter a CPU / RAM value in the VM configuration, you are entering how much of that resource you want to allocate exclusively for the VM. So even if your host has 36 threads, you can't give the VM all 36 because the host still needs something to run on.

And just to explain what I wanted out of that command, the first time you posted it was missing the output for the VM process (/usr/bin/kvm -id 100), meaning that the VM never actually started, but in the second it is present and shows that Proxmox is generating the parameters -cpu host,kvm=off, which is what you need for passthrough.

 

[futiless]

I'm not really following but if it is working then fantastic!

When you enter a CPU / RAM value in the VM configuration, you are entering how much of that resource you want to allocate exclusively for the VM. So even if your host has 36 threads, you can't give the VM all 36 because the host still needs something to run on.

And just to explain what I wanted out of that command, the first time you posted it was missing the output for the VM process (/usr/bin/kvm -id 100), meaning that the VM never actually started, but in the second it is present and shows that Proxmox is generating the parameters -cpu host,kvm=off, which is what you need for passthrough.

I see how the CPU is exclusive. Though would this be the optimal setting for me? In Esxi from what function I thought I was performing I was creating a machine with pass-through and 36 cores. How is this different? In this scenario I always have unused cores to consider of staticly balancing the cores between the clients will never be optimal ... Hmm maybe this is me just not understand Proxmox's function.

 

[brwainer]

I'm sorry that I disappeared for a few days and couldn't offer the help I promised at the time. I'm glad that someone has been able to point you in the right direction.

Is 14 definitely the limit on cores you can assign? Have you tried 15 or 16? Also, have you tried enabling NUMA passthrough? Normally NUMA passthrough helps with performance but I wonder if it might help you boot as well. NUMA allows the operating system to know what memory addresses are more directly connected to what cores or CPUs - the modern high-core-count Xeons use a dual ring bus topology, so they are like two separate processors side by side, and each side has 2 of the 4 memory channels.

 

[futiless]

in

I'm sorry that I disappeared for a few days and couldn't offer the help I promised at the time. I'm glad that someone has been able to point you in the right direction.

Is 14 definitely the limit on cores you can assign? Have you tried 15 or 16? Also, have you tried enabling NUMA passthrough? Normally NUMA passthrough helps with performance but I wonder if it might help you boot as well. NUMA allows the operating system to know what memory addresses are more directly connected to what cores or CPUs - the modern high-core-count Xeons use a dual ring bus topology, so they are like two separate processors side by side, and each side has 2 of the 4 memory channels.

Very interesting indeed; heres a list of things I can't make work for me ;

1) getting a message to apio into x2 mode ; I think the mmio is a mess i'm not sure.

use the acs=downstream option if your IOMMU groups don't split correctly (see Wiki)?

2) ALWAYS got a mess during grub handoff to (framebuffer?) graphics become useless during proxmox first boot. (from grub)

use the new vfio-pci.ids boot option instead of the older pci-stub.ids (i.e. "vfio-pci.ids=8086:0151,10de:1401,10de:0fba" in grub cmdline in my case)?

3) I have a dreaded setup of UEFI 16x channels of beauty; with NO integrated graphics. Some OP's posted that ProxMox CANNOT Primary passthrough or Headless mode ; I don't know if this is still the case ? ..

same as last..

4)ZFS is taking 64 gb of memory and seems fairly slow how do I tune it properly ? Where to find the RIGHT guide for ProxMox.

5) I think this is a stupid question for most but.... Can I plugin my own frontend for proxmox (like are there easier ones for day to day use). Again Stupid question but if install linux can I utilize a better hypervisor like with others?

6) Important question : I am getting a beep every 10 seconds but I can't read grub due to item 2. It is saying something I think it is a kernel issue that someone was reporting about back in April. ? It seems bad how can I fix it PROPERLY..

7) Every SINGLE time the server reboots my UEFI BIOS, installs an additional boot option. So unless I clear my CMOS my computer has 40 boot entries ?!? WTF Lol ?

anways total noob and thank you very much to anyone who helps me!!

 

[futiless]

If I reboot with
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pcie_acs_override=downstream vfio-pci.ids=10de:1b80,10de:10f0"

I am receiving grub display after a brief 2-3 second period (maybe less) after grub boot up menu and it lets me see what happening now.

But I am getting the following kernel problems too ?!?

I get :
kernel:[ 550.855437] page:fffff0d9fef82bc0 count:1 mapcount:-256 mapping:ffff927e844e8281 index:0x55fbc982c
errors all the time ?

 

[futiless]

lspci:
00:00.0 Host bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMI2 (rev 02)
00:01.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02)
00:01.1 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02)
00:02.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02)
00:03.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02)
00:03.2 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02)
00:05.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management (rev 02)
00:05.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug (rev 02)
00:05.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors (rev 02)
00:05.4 PIC: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC (rev 02)
00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR (rev 05)
00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05)
00:16.0 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller #1 (rev 05)
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (2) I218-V (rev 05)
00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05)
00:1b.0 Audio device: Intel Corporation C610/X99 series chipset HD Audio Controller (rev 05)
00:1c.0 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 (rev d5)
00:1c.2 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #3 (rev d5)
00:1c.4 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #5 (rev d5)
00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation C610/X99 series chipset LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)
00:1f.3 SMBus: Intel Corporation C610/X99 series chipset SMBus Controller (rev 05)
01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 (rev 01)
02:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 (rev 01)
03:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
04:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
04:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
04:11.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
07:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1080] (rev a1)
07:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
08:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
09:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
0a:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
0a:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
0c:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1080] (rev a1)
0c:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
0e:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
0f:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller
ff:0b.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
ff:0b.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
ff:0b.2 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
ff:0c.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0d.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0d.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0d.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0d.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0d.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0d.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0d.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0d.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0e.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0e.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0f.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
ff:0f.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
ff:0f.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
ff:0f.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
ff:0f.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
ff:0f.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
ff:0f.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
ff:10.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02)
ff:10.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02)
ff:10.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
ff:10.6 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
ff:10.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
ff:12.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 (rev 02)
ff:12.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 (rev 02)
ff:12.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 (rev 02)
ff:12.5 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 (rev 02)
ff:13.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02)
ff:13.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02)
ff:13.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)
ff:13.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)
ff:13.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast (rev 02)
ff:13.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02)
ff:14.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control (rev 02)
ff:14.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control (rev 02)
ff:14.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers (rev 02)
ff:14.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers (rev 02)
ff:14.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
ff:14.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
ff:14.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
ff:14.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
ff:16.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers (rev 02)
ff:16.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers (rev 02)
ff:16.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder (rev 02)
ff:16.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder (rev 02)
ff:16.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast (rev 02)
ff:16.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02)
ff:17.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal Control (rev 02)
ff:17.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 1 Thermal Control (rev 02)
ff:17.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 ERROR Registers (rev 02)
ff:17.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 1 ERROR Registers (rev 02)
ff:17.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
ff:17.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
ff:17.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
ff:17.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
ff:1e.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
ff:1e.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
ff:1e.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
ff:1e.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
ff:1e.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 02)
ff:1f.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU (rev 02)
ff:1f.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU (rev 02)

--------------------------------------

I also found out how to list lspci, and noticed that my addresses have all moved around ? Can anyone explain how this happened ? I attempted to add an old gpu to slot one forgetting that it had not UEFI, which triggered a legacy boot and moved the devices all around ?! Is this normal, I do understand IRQ, DMA and so on. But regarding weather or not this is better than what I have had for months or not I am not sure. One thing is it messed up my nic's and the system would boot to disabled network ?!? I am really confused by this. I just dumped and started over. Hopefully I will learn quickly so I can actually witness/showoff proxmox to people who have for 4 months seen me struggling with this this at the command prompt .. grrrrr..

 

[futiless]

PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0008(Receiver ID)
[ 8593.488831] pcieport 0000:00:01.0: device [8086:2f02] error status/mask=00000080/00002000
[ 8593.489830] pcieport 0000:00:01.0: [ 7] Bad DLLP

woes now ; locking freezing and the like...