[TUTORIAL] AMD S7150 MxGPU with Proxmox VE 5.x

dcsapak

dcsapak

Proxmox Staff Member
Staff member
Feb 1, 2016
10,192
1,484
273
36
Vienna
Hi,

we have had the chance to test the AMD S7150 Graphics card, which enables SR-IOV for GPUs, meaning that a single physical card can supply multiple VMs with a 'real' GPU (incl. OpenCL and 3D hardware acceleration).

I wrote a wiki article about our results here: https://pve.proxmox.com/wiki/MxGPU_with_AMD_S7150_under_Proxmox_VE_5.x

TL;DR: it seems to work ok with the right Hardware and Windows guests, but fails completely with 'Low-End' Hardware or when you want to use Linux Guests.

Hopefully this technology will advance in the future.

If you have any questions or suggestions what we could test with the card, just ask here.
 
  • Like
Reactions: Tmanok, fireon, Max P and 1 other person
Going through the steps to set this up now! Really hoping I can get it worked relatively well with an MSI Z270 mobo and windows 10 guests.
 
emagray said:
Going through the steps to set this up now! Really hoping I can get it worked relatively well with an MSI Z270 mobo and windows 10 guests.
Click to expand...
as you can see in the wiki, it did not work here with consumer boards, did you have more success?
 
I read the article in the link, and I got few questions:
  1. In the docs on the Github it mentions using UltraVNC (on Windows) or VNC on Linux. Did you use this method or did you connect directly through RDP (on Windows)?
  2. Card doesn't work on low end - in many cases if you use GPU, the 2nd GPU will only get X8 PCIe on X16 physical connection. Have you checked it? this card really needs all the 16 lanes.
  3. Care to share (if you have) the gim configuration?
  4. Which distribution do you use for compiling the kernel?
  5. The driver, as well as the VBIOS have been updated few days ago to take care of mixed geust OS's. Want to retry? :)
 
Hetz Ben Hamo said:
  • In the docs on the Github it mentions using UltraVNC (on Windows) or VNC on Linux. Did you use this method or did you connect directly through RDP (on Windows)?
Click to expand...
as mentioned in our wiki article, i used Remote Desktop (aka RDP) on windows

Hetz Ben Hamo said:
  • Card doesn't work on low end - in many cases if you use GPU, the 2nd GPU will only get X8 PCIe on X16 physical connection. Have you checked it? this card really needs all the 16 lanes.
Click to expand...
the only box i got it to work was on an epyc server where it had a physical x16 connection

Hetz Ben Hamo said:
  • Care to share (if you have) the gim configuration?
Click to expand...

Code: 
fb_option=0
sched_option=0
vf_num=4
pf_fb=0
vf_fb=0
sched_interval=0
sched_interval_us=0

i limited it to 4 vf to not clutter my pci devices list, but it did also work with the default configuration


Hetz Ben Hamo said:
  • Which distribution do you use for compiling the kernel?
Click to expand...
i did not compile any kernel, only the module, which i did on a current proxmox installation (the host i installed it on)

Hetz Ben Hamo said:
  1. The driver, as well as the VBIOS have been updated few days ago to take care of mixed geust OS's. Want to retry? :)
Click to expand...
i already updated the gim kernel module (which takes care of the vbios i think, at least i did not found any other method)
i will update the drivers and test it again with linux guests in the near future, no promises as to when though

just fyi: i have the card and some vms running most of the time on my machine, so if any qemu/kernel/gim update breaks it completely, i will notice ;)
 
dcsapak said:
as you can see in the wiki, it did not work here with consumer boards, did you have more success?
Click to expand...
Unfortunately no. My board is not SR-IOV capable and I'm not ready to spend the money on building a new server =\

I got a great deal on this card and would like to make use of it for virtualizing a number of engineering workstations.
 
Sorry for digging old post.
Have you ever tested on older generation like E5 V1/V2?
I really want to use S7150 on either Dell T5610 or Z620 but cant find any details
 
@dcsapak

Have you tried a recent build? I'm using the 6.0 beta with kernel 5.0.15-1-pve on an AMD EPYC 7551P / Supermicro H11SSL-NC and it's throwing errors for me in DKMS:

Code: 
DKMS make.log for amd_gim-3.0 for kernel 5.0.15-1-pve (x86_64)
Thu 18 Jul 2019 02:06:51 PM MST
make: Entering directory '/usr/src/linux-headers-5.0.15-1-pve'
  CC [M]  /var/lib/dkms/amd_gim/3.0/build/gim_drv.o
  CC [M]  /var/lib/dkms/amd_gim/3.0/build/gim_interface.o
  CC [M]  /var/lib/dkms/amd_gim/3.0/build/gim_adapter.o
  CC [M]  /var/lib/dkms/amd_gim/3.0/build/gim_pci.o
  CC [M]  /var/lib/dkms/amd_gim/3.0/build/gim_unwrapper.o
  CC [M]  /var/lib/dkms/amd_gim/3.0/build/gim_gpuiov.o
  CC [M]  /var/lib/dkms/amd_gim/3.0/build/gim_config.o
  CC [M]  /var/lib/dkms/amd_gim/3.0/build/gim_timer.o
  CC [M]  /var/lib/dkms/amd_gim/3.0/build/gim_fb.o
  CC [M]  /var/lib/dkms/amd_gim/3.0/build/gim_debug.o
  CC [M]  /var/lib/dkms/amd_gim/3.0/build/gim_flr.o
  CC [M]  /var/lib/dkms/amd_gim/3.0/build/gim_atom.o
  CC [M]  /var/lib/dkms/amd_gim/3.0/build/gim_atombios.o
  CC [M]  /var/lib/dkms/amd_gim/3.0/build/gim_os_service.o
  CC [M]  /var/lib/dkms/amd_gim/3.0/build/gim_irqmgr.o
  CC [M]  /var/lib/dkms/amd_gim/3.0/build/gim_kcl_os.o
  CC [M]  /var/lib/dkms/amd_gim/3.0/build/gim_pci_config.o
  CC [M]  /var/lib/dkms/amd_gim/3.0/build/gim_kcl_pci.o
  CC [M]  /var/lib/dkms/amd_gim/3.0/build/gim_reset.o
  CC [M]  /var/lib/dkms/amd_gim/3.0/build/gim_dma.o
  CC [M]  /var/lib/dkms/amd_gim/3.0/build/gim_monitor.o
  CC [M]  /var/lib/dkms/amd_gim/3.0/build/gim_monitor_ioctl.o
  CC [M]  /var/lib/dkms/amd_gim/3.0/build/gim_monitor_tonga.o
/var/lib/dkms/amd_gim/3.0/build/gim_monitor.c: In function ‘amdgim_op_gpuvf_vf’:
/var/lib/dkms/amd_gim/3.0/build/gim_monitor.c:997:2: error: implicit declaration of function ‘do_gettimeofday’; did you mean ‘do_settimeofday64’? [-Werror=implicit-function-declaration]
  do_gettimeofday(&cur_time);
  ^~~~~~~~~~~~~~~
  do_settimeofday64
/var/lib/dkms/amd_gim/3.0/build/gim_monitor_ioctl.c: In function ‘amdgim_mutex_get_lock’:
/var/lib/dkms/amd_gim/3.0/build/gim_monitor_ioctl.c:61:3: error: implicit declaration of function ‘do_gettimeofday’; did you mean ‘do_settimeofday64’? [-Werror=implicit-function-declaration]
   do_gettimeofday(&curr_t);
   ^~~~~~~~~~~~~~~
   do_settimeofday64
/var/lib/dkms/amd_gim/3.0/build/gim_flr.c: In function ‘gim_vf_flr’:
/var/lib/dkms/amd_gim/3.0/build/gim_flr.c:324:2: error: implicit declaration of function ‘do_gettimeofday’; did you mean ‘do_settimeofday64’? [-Werror=implicit-function-declaration]
  do_gettimeofday(&vf->time_log.reset_time);
  ^~~~~~~~~~~~~~~
  do_settimeofday64
/var/lib/dkms/amd_gim/3.0/build/gim_irqmgr.c: In function ‘handle_req_gpu_init_access’:
/var/lib/dkms/amd_gim/3.0/build/gim_irqmgr.c:1259:3: error: implicit declaration of function ‘do_gettimeofday’; did you mean ‘do_settimeofday64’? [-Werror=implicit-function-declaration]
   do_gettimeofday(&function->time_log.init_start);
   ^~~~~~~~~~~~~~~
   do_settimeofday64
/var/lib/dkms/amd_gim/3.0/build/gim_adapter.c: In function ‘idle_vf’:
/var/lib/dkms/amd_gim/3.0/build/gim_adapter.c:1799:9: error: implicit declaration of function ‘timespec_sub’; did you mean ‘timespec64_sub’? [-Werror=implicit-function-declaration]
   tmp = timespec_sub(tmp, func->time_log.active_last_tick);
         ^~~~~~~~~~~~
         timespec64_sub
/var/lib/dkms/amd_gim/3.0/build/gim_adapter.c:1799:7: error: incompatible types when assigning to type ‘struct timespec’ from type ‘int’
   tmp = timespec_sub(tmp, func->time_log.active_last_tick);
       ^
/var/lib/dkms/amd_gim/3.0/build/gim_adapter.c:1801:4: error: implicit declaration of function ‘timespec_add’; did you mean ‘timespec64_add’? [-Werror=implicit-function-declaration]
    timespec_add(func->time_log.active_time, tmp);
    ^~~~~~~~~~~~
    timespec64_add
/var/lib/dkms/amd_gim/3.0/build/gim_adapter.c:1800:30: error: incompatible types when assigning to type ‘struct timespec’ from type ‘int’
   func->time_log.active_time =
                              ^
cc1: some warnings being treated as errors
make[1]: *** [scripts/Makefile.build:286: /var/lib/dkms/amd_gim/3.0/build/gim_adapter.o] Error 1
make[1]: *** Waiting for unfinished jobs....
cc1: some warnings being treated as errors
make[1]: *** [scripts/Makefile.build:284: /var/lib/dkms/amd_gim/3.0/build/gim_flr.o] Error 1
cc1: some warnings being treated as errors
make[1]: *** [scripts/Makefile.build:286: /var/lib/dkms/amd_gim/3.0/build/gim_monitor_ioctl.o] Error 1
cc1: some warnings being treated as errors
make[1]: *** [scripts/Makefile.build:284: /var/lib/dkms/amd_gim/3.0/build/gim_irqmgr.o] Error 1
cc1: some warnings being treated as errors
make[1]: *** [scripts/Makefile.build:284: /var/lib/dkms/amd_gim/3.0/build/gim_monitor.o] Error 1
make: *** [Makefile:1606: _module_/var/lib/dkms/amd_gim/3.0/build] Error 2
make: Leaving directory '/usr/src/linux-headers-5.0.15-1-pve'

Is this a fullstop until AMD updates?

So made this compile successfully. There is a 5.0 kernel fix / patch on pull request for the MxGPU repo (can't post link, but only 3 pull requests and it's labeled well enough). Just checkout the branch and it'll build.

Here's the DKMS file I used:

Code: 
PACKAGE_NAME=amd_gim
PACKAGE_VERSION=3.0
BUILT_MODULE_NAME[0]=gim
MAKE[0]="make -C ${kernel_source_dir} M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build"
CLEAN="make -C ${kernel_source_dir} M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build clean"
DEST_MODULE_LOCATION[0]=/extra
REMAKE_INITRD=yes
AUTOINSTALL=yes
 
Last edited:
armouredking said:
Is this a fullstop until AMD updates?
Click to expand...
basically yes, but the changes are small enough

armouredking said:
So made this compile successfully. There is a 5.0 kernel fix / patch on pull request for the MxGPU repo (can't post link, but only 3 pull requests and it's labeled well enough). Just checkout the branch and it'll build.
Click to expand...
i know, the pull request if from me ;)
 
  • Like
Reactions: armouredking
Hey, @dcsapak have you noticed any unusual behaviour on reboot of the host for your card? I've found that a poweroff and poweron will show the expected lspci with all the various Tonga virtual adapters, but if I reboot the machine (ie by typing reboot at prompt) the machine will come back up with no Tonga devices. Not even the parent card itself will be present in lspci output.

I haven't been able to isolate specifically what's causing it, I've tried doing just manual loads to see if it was some kernel boot order issue but at the moment doesn't appear to be that. It's not gamebreaking, just haven't experienced a similar behaviour before where a hard power cycle was required to make PCI devices appear.

Edit:

After a reboot:
Code: 
# lspci -nnk
00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex [1022:1450]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex [1022:1450]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit [1022:1451]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit [1022:1451]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
00:01.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
00:01.5 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
00:03.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
00:03.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
   Kernel driver in use: pcieport
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
   Kernel driver in use: pcieport
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
   Subsystem: Super Micro Computer Inc FCH SMBus Controller [15d9:790b]
   Kernel driver in use: piix4_smbus
   Kernel modules: i2c_piix4, sp5100_tco
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
   Subsystem: Super Micro Computer Inc FCH LPC Bridge [15d9:790e]
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
   Kernel driver in use: k10temp
   Kernel modules: k10temp
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
00:19.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
00:19.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
00:19.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
00:19.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
   Kernel driver in use: k10temp
   Kernel modules: k10temp
00:19.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
00:19.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
00:19.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
00:19.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
00:1a.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
00:1a.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
00:1a.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
00:1a.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
   Kernel driver in use: k10temp
   Kernel modules: k10temp
00:1a.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
00:1a.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
00:1a.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
00:1a.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
00:1b.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
00:1b.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
00:1b.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
00:1b.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
   Kernel driver in use: k10temp
   Kernel modules: k10temp
00:1b.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
00:1b.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
00:1b.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
00:1b.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
01:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 04)
02:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 41)
   Subsystem: Super Micro Computer Inc ASPEED Graphics Family [15d9:1a03]
   Kernel driver in use: ast
   Kernel modules: ast
03:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
   Subsystem: Super Micro Computer Inc ASM1042A USB 3.0 Host Controller [15d9:1142]
   Kernel driver in use: xhci_hcd
04:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
   Subsystem: Super Micro Computer Inc I210 Gigabit Network Connection [15d9:1533]
   Kernel driver in use: igb
   Kernel modules: igb
05:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
   Subsystem: Super Micro Computer Inc I210 Gigabit Network Connection [15d9:1533]
   Kernel driver in use: igb
   Kernel modules: igb
06:00.0 Non-Volatile memory controller [0108]: Intel Corporation Device [8086:f1a8] (rev 03)
   Subsystem: Intel Corporation Device [8086:390d]
   Kernel driver in use: nvme
07:00.0 Non-Volatile memory controller [0108]: Intel Corporation Device [8086:f1a8] (rev 03)
   Subsystem: Intel Corporation Device [8086:390d]
   Kernel driver in use: nvme
08:00.0 Non-Volatile memory controller [0108]: Intel Corporation Device [8086:f1a8] (rev 03)
   Subsystem: Intel Corporation Device [8086:390d]
   Kernel driver in use: nvme
09:00.0 Non-Volatile memory controller [0108]: Intel Corporation Device [8086:f1a8] (rev 03)
   Subsystem: Intel Corporation Device [8086:390d]
   Kernel driver in use: nvme
0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
0a:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
   Kernel driver in use: ccp
   Kernel modules: ccp
0a:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB 3.0 Host controller [1022:145f]
   Subsystem: Super Micro Computer Inc USB 3.0 Host controller [15d9:145c]
   Kernel driver in use: xhci_hcd
0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
0b:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Zeppelin Cryptographic Coprocessor NTBCCP [1022:1468]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin Cryptographic Coprocessor NTBCCP [1022:1468]
   Kernel driver in use: ccp
   Kernel modules: ccp
0b:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
   Subsystem: Super Micro Computer Inc FCH SATA Controller [AHCI mode] [15d9:7901]
   Kernel driver in use: ahci
   Kernel modules: ahci
20:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex [1022:1450]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex [1022:1450]
20:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit [1022:1451]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit [1022:1451]
20:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
20:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
20:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
20:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
20:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
20:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
20:03.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
20:03.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
20:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
20:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
20:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
   Kernel driver in use: pcieport
20:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
20:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
   Kernel driver in use: pcieport
21:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS3224 PCI-Express Fusion-MPT SAS-3 [1000:00c4] (rev 01)
   Subsystem: LSI Logic / Symbios Logic SAS3224 PCI-Express Fusion-MPT SAS-3 [1000:31a0]
   Kernel driver in use: mpt3sas
   Kernel modules: mpt3sas
22:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
   Subsystem: Super Micro Computer Inc ASM1042A USB 3.0 Host Controller [15d9:1142]
   Kernel driver in use: xhci_hcd
23:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
   Subsystem: Super Micro Computer Inc ASM1042A USB 3.0 Host Controller [15d9:1142]
   Kernel driver in use: xhci_hcd
24:00.0 Non-Volatile memory controller [0108]: Intel Corporation Device [8086:f1a8] (rev 03)
   Subsystem: Intel Corporation Device [8086:390d]
   Kernel driver in use: nvme
25:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA [10b5:8724] (rev ca)
   Kernel driver in use: pcieport
26:01.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA [10b5:8724] (rev ca)
   Kernel driver in use: pcieport
26:08.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA [10b5:8724] (rev ca)
   Kernel driver in use: pcieport
26:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA [10b5:8724] (rev ca)
   Kernel driver in use: pcieport
27:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 [1000:0087] (rev 05)
   Subsystem: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 [1000:3070]
   Kernel driver in use: mpt3sas
   Kernel modules: mpt3sas
29:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 [1000:0087] (rev 05)
   Subsystem: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 [1000:3070]
   Kernel driver in use: mpt3sas
   Kernel modules: mpt3sas
2a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
2a:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
   Kernel driver in use: ccp
   Kernel modules: ccp
2a:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB 3.0 Host controller [1022:145f]
   Subsystem: Super Micro Computer Inc USB 3.0 Host controller [15d9:145c]
   Kernel driver in use: xhci_hcd
2b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
2b:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Zeppelin Cryptographic Coprocessor NTBCCP [1022:1468]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin Cryptographic Coprocessor NTBCCP [1022:1468]
   Kernel driver in use: ccp
   Kernel modules: ccp
40:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex [1022:1450]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex [1022:1450]
40:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit [1022:1451]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit [1022:1451]
40:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
40:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
40:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
40:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
40:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
40:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
40:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
   Kernel driver in use: pcieport
40:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
40:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
   Kernel driver in use: pcieport
41:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)
   Subsystem: Super Micro Computer Inc SAS3008 PCI-Express Fusion-MPT SAS-3 (AOC-S3008L-L8e) [15d9:0808]
   Kernel driver in use: mpt3sas
   Kernel modules: mpt3sas
42:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
42:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
   Kernel driver in use: ccp
   Kernel modules: ccp
43:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
43:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Zeppelin Cryptographic Coprocessor NTBCCP [1022:1468]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin Cryptographic Coprocessor NTBCCP [1022:1468]
   Kernel driver in use: ccp
   Kernel modules: ccp
43:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
   Subsystem: Super Micro Computer Inc FCH SATA Controller [AHCI mode] [15d9:7901]
   Kernel driver in use: ahci
   Kernel modules: ahci
60:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex [1022:1450]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex [1022:1450]
60:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit [1022:1451]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit [1022:1451]
60:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
60:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
60:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
60:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
60:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
60:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
60:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
60:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
60:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
   Kernel driver in use: pcieport
60:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
60:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
   Kernel driver in use: pcieport
63:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
   Subsystem: Intel Corporation Ethernet Server Adapter X520-2 [8086:000c]
   Kernel driver in use: ixgbe
   Kernel modules: ixgbe
63:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
   Subsystem: Intel Corporation Ethernet Server Adapter X520-2 [8086:000c]
   Kernel driver in use: ixgbe
   Kernel modules: ixgbe
65:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
65:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
   Kernel driver in use: ccp
   Kernel modules: ccp
66:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
66:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Zeppelin Cryptographic Coprocessor NTBCCP [1022:1468]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin Cryptographic Coprocessor NTBCCP [1022:1468]
   Kernel driver in use: ccp
   Kernel modules: ccp
 
Last edited:
Tried to edit but apparently message is too long.

Here's the system report with regards to the PCI devices:

After a power cycle:

Code: 
# lspci -nnk
00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex [1022:1450]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex [1022:1450]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit [1022:1451]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit [1022:1451]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
00:01.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
00:01.5 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
00:03.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
00:03.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
   Kernel driver in use: pcieport
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
   Kernel driver in use: pcieport
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
   Subsystem: Super Micro Computer Inc FCH SMBus Controller [15d9:790b]
   Kernel driver in use: piix4_smbus
   Kernel modules: i2c_piix4, sp5100_tco
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
   Subsystem: Super Micro Computer Inc FCH LPC Bridge [15d9:790e]
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
   Kernel driver in use: k10temp
   Kernel modules: k10temp
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
00:19.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
00:19.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
00:19.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
00:19.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
   Kernel driver in use: k10temp
   Kernel modules: k10temp
00:19.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
00:19.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
00:19.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
00:19.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
00:1a.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
00:1a.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
00:1a.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
00:1a.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
   Kernel driver in use: k10temp
   Kernel modules: k10temp
00:1a.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
00:1a.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
00:1a.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
00:1a.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
00:1b.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
00:1b.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
00:1b.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
00:1b.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
   Kernel driver in use: k10temp
   Kernel modules: k10temp
00:1b.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
00:1b.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
00:1b.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
00:1b.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
01:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 04)
02:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 41)
   Subsystem: Super Micro Computer Inc ASPEED Graphics Family [15d9:1a03]
   Kernel driver in use: ast
   Kernel modules: ast
03:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
   Subsystem: Super Micro Computer Inc ASM1042A USB 3.0 Host Controller [15d9:1142]
   Kernel driver in use: xhci_hcd
04:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
   Subsystem: Super Micro Computer Inc I210 Gigabit Network Connection [15d9:1533]
   Kernel driver in use: igb
   Kernel modules: igb
05:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
   Subsystem: Super Micro Computer Inc I210 Gigabit Network Connection [15d9:1533]
   Kernel driver in use: igb
   Kernel modules: igb
06:00.0 Non-Volatile memory controller [0108]: Intel Corporation Device [8086:f1a8] (rev 03)
   Subsystem: Intel Corporation Device [8086:390d]
   Kernel driver in use: nvme
07:00.0 Non-Volatile memory controller [0108]: Intel Corporation Device [8086:f1a8] (rev 03)
   Subsystem: Intel Corporation Device [8086:390d]
   Kernel driver in use: nvme
08:00.0 Non-Volatile memory controller [0108]: Intel Corporation Device [8086:f1a8] (rev 03)
   Subsystem: Intel Corporation Device [8086:390d]
   Kernel driver in use: nvme
09:00.0 Non-Volatile memory controller [0108]: Intel Corporation Device [8086:f1a8] (rev 03)
   Subsystem: Intel Corporation Device [8086:390d]
   Kernel driver in use: nvme
0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
0a:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
   Kernel driver in use: ccp
   Kernel modules: ccp
0a:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB 3.0 Host controller [1022:145f]
   Subsystem: Super Micro Computer Inc USB 3.0 Host controller [15d9:145c]
   Kernel driver in use: xhci_hcd
0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
0b:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Zeppelin Cryptographic Coprocessor NTBCCP [1022:1468]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin Cryptographic Coprocessor NTBCCP [1022:1468]
   Kernel driver in use: ccp
   Kernel modules: ccp
0b:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
   Subsystem: Super Micro Computer Inc FCH SATA Controller [AHCI mode] [15d9:7901]
   Kernel driver in use: ahci
   Kernel modules: ahci
20:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex [1022:1450]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex [1022:1450]
20:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit [1022:1451]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit [1022:1451]
20:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
20:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
20:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
20:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
20:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
20:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
20:03.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
20:03.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
20:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
20:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
20:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
   Kernel driver in use: pcieport
20:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
20:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
   Kernel driver in use: pcieport
21:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS3224 PCI-Express Fusion-MPT SAS-3 [1000:00c4] (rev 01)
   Subsystem: LSI Logic / Symbios Logic SAS3224 PCI-Express Fusion-MPT SAS-3 [1000:31a0]
   Kernel driver in use: mpt3sas
   Kernel modules: mpt3sas
22:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
   Subsystem: Super Micro Computer Inc ASM1042A USB 3.0 Host Controller [15d9:1142]
   Kernel driver in use: xhci_hcd
23:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
   Subsystem: Super Micro Computer Inc ASM1042A USB 3.0 Host Controller [15d9:1142]
   Kernel driver in use: xhci_hcd
24:00.0 Non-Volatile memory controller [0108]: Intel Corporation Device [8086:f1a8] (rev 03)
   Subsystem: Intel Corporation Device [8086:390d]
   Kernel driver in use: nvme
25:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA [10b5:8724] (rev ca)
   Kernel driver in use: pcieport
26:01.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA [10b5:8724] (rev ca)
   Kernel driver in use: pcieport
26:08.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA [10b5:8724] (rev ca)
   Kernel driver in use: pcieport
26:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA [10b5:8724] (rev ca)
   Kernel driver in use: pcieport
27:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 [1000:0087] (rev 05)
   Subsystem: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 [1000:3070]
   Kernel driver in use: mpt3sas
   Kernel modules: mpt3sas
29:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 [1000:0087] (rev 05)
   Subsystem: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 [1000:3070]
   Kernel driver in use: mpt3sas
   Kernel modules: mpt3sas
2a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
2a:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
   Kernel driver in use: ccp
   Kernel modules: ccp
2a:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB 3.0 Host controller [1022:145f]
   Subsystem: Super Micro Computer Inc USB 3.0 Host controller [15d9:145c]
   Kernel driver in use: xhci_hcd
2b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
2b:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Zeppelin Cryptographic Coprocessor NTBCCP [1022:1468]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin Cryptographic Coprocessor NTBCCP [1022:1468]
   Kernel driver in use: ccp
   Kernel modules: ccp
40:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex [1022:1450]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex [1022:1450]
40:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit [1022:1451]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit [1022:1451]
40:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
40:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
40:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
40:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
40:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
40:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
40:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
40:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
   Kernel driver in use: pcieport
40:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
40:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
   Kernel driver in use: pcieport
41:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)
   Subsystem: Super Micro Computer Inc SAS3008 PCI-Express Fusion-MPT SAS-3 (AOC-S3008L-L8e) [15d9:0808]
   Kernel driver in use: mpt3sas
   Kernel modules: mpt3sas
42:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ca)
   Kernel driver in use: pcieport
43:08.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ca)
   Kernel driver in use: pcieport
43:10.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ca)
   Kernel driver in use: pcieport
44:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XT GL [FirePro S7150] [1002:6929]
   Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XT GL [FirePro S7150] [1002:0334]
   Kernel modules: amdgpu
44:02.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
44:02.1 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
44:02.2 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
44:02.3 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
44:02.4 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
44:02.5 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
44:02.6 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
44:02.7 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
44:03.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
44:03.1 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
44:03.2 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
44:03.3 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
44:03.4 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
44:03.5 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
44:03.6 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
44:03.7 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
46:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XT GL [FirePro S7150] [1002:6929]
   Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XT GL [FirePro S7150] [1002:0334]
   Kernel modules: amdgpu
46:02.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
46:02.1 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
46:02.2 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
46:02.3 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
46:02.4 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
46:02.5 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
46:02.6 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
46:02.7 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
46:03.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
46:03.1 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
46:03.2 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
46:03.3 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
46:03.4 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
46:03.5 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
46:03.6 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
46:03.7 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] [1002:692f]
   Kernel modules: amdgpu
48:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
48:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
   Kernel driver in use: ccp
   Kernel modules: ccp
49:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
49:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Zeppelin Cryptographic Coprocessor NTBCCP [1022:1468]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin Cryptographic Coprocessor NTBCCP [1022:1468]
   Kernel driver in use: ccp
   Kernel modules: ccp
49:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
   Subsystem: Super Micro Computer Inc FCH SATA Controller [AHCI mode] [15d9:7901]
   Kernel driver in use: ahci
   Kernel modules: ahci
60:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex [1022:1450]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex [1022:1450]
60:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit [1022:1451]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit [1022:1451]
60:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
60:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
60:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
60:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
60:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
60:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
   Kernel driver in use: pcieport
60:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
60:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
60:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
   Kernel driver in use: pcieport
60:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
60:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
   Kernel driver in use: pcieport
63:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
   Subsystem: Intel Corporation Ethernet Server Adapter X520-2 [8086:000c]
   Kernel driver in use: ixgbe
   Kernel modules: ixgbe
63:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
   Subsystem: Intel Corporation Ethernet Server Adapter X520-2 [8086:000c]
   Kernel driver in use: ixgbe
   Kernel modules: ixgbe
65:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
65:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
   Kernel driver in use: ccp
   Kernel modules: ccp
66:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
66:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Zeppelin Cryptographic Coprocessor NTBCCP [1022:1468]
   Subsystem: Advanced Micro Devices, Inc. [AMD] Zeppelin Cryptographic Coprocessor NTBCCP [1022:1468]
   Kernel driver in use: ccp
   Kernel modules: ccp

It's always at addresses 44 and 46 at least.
 
armouredking said:
Hey, @dcsapak have you noticed any unusual behaviour on reboot of the host for your card? I've found that a poweroff and poweron will show the expected lspci with all the various Tonga virtual adapters, but if I reboot the machine (ie by typing reboot at prompt) the machine will come back up with no Tonga devices. Not even the parent card itself will be present in lspci output.
Click to expand...
no i did not notice such behaviour...
what mainboard do you use? maybe there is some bios setting relevant for this? e.g. some kind of power saving feature ?
 
It's a Supermicro H11SSL-NC. I have PCIE Spread Spectrum off. Not sure what else it could be.

upload_2019-7-25_5-39-4.png

upload_2019-7-25_5-39-30.png

The card is installed in Slot 4 (hence why 3 is disabled, since it's unusable).
 
mhmm... looks normal... anything in dmesg/journal during shutdown or boot ?
 
Well, there is a definite difference in the dmesg/journal between the two. Initial comparison between the files so far I only see that in the reboot set of logs the Tonga adapters are not mentioned (at all, again). The PCI sections are just blank in reboot compared to power cycle. Attaching files.

upload_2019-7-25_6-43-12.png

Edit:

This entry section is interesting:

upload_2019-7-25_6-58-16.png

It looks like there is some overlap here, and MTRR is at least peripherally related to GPUs in my understanding.
 

Attachments

  • PowerCycle-dmesg.txt
    200.6 KB · Views: 1
  • PowerCycle-journalctl.txt
    270.9 KB · Views: 0
  • Reboot-dmesg.txt
    150.2 KB · Views: 0
  • Reboot-journalctl.txt
    209.8 KB · Views: 0
Last edited:
also did you check if there is a bios update available for your mainboard?
 
Yes, I have checked. No BIOS updates for the mainboard (v1.1 is latest, mainboard is at v1.1). Been trying to research the MTRR thing. I've run across the MTRR cleanup option but I wanted to see how cat /proc/mtrr displayed, if it was different from a cold boot to a hot one. I have the answer to that question just now:

Hot reboot:
upload_2019-7-25_7-36-38.png

Cold boot:
upload_2019-7-25_7-36-56.png

Going to try and do the kernel edits now.
 
And that did it. I made the edit to the kernel cmdline and did a reboot, but it was no good. Nothing seemed to change. Following another power cycle, the hot reboots are now working. Additionally, I see the correct entries in /proc/mtrr, and I noticed another little something during boot:

upload_2019-7-25_8-7-10.png

Here's the extra tidbit. This is the hot reboot logs from dmesg, with default cmdline on left and with the modified one on right:

upload_2019-7-25_8-8-6.png

The right side is also a match for how memory looks during a cold boot. It looks like without that enable_mtrr_cleanup option in the kernel cmdline, some memory remains attached in a hot reboot and that buggers it up.

MTRR reboot on left, unmodified / modified (no change) cold boot on right:

upload_2019-7-25_8-11-20.png

So that's finally sorted. Thanks for the help with that.

On another, unrelated note: have you done any experimentation with assigning multiple vGPU? The current tutorial in wiki doesn't discuss it other than to say something is passed through. I know in ESXi and XCP-ng you're supposed to "split" the card into multiples of vGPU. See: https://drivers.amd.com/relnotes/amd_mxgpu_deploymentguide_vmware.pdf
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back