kernel: kernel BUG at mm/mmu_notifier.c:805!

[SC8198]

Having a weird issue where my server juts randomly hard locks and loses all connectivity.

Looks like a Kernel bug but I really don't know where to look. Did a google search and didn't find much of anything.

Virtual Environment 8.2.7

Spoiler: Code

Oct 01 07:58:04 jf kernel: ------------[ cut here ]------------
Oct 01 07:58:04 jf kernel: WARNING: CPU: 1 PID: 95602 at kernel/fork.c:920 __mmdrop+0x1ae/0x1d0
Oct 01 07:58:04 jf kernel: Modules linked in: veth rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache netfs vfio_pci vfio_pci_core vfio_iommu_type1 vfio iommufd ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter sctp ip6_udp_tunnel udp_tunnel nf_tables bonding tls softdog sunrpc nfnetlink_log nfnetlink binfmt_misc intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel snd_hda_codec_hdmi kvm nouveau ipmi_ssif irqbypass crct10dif_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel snd_hda_intel aesni_intel snd_intel_dspcfg snd_intel_sdw_acpi mxm_wmi drm_ttm_helper snd_hda_codec crypto_simd ttm cryptd snd_hda_core drm_display_helper rapl snd_hwdep cec snd_pcm mgag200 rc_core snd_timer ucsi_ccg drm_shmem_helper intel_cstate pcspkr snd drm_kms_helper typec_ucsi mei_me video soundcore typec mei ioatdma acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad joydev input_leds mac_hid zfs(PO) spl(O) vhost_net vhost vhost_iotlb
Oct 01 07:58:04 jf kernel: tap drm efi_pstore dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq hid_logitech_hidpp hid_logitech_dj dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c hid_generic usbmouse usbkbd usbhid hid isci xhci_pci xhci_pci_renesas crc32_pclmul igb ahci i2c_i801 ehci_pci libsas i2c_algo_bit libahci xhci_hcd i2c_smbus lpc_ich ehci_hcd dca i2c_nvidia_gpu scsi_transport_sas i2c_ccgx_ucsi wmi
Oct 01 07:58:04 jf kernel: CPU: 1 PID: 95602 Comm: iou-wrk-95342 Tainted: P O 6.5.11-4-pve #1
Oct 01 07:58:04 jf kernel: Hardware name: Supermicro X9DRi-LN4+/X9DR3-LN4+/X9DRi-LN4+/X9DR3-LN4+, BIOS 3.4 11/20/2019
Oct 01 07:58:04 jf kernel: RIP: 0010:__mmdrop+0x1ae/0x1d0
Oct 01 07:58:04 jf kernel: Code: 93 9d e8 c5 1f 0a 00 e9 29 ff ff ff be 03 00 00 00 48 89 d7 e8 a3 66 6b 00 e9 48 ff ff ff e8 69 19 13 00 e9 3e ff ff ff 0f 0b <0f> 0b e9 83 fe ff ff 0f 0b e9 92 fe ff ff 48 89 df e8 1c af 32 00
Oct 01 07:58:04 jf kernel: RSP: 0018:ffffc14e0d21bcc0 EFLAGS: 00010246
Oct 01 07:58:04 jf kernel: RAX: ffffa081598c1980 RBX: ffffa08152c69080 RCX: 0000000000000000
Oct 01 07:58:04 jf kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffa08152c69080
Oct 01 07:58:04 jf kernel: RBP: ffffc14e0d21bce8 R08: 0000000000000000 R09: 0000000000000000
Oct 01 07:58:04 jf kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffa0812f873100
Oct 01 07:58:04 jf kernel: R13: ffffa07dc02f1980 R14: ffffa08152c69080 R15: 0000000000000000
Oct 01 07:58:04 jf kernel: FS: 00007f2968ff96c0(0000) GS:ffffa0812f840000(0000) knlGS:0000000000000000
Oct 01 07:58:04 jf kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 01 07:58:04 jf kernel: CR2: 0000029475cfda7e CR3: 000000076c352001 CR4: 00000000001726e0
Oct 01 07:58:04 jf kernel: Call Trace:
Oct 01 07:58:04 jf kernel: <TASK>
Oct 01 07:58:04 jf kernel: ? show_regs+0x6d/0x80
Oct 01 07:58:04 jf kernel: ? __warn+0x89/0x160
Oct 01 07:58:04 jf kernel: ? __mmdrop+0x1ae/0x1d0
Oct 01 07:58:04 jf kernel: ? report_bug+0x17e/0x1b0
Oct 01 07:58:04 jf kernel: ? handle_bug+0x46/0x90
Oct 01 07:58:04 jf kernel: ? exc_invalid_op+0x18/0x80
Oct 01 07:58:04 jf kernel: ? asm_exc_invalid_op+0x1b/0x20
Oct 01 07:58:04 jf kernel: ? __mmdrop+0x1ae/0x1d0
Oct 01 07:58:04 jf kernel: finish_task_switch.isra.0+0x1a9/0x2c0
Oct 01 07:58:04 jf kernel: __schedule+0x405/0x1450
Oct 01 07:58:04 jf kernel: ? sysvec_apic_timer_interrupt+0xa6/0xd0
Oct 01 07:58:04 jf kernel: schedule+0x63/0x110
Oct 01 07:58:04 jf kernel: schedule_timeout+0x95/0x170
Oct 01 07:58:04 jf kernel: ? __pfx_process_timeout+0x10/0x10
Oct 01 07:58:04 jf kernel: io_wq_worker+0x1e9/0x3c0
Oct 01 07:58:04 jf kernel: ? raw_spin_rq_unlock+0x10/0x40
Oct 01 07:58:04 jf kernel: ? finish_task_switch.isra.0+0x85/0x2c0
Oct 01 07:58:04 jf kernel: ? __pfx_io_wq_worker+0x10/0x10
Oct 01 07:58:04 jf kernel: ret_from_fork+0x47/0x70
Oct 01 07:58:04 jf kernel: ? __pfx_io_wq_worker+0x10/0x10
Oct 01 07:58:04 jf kernel: ret_from_fork_asm+0x1b/0x30
Oct 01 07:58:04 jf kernel: RIP: 0033:0x0
Oct 01 07:58:04 jf kernel: Code: Unable to access opcode bytes at 0xffffffffffffffd6.
Oct 01 07:58:04 jf kernel: RSP: 002b:0000000000000000 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
Oct 01 07:58:04 jf kernel: RAX: 0000000000000000 RBX: 000055edc3144df0 RCX: 00007f2db7237b95
Oct 01 07:58:04 jf kernel: RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000011
Oct 01 07:58:04 jf kernel: RBP: 000055edc3144df8 R08: 0000000000000000 R09: 0000000000000008
Oct 01 07:58:04 jf kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000055edc3144ee0
Oct 01 07:58:04 jf kernel: R13: 0000000002846000 R14: 0000000000000012 R15: 00007f2908ecf790
Oct 01 07:58:04 jf kernel: </TASK>
Oct 01 07:58:04 jf kernel: ---[ end trace 0000000000000000 ]---
Oct 01 07:58:04 jf kernel: ------------[ cut here ]------------
Oct 01 07:58:04 jf kernel: kernel BUG at mm/mmu_notifier.c:805!

Any help would be appreciated. Lmk if you need anymore info.

 

[dakralex]

I haven't seen this before, but just for reference the warning occurs at [0] and the bug at [1] and seem to stem from an io_uring worker thread.

Has this occurred more than once or was this a one off? Is there are specific reason that the server runs the kernel version 6.5.11-4?

[0] https://elixir.bootlin.com/linux/v6.5.11/source/kernel/fork.c#L920
[1] https://elixir.bootlin.com/linux/v6.5.11/source/mm/mmu_notifier.c#L805

 

[SC8198]

This has been an occurring issue with this server for a few weeks. Last night was the first time I caught a error like that tho. I reinstalled proxmox on this node Two days ago so that might be why the Kernel is a weird version. I had upgraded the Kernel from the updates page but hadn't rebooted yet.

So it's most likely a io issue?

Do you think upgrading the Kernel would possibly fix the issue?

 

[SC8198]

Well, I upgraded the Kernel and changed my VM's to use async IO and not io_uring so that seems to have fixed it but it was most likely the kernel update that did the trick.
Thanks @dakralex for pointing me in the right direction.

 

[dakralex]

Thanks for getting back and great to hear that the update and settings change has fixed the problem.

Unfortunately, I couldn't find a similar bug report with a search on the LKML or kernel changelog either. If it should happen again, I will take a closer look into it!

 

[SC8198]

Okay it is continuing to crash haha. I spoke too soon. I am not able to find any errors except a ATA error that corresponds with my boot volume.

The previous error did show up on the video out but nothing in the logs and that happened only during one crash I haven't seen it since.

Oct 08 14:41:26 jf kernel: ata4.00: exception Emask 0x10 SAct 0x2000 SErr 0x400100 action 0x6 frozen
Oct 08 14:41:26 jf kernel: ata4.00: irq_stat 0x08000000, interface fatal error
Oct 08 14:41:26 jf kernel: ata4: SError: { UnrecovData Handshk }
Oct 08 14:41:26 jf kernel: ata4.00: failed command: WRITE FPDMA QUEUED
Oct 08 14:41:26 jf kernel: ata4.00: cmd 61/50:68:80:fc:80/00:00:0e:00:00/40 tag 13 ncq dma 40960 out
         res 40/00:01:00:00:00/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
Oct 08 14:41:26 jf kernel: ata4.00: status: { DRDY }
Oct 08 14:41:26 jf kernel: ata4: hard resetting link
Oct 08 14:41:26 jf kernel: ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Oct 08 14:41:26 jf kernel: ata4.00: configured for UDMA/133
Oct 08 14:41:26 jf kernel: ata4: EH complete

Would this cause this kind of instability? I assume it would but I am not entirely sure. I again tried to look this up and couldn't find info about what exactly WRITE FPDMA QUEUED meant. I understand I will need to possibly reseat or move sata ports and cables but if anyone is able to explain what WRITE FPDMA QUEUED means I would be grateful for the knowledge.

 

[SC8198]

also am seeing this as well. I also noticed it crashes when I have the two GPU's in the system passed through to VM's. I first thought it was power related but it has a 900watt and 850 watt redundant power supplies installed so I am not entirely sure.

kernel: sas: Enter sas_scsi_recover_host busy: 1 failed: 1
Oct 08 14:35:47 jf kernel: sas: ata8: end_device-6:0: cmd error handler
Oct 08 14:35:47 jf kernel: sas: ata8: end_device-6:0: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata9: end_device-6:1: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata12: end_device-6:2: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata14: end_device-6:3: dev error handler
Oct 08 14:35:47 jf kernel: sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 1 tries: 1
Oct 08 14:35:47 jf kernel: nouveau 0000:05:00.0: DRM: VRAM: 6144 MiB
Oct 08 14:35:47 jf kernel: nouveau 0000:05:00.0: DRM: GART: 536870912 MiB
Oct 08 14:35:47 jf kernel: nouveau 0000:05:00.0: DRM: BIT table 'A' not found
Oct 08 14:35:47 jf kernel: nouveau 0000:05:00.0: DRM: BIT table 'L' not found
Oct 08 14:35:47 jf kernel: nouveau 0000:05:00.0: DRM: TMDS table version 2.0
Oct 08 14:35:47 jf kernel: nouveau 0000:05:00.0: DRM: MM: using COPY for buffer copies
Oct 08 14:35:47 jf kernel: snd_hda_intel 0000:05:00.1: bound 0000:05:00.0 (ops nv50_audio_component_bind_ops [nouveau])
Oct 08 14:35:47 jf kernel: sas: Enter sas_scsi_recover_host busy: 1 failed: 1
Oct 08 14:35:47 jf kernel: sas: ata8: end_device-6:0: cmd error handler
Oct 08 14:35:47 jf kernel: sas: ata8: end_device-6:0: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata9: end_device-6:1: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata12: end_device-6:2: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata14: end_device-6:3: dev error handler
Oct 08 14:35:47 jf kernel: sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 1 tries: 1
Oct 08 14:35:47 jf kernel: sas: Enter sas_scsi_recover_host busy: 1 failed: 1
Oct 08 14:35:47 jf kernel: sas: ata8: end_device-6:0: cmd error handler
Oct 08 14:35:47 jf kernel: sas: ata8: end_device-6:0: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata9: end_device-6:1: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata12: end_device-6:2: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata14: end_device-6:3: dev error handler
Oct 08 14:35:47 jf kernel: sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 1 tries: 1
Oct 08 14:35:47 jf kernel: sas: Enter sas_scsi_recover_host busy: 1 failed: 1
Oct 08 14:35:47 jf kernel: sas: ata8: end_device-6:0: cmd error handler
Oct 08 14:35:47 jf kernel: sas: ata8: end_device-6:0: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata9: end_device-6:1: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata12: end_device-6:2: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata14: end_device-6:3: dev error handler
Oct 08 14:35:47 jf kernel: sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 1 tries: 1
Oct 08 14:35:47 jf kernel: sas: Enter sas_scsi_recover_host busy: 1 failed: 1
Oct 08 14:35:47 jf kernel: sas: ata9: end_device-6:1: cmd error handler
Oct 08 14:35:47 jf kernel: sas: ata8: end_device-6:0: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata9: end_device-6:1: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata12: end_device-6:2: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata14: end_device-6:3: dev error handler
Oct 08 14:35:47 jf kernel: sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 1 tries: 1
Oct 08 14:35:47 jf kernel: sas: Enter sas_scsi_recover_host busy: 1 failed: 1
Oct 08 14:35:47 jf kernel: sas: ata9: end_device-6:1: cmd error handler
Oct 08 14:35:47 jf kernel: sas: ata8: end_device-6:0: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata9: end_device-6:1: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata12: end_device-6:2: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata14: end_device-6:3: dev error handler
Oct 08 14:35:47 jf kernel: sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 1 tries: 1
Oct 08 14:35:47 jf kernel: sas: Enter sas_scsi_recover_host busy: 1 failed: 1
Oct 08 14:35:47 jf kernel: sas: ata9: end_device-6:1: cmd error handler
Oct 08 14:35:47 jf kernel: sas: ata8: end_device-6:0: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata9: end_device-6:1: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata12: end_device-6:2: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata14: end_device-6:3: dev error handler
Oct 08 14:35:47 jf kernel: sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 1 tries: 1
Oct 08 14:35:47 jf kernel: sas: Enter sas_scsi_recover_host busy: 1 failed: 1
Oct 08 14:35:47 jf kernel: sas: ata9: end_device-6:1: cmd error handler
Oct 08 14:35:47 jf kernel: sas: ata8: end_device-6:0: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata9: end_device-6:1: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata12: end_device-6:2: dev error handler
Oct 08 14:35:47 jf kernel: sas: ata14: end_device-6:3: dev error handler
Oct 08 14:35:47 jf kernel: sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 1 tries: 1
Oct 08 14:35:47 jf kernel: [drm] Initialized nouveau 1.4.0 20120801 for 0000:05:00.0 on minor 1
Oct 08 14:35:47 jf kernel: nouveau 0000:05:00.0: [drm] Cannot find any crtc or sizes
Oct 08 14:35:47 jf kernel: nouveau 0000:05:00.0: [drm] Cannot find any crtc or sizes
Oct 08 14:35:47 jf kernel: nouveau 0000:05:00.0: [drm] Cannot find any crtc or sizes
Oct 08 14:35:47 jf kernel: nouveau 0000:83:00.0: enabling device (0000 -> 0003)
Oct 08 14:35:47 jf kernel: nouveau 0000:83:00.0: NVIDIA TU106 (166000a1)
Oct 08 14:35:47 jf kernel: nouveau 0000:83:00.0: bios: version 90.06.4c.00.50
Oct 08 14:35:47 jf kernel: nouveau 0000:83:00.0: pmu: firmware unavailable
Oct 08 14:35:47 jf kernel: nouveau 0000:83:00.0: fb: 8192 MiB GDDR6
Oct 08 14:35:47 jf kernel: nouveau 0000:83:00.0: DRM: VRAM: 8192 MiB
Oct 08 14:35:47 jf kernel: nouveau 0000:83:00.0: DRM: GART: 536870912 MiB
Oct 08 14:35:47 jf kernel: nouveau 0000:83:00.0: DRM: BIT table 'A' not found
Oct 08 14:35:47 jf kernel: nouveau 0000:83:00.0: DRM: BIT table 'L' not found
Oct 08 14:35:47 jf kernel: nouveau 0000:83:00.0: DRM: TMDS table version 2.0
Oct 08 14:35:47 jf kernel: nouveau 0000:83:00.0: DRM: MM: using COPY for buffer copies
Oct 08 14:35:47 jf kernel: snd_hda_intel 0000:83:00.1: bound 0000:83:00.0 (ops nv50_audio_component_bind_ops [nouveau])
Oct 08 14:35:47 jf kernel: [drm] Initialized nouveau 1.4.0 20120801 for 0000:83:00.0 on minor 2
Oct 08 14:35:47 jf kernel: nouveau 0000:83:00.0: [drm] Cannot find any crtc or sizes
Oct 08 14:35:47 jf kernel: nouveau 0000:83:00.0: [drm] Cannot find any crtc or sizes
Oct 08 14:35:47 jf kernel: nouveau 0000:83:00.0: [drm] Cannot find any crtc or sizes
Oct 08 14:35:47 jf kernel: nouveau 0000:83:00.0: [drm] Cannot find any crtc or sizes
Oct 08 14:35:47 jf kernel: nouveau 0000:83:00.0: [drm] Cannot find any crtc or sizes

 

[dakralex]

The SATA error you are getting seems to be caused by a hardware error (specifically between the controller and device as indicated by Emask 0x10). The "WRITE FPDMA QUEUED" is a Native Command Queuing command [0], which would fail on CRC errors, that is transmission errors (which I assume here). Have you checked if the SATA cable is loose or damaged? As it's your boot volume, it would be hard to ask you to see if the error happens when it is unplugged

[0] Page 289 in https://people.freebsd.org/~imp/asiabsdcon2015/works/d2161r5-ATAATAPI_Command_Set_-_3.pdf

 

[dakralex]

Could you tell me how you have set up the GPU passthrough on your host and VM? An additional (verbose) output of lspci and how your IOMMU groups are setup would also be helpful in finding an answer to the problem.

 

[SC8198]

Hey thanks for the response.

Here is how it they are passed through to the VM's

 

[SC8198]

Here is the output to lscpi I haven't done anything special to passthrough the GPUS to the VM's. I guess I should have done a little bit more research.

00:00.0 Host bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 DMI2 (rev 04)
00:01.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a (rev 04)
00:01.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1b (rev 04)
00:02.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04)
00:03.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04)
00:04.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 (rev 04)
00:04.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 (rev 04)
00:04.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 (rev 04)
00:04.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 (rev 04)
00:04.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 (rev 04)
00:04.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 (rev 04)
00:04.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 (rev 04)
00:04.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 (rev 04)
00:05.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc (rev 04)
00:05.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS (rev 04)
00:05.4 PIC: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC (rev 04)
00:16.0 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #1 (rev 05)
00:16.1 Communication controller: Intel Corporation C600/X79 series chipset MEI Controller #2 (rev 05)
00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 06)
00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 (rev b6)
00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 06)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a6)
00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller (rev 06)
00:1f.2 SATA controller: Intel Corporation C600/X79 series chipset 6-Port SATA AHCI Controller (rev 06)
00:1f.3 SMBus: Intel Corporation C600/X79 series chipset SMBus Host Controller (rev 06)
00:1f.6 Signal processing controller: Intel Corporation C600/X79 series chipset Thermal Management Controller (rev 06)
01:00.0 PCI bridge: Intel Corporation C608/C606/X79 series chipset PCI Express Upstream Port (rev 06)
02:08.0 PCI bridge: Intel Corporation C608/C606/X79 series chipset PCI Express Virtual Switch Port (rev 06)
03:00.0 Serial Attached SCSI controller: Intel Corporation C606 chipset Dual 4-Port SATA/SAS Storage Control Unit (rev 06)
05:00.0 VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1660 SUPER] (rev a1)
05:00.1 Audio device: NVIDIA Corporation TU116 High Definition Audio Controller (rev a1)
05:00.2 USB controller: NVIDIA Corporation TU116 USB 3.1 Host Controller (rev a1)
05:00.3 Serial bus controller: NVIDIA Corporation TU116 USB Type-C UCSI Controller (rev a1)
07:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
07:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
07:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
07:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
09:01.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)
7f:08.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 (rev 04)
7f:09.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 1 (rev 04)
7f:0a.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 0 (rev 04)
7f:0a.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 1 (rev 04)
7f:0a.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 2 (rev 04)
7f:0a.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 3 (rev 04)
7f:0b.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04)
7f:0b.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers (rev 04)
7f:0c.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0c.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0c.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0c.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0c.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0c.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0d.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0d.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0d.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0d.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0d.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0d.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers (rev 04)
7f:0e.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
7f:0e.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 (rev 04)
7f:0f.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers (rev 04)
7f:0f.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers (rev 04)
7f:0f.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
7f:0f.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
7f:0f.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
7f:0f.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers (rev 04)
7f:10.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 (rev 04)
7f:10.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 (rev 04)
7f:10.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 (rev 04)
7f:10.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 (rev 04)
7f:10.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 (rev 04)
7f:10.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 (rev 04)
7f:10.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 (rev 04)
7f:10.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 (rev 04)
7f:13.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04)
7f:13.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe (rev 04)
7f:13.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Registers (rev 04)
7f:13.5 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring (rev 04)
7f:16.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 System Address Decoder (rev 04)
7f:16.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
7f:16.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers (rev 04)
7f:1c.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 1 (rev 04)
7f:1c.1 Performance counters: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 1 (rev 04)
7f:1d.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Target Address/Thermal Registers (rev 04)
7f:1d.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 RAS Registers (rev 04)
7f:1d.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
7f:1d.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
7f:1d.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
7f:1d.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder Registers (rev 04)
7f:1e.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 0 (rev 04)
7f:1e.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 1 (rev 04)
7f:1e.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 0 (rev 04)
7f:1e.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 1 (rev 04)
7f:1e.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 2 (rev 04)
7f:1e.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 Thermal Control 3 (rev 04)
7f:1e.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 2 (rev 04)
7f:1e.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel 0-3 ERROR Registers 3 (rev 04)
80:00.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port in DMI2 Mode (rev 04)
80:01.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a (rev 04)
80:02.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04)
80:03.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04)
80:04.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 (rev 04)
80:04.1 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 (rev 04)
80:04.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 (rev 04)
80:04.3 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 (rev 04)
80:04.4 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 (rev 04)
80:04.5 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 (rev 04)
80:04.6 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 (rev 04)
80:04.7 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 (rev 04)
80:05.0 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc (rev 04)
80:05.2 System peripheral: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS (rev 04)
80:05.4 PIC: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC (rev 04)
83:00.0 VGA compatible controller: NVIDIA Corporation TU106 [GeForce RTX 2060 SUPER] (rev a1)
83:00.1 Audio device: NVIDIA Corporation TU106 High Definition Audio Controller (rev a1)
83:00.2 USB controller: NVIDIA Corporation TU106 USB 3.1 Host Controller (rev a1)
83:00.3 Serial bus controller: NVIDIA Corporation TU106 USB Type-C UCSI Controller (rev a1)

Had to remove some stuff from the bottom but it was mostly Integrated Memory controller.