Trying to pass through this NIC to a VM for OPNsense to manage on its own, but it's not working.
dmesg when I attempt to start vm with PCIE device:
dmesg when I attempt to start vm with PCIE device:
[Tue Aug 29 13:31:47 2023] vfio-pci 0000:06:00.0: not ready 1023ms after bus reset; waiting
[Tue Aug 29 13:31:48 2023] vfio-pci 0000:06:00.0: not ready 2047ms after bus reset; waiting
[Tue Aug 29 13:31:51 2023] vfio-pci 0000:06:00.0: not ready 4095ms after bus reset; waiting
[Tue Aug 29 13:31:55 2023] vfio-pci 0000:06:00.0: not ready 8191ms after bus reset; waiting
[Tue Aug 29 13:32:03 2023] vfio-pci 0000:06:00.0: not ready 16383ms after bus reset; waiting
[Tue Aug 29 13:32:21 2023] vfio-pci 0000:06:00.0: not ready 32767ms after bus reset; waiting
[Tue Aug 29 13:32:56 2023] vfio-pci 0000:06:00.0: not ready 65535ms after bus reset; giving up
[Tue Aug 29 13:32:56 2023] vfio-pci 0000:06:00.1: vfio_bar_restore: reset recovery - restoring BARs
[Tue Aug 29 13:32:56 2023] vfio-pci 0000:06:00.0: vfio_bar_restore: reset recovery - restoring BARs
[Tue Aug 29 13:32:57 2023] vfio-pci 0000:06:00.0: not ready 1023ms after bus reset; waiting
[Tue Aug 29 13:32:58 2023] vfio-pci 0000:06:00.0: not ready 2047ms after bus reset; waiting
[Tue Aug 29 13:33:00 2023] vfio-pci 0000:06:00.0: not ready 4095ms after bus reset; waiting
[Tue Aug 29 13:33:05 2023] vfio-pci 0000:06:00.0: not ready 8191ms after bus reset; waiting
[Tue Aug 29 13:33:13 2023] vfio-pci 0000:06:00.0: not ready 16383ms after bus reset; waiting
[Tue Aug 29 13:33:30 2023] vfio-pci 0000:06:00.0: not ready 32767ms after bus reset; waiting
[Tue Aug 29 13:34:05 2023] vfio-pci 0000:06:00.0: not ready 65535ms after bus reset; giving up
[Tue Aug 29 13:34:05 2023] vfio-pci 0000:06:00.1: Unable to change power state from D0 to D3hot, device inaccessible
[Tue Aug 29 13:34:05 2023] vfio-pci 0000:06:00.0: Unable to change power state from D0 to D3hot, device inaccessible
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX="quiet amd_iommu=on iommu=pt"
06:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)
Subsystem: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet [14e4:0907]
Kernel driver in use: vfio-pci
Kernel modules: bnx2
06:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)
Subsystem: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet [14e4:0907]
Kernel driver in use: vfio-pci
Kernel modules: bnx2
# cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[ 0.078373] AMD-Vi: Unknown option - 'on'
[ 0.182259] AMD-Vi: ivrs, add hid:PNPD0040, uid:, rdevid:152
[ 0.182261] AMD-Vi: Using global IVHD EFR:0xf77ef22294ada, EFR2:0x0
[ 0.443602] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.444614] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 0.444616] AMD-Vi: Extended features (0xf77ef22294ada, 0x0): PPR NX GT IA GA PC GA_vAPIC
[ 0.444623] AMD-Vi: Interrupt remapping enabled
[ 0.444630] AMD-Vi: Virtual APIC enabled
[ 0.444949] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).