Proxmox PCIE Passthrough Multiple NIC Interfaces

S

Splise

Member
Nov 11, 2019
23
0
21
Hi,

I am working to implement a Proxmox host, solely dedicated to a Palo Alto VM series firewall. I am wanting to enable PCIe pass-through on the following NICs/interfaces:
1 x Chelsio T520 -BT (Dual port 10 Gb)
1 x Intel I350 (Quad port 1 Gb)
1 x Intel 82574L (Single port Gb)

All of the respective card's interfaces are recognized by the Palo Alto firewall, except for the second port on the Chelsio T520-BT. I have attempted a multitude of different configurations/suggestions in an attempt to bring up the second interface without success. I believe I now have a lot of extraneous configuration that is not required. From what I've read, there doesn't appear to be a clear solution to getting this working within Proxmox. If anyone has any guidance on how to bring up the additional interface, it would be greatly appreciated. Also, if there are suggestions on config optimization, I am also wanting to remove anything that is unneeded.

System specs:
ThinkCentre M93p:
Haswell i7-4770
Intel Q87 express chipset

#dmesg | grep vfio
[ 17.981032] vfio_pci: add [0001:0000[ffffffff:ffffffff]] class 0x000000/00000000
[ 17.981035] vfio_pci: add [0001:0000[ffffffff:ffffffff]] class 0x000000/00000000
[ 17.981037] vfio_pci: add [0001:0000[ffffffff:ffffffff]] class 0x000000/00000000
[ 17.981041] vfio_pci: add [0001:0000[ffffffff:ffffffff]] class 0x000000/00000000
[ 17.981043] vfio_pci: add [0001:0000[ffffffff:ffffffff]] class 0x000000/00000000
[ 17.981046] vfio_pci: add [0001:0000[ffffffff:ffffffff]] class 0x000000/00000000
[ 17.981048] vfio_pci: add [0001:0000[ffffffff:ffffffff]] class 0x000000/00000000
[ 17.981051] vfio_pci: add [0003:0000[ffffffff:ffffffff]] class 0x000000/00000000
[ 17.981053] vfio_pci: add [0006:0000[ffffffff:ffffffff]] class 0x000000/00000000
[ 17.981056] vfio_pci: add [0006:0000[ffffffff:ffffffff]] class 0x000000/00000000
[ 17.981059] vfio_pci: add [0006:0000[ffffffff:ffffffff]] class 0x000000/00000000
[ 17.981076] vfio_pci: add [0006:0000[ffffffff:ffffffff]] class 0x000000/00000000
[ 781.853524] vfio-pci 0000:03:00.0: enabling device (0000 -> 0003)
[ 782.003747] vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 782.111740] vfio-pci 0000:01:00.1: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 782.219737] vfio-pci 0000:01:00.2: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 782.327743] vfio-pci 0000:01:00.3: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 782.435728] vfio-pci 0000:01:00.4: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 782.437368] vfio-pci 0000:06:00.0: enabling device (0000 -> 0002)
[ 782.546196] vfio-pci 0000:06:00.1: enabling device (0000 -> 0002)
[ 782.677932] vfio-pci 0000:06:00.2: enabling device (0000 -> 0002)
[ 782.786082] vfio-pci 0000:06:00.3: enabling device (0000 -> 0002)
[ 1779.692394] vfio-pci 0000:03:00.0: enabling device (0000 -> 0003)
[ 1779.858485] vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 1779.966475] vfio-pci 0000:01:00.1: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 1780.074418] vfio-pci 0000:01:00.2: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 1780.182473] vfio-pci 0000:01:00.3: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 1780.290414] vfio-pci 0000:01:00.4: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 1780.398462] vfio-pci 0000:01:00.5: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 1780.526421] vfio-pci 0000:01:00.6: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 2923.665032] vfio-pci 0000:03:00.0: enabling device (0000 -> 0003)
[ 2923.821258] vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 2923.929248] vfio-pci 0000:01:00.1: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 2924.037245] vfio-pci 0000:01:00.2: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 2924.145243] vfio-pci 0000:01:00.3: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 2924.253241] vfio-pci 0000:01:00.4: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 2924.361260] vfio-pci 0000:01:00.5: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 2924.485238] vfio-pci 0000:01:00.6: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 2932.384224] vfio-pci 0000:06:00.1: Invalid PCI ROM header signature: expecting 0xaa55, got 0xaa55
[ 3463.618851] vfio-pci 0000:03:00.0: enabling device (0000 -> 0003)
[ 3463.768974] vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 3463.876966] vfio-pci 0000:01:00.1: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 3463.984924] vfio-pci 0000:01:00.2: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 3464.092956] vfio-pci 0000:01:00.3: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 3464.200930] vfio-pci 0000:01:00.4: vfio_ecap_init: hiding ecap 0x19@0x1a0
[ 3464.308955] vfio-pci 0000:01:00.5: vfio_ecap_init: hiding ecap 0x19@0x1a0


#lspci -t
-[0000:00]-+-00.0
+-01.0-[01]--+-00.0
| +-00.1
| +-00.2
| +-00.3
| +-00.4
| +-00.5
| \-00.6
+-02.0
+-03.0
+-14.0
+-16.0
+-19.0
+-1a.0
+-1b.0
+-1c.0-[02]--
+-1c.1-[03]----00.0
+-1c.3-[04-05]----00.0-[05]--
+-1c.4-[06]--+-00.0
| +-00.1
| +-00.2
| \-00.3
+-1d.0
+-1f.0
+-1f.2
\-1f.3


Modules:
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

/sbin/vfio-pci.sh:
#!/bin/sh

for device in "$@"; do
if [ -e /sys/bus/pci/devices/$device/driver ]; then
# force an unbind if another driver got to the device first
echo $device > /sys/bus/pci/devices/$device/driver/unbind
fi
echo vfio-pci > /sys/bus/pci/devices/$device/driver_override
done

# install the vfio-pci module as usual
modprobe -i vfio-pci


igb.conf:
options igb max_vfs=4

iommu_unsafe_interrupts.conf:
options vfio_iommu_type1 allow_unsafe_interrupts=1

kvm.conf:
options kvm ignore_msrs=1

vfio.conf:
options vfio-pci ids=01:00.0,01:00.1,01:00.2,01:00.3,01:00.4,01:00.5,01:00.6,03:00.0,06:00.0,06:00.1,06:00.2,06:00.3

vfio-pci.conf:
install vfio-pci /sbin/vfio-pci.sh 0000:01:00.0 0000:01:00.1 0000:01:00.2 0000:01:00.3 0000:01:00.4 0000:01:00.5 0000:01:00.6 0000:06:00.0 0000:06:00.1 0000:06:00.2 0000:06:00.3


/etc/pve/qemu-server/100.conf:
agent: 1
balloon: 0
bootdisk: scsi0
cores: 4
hostpci1: 03:00.0,pcie=1
hostpci2: 01:00.0;01:00.1;01:00.2;01:00.3;01:00.4;01:00.5;01:00.6,pcie=1
hostpci3: 06:00.0;06:00.1;06:00.2;06:00.3,pcie=1
ide2: none,media=cdrom
machine: q35
memory: 12288
name: tu-vfw-fw-01
net0: virtio=D6:52:66:37:D1:E3,bridge=vmbr0
numa: 0
ostype: l26
scsi0: images:100/vm-100-disk-0.qcow2,cache=writethrough,size=80G
scsihw: virtio-scsi-pci
serial1: socket
smbios1: uuid=7d6e6226-2f01-42e8-aa22-1b04856d52ba
sockets: 1
vmgenid: 0f832cde-5d74-4d8d-afed-dd51669aa56a


/etc/default/grub:
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="rootdelay=10 quiet intel_iommu=on iommu=pt"
GRUB_CMDLINE_LINUX="root=ZFS=rpool/ROOT/pve-1 boot=zfs"

# Disable os-prober, it might add menu entries for each guest
GRUB_DISABLE_OS_PROBER=true

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
GRUB_INIT_TUNE="480 440 1"
 
Last edited:
#find /sys/kernel/iommu_groups -type l | sort -t '/' -n -k 5
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.1
/sys/kernel/iommu_groups/1/devices/0000:01:00.2
/sys/kernel/iommu_groups/1/devices/0000:01:00.3
/sys/kernel/iommu_groups/1/devices/0000:01:00.4
/sys/kernel/iommu_groups/1/devices/0000:01:00.5
/sys/kernel/iommu_groups/1/devices/0000:01:00.6
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/3/devices/0000:00:03.0
/sys/kernel/iommu_groups/4/devices/0000:00:14.0
/sys/kernel/iommu_groups/5/devices/0000:00:16.0
/sys/kernel/iommu_groups/6/devices/0000:00:19.0
/sys/kernel/iommu_groups/7/devices/0000:00:1a.0
/sys/kernel/iommu_groups/8/devices/0000:00:1b.0
/sys/kernel/iommu_groups/9/devices/0000:00:1c.0
/sys/kernel/iommu_groups/10/devices/0000:00:1c.1
/sys/kernel/iommu_groups/11/devices/0000:00:1c.3
/sys/kernel/iommu_groups/12/devices/0000:00:1c.4
/sys/kernel/iommu_groups/13/devices/0000:00:1d.0
/sys/kernel/iommu_groups/14/devices/0000:00:1f.0
/sys/kernel/iommu_groups/14/devices/0000:00:1f.2
/sys/kernel/iommu_groups/14/devices/0000:00:1f.3
/sys/kernel/iommu_groups/15/devices/0000:03:00.0
/sys/kernel/iommu_groups/16/devices/0000:04:00.0
/sys/kernel/iommu_groups/17/devices/0000:06:00.0
/sys/kernel/iommu_groups/18/devices/0000:06:00.1
/sys/kernel/iommu_groups/19/devices/0000:06:00.2
/sys/kernel/iommu_groups/20/devices/0000:06:00.3


#dmesg | grep -e DMAR -e IOMMU
root@tu-psvr-vs-02:/etc/modprobe.d# dmesg | grep -e DMAR -e IOMMU
[ 0.007394] ACPI: DMAR 0x00000000DADAD378 0000B8 (v01 LENOVO TC-FB 31444230 INTL 00000001)
[ 0.044311] DMAR: IOMMU enabled
[ 0.101254] DMAR: Host address width 39
[ 0.101256] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.101260] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[ 0.101262] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.101264] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da
[ 0.101266] DMAR: RMRR base: 0x000000dba4b000 end: 0x000000dba71fff
[ 0.101267] DMAR: RMRR base: 0x000000dd000000 end: 0x000000df1fffff
[ 0.101270] DMAR-IR: IOAPIC id 8 under DRHD base 0xfed91000 IOMMU 1
[ 0.101271] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ 0.101272] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.101648] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.634750] DMAR: No ATSR found
[ 0.634790] DMAR: dmar0: Using Queued invalidation
[ 0.634795] DMAR: dmar1: Using Queued invalidation
[ 0.706016] DMAR: Intel(R) Virtualization Technology for Directed I/O
[ 18.202617] [drm] DMAR active, disabling use of stolen memory

Available interfaces on Palo Alto VM:
Interface_name Base-OS_port Base-OS_MAC PCI-ID Driver
mgt eth0 a1:05:ca:3b:a7:aa 0000:01:00.0 e1000e
Ethernet1/5 eth5 b1:07:43:34:50:ba 0000:02:00.4 cxgb4
Ethernet1/6 eth6 c2:36:9f:83:cc:ca 0000:03:00.0 igb
Ethernet1/7 eth7 c2:36:9f:83:cc:cb 0000:03:00.1 igb
Ethernet1/8 eth8 c2:36:9f:83:cc:cc 0000:03:00.2 igb
Ethernet1/9 eth9 c2:36:9f:83:cc:cd 0000:03:00.3 igb
Ethernet1/10 eth10 d1:ea:70:91:b7:da 0000:06:12.0 virtio_net
 
Last edited:
Is anyone able to provided some guidance on this? I would really like to get this working.
 
I am fairly new to Proxmox, so maybe there's an obvious solution I am just not seeing? I was reading other posts where people seemed to be experiencing a similar issue, I am just looking to see if anyone has any idea why this may not be working? Would buying a Community Subscription provide additional support for issues like this?
 
Splise said:
All of the respective card's interfaces are recognized by the Palo Alto firewall, except for the second port on the Chelsio T520-BT.
Click to expand...
this sound rather like a guest driver problem...

Splise said:
hostpci1: 03:00.0,pcie=1
hostpci2: 01:00.0;01:00.1;01:00.2;01:00.3;01:00.4;01:00.5;01:00.6,pcie=1
hostpci3: 06:00.0;06:00.1;06:00.2;06:00.3,pcie=1
Click to expand...
if those are all of the functions you could use the shorthand syntax (just a tip)
hostpci2: 01:00,pcie=1

can you also post the output of 'lspci -k' ?
 
Thanks for your response! It looks like Proxmox is not recognizing the second port of the NIC from what I can tell.

dcsapak said:
this sound rather like a guest driver problem...

I am seeing one of the interfaces on the guest itself. I thought both ports were showing up in Proxmox, but when I took another look, it appears as though Proxmox may not be recognizing the second port as well? I assumed there should be a "enp1s0f4d2" entry?

Code: 
auto lo
iface lo inet loopback

iface eno1 inet manual

iface enp6s0f0 inet manual

iface enp3s0 inet manual

iface enp6s0f1 inet manual

iface enp6s0f2 inet manual

iface enp6s0f3 inet manual

iface enp1s0f4 inet manual

iface enp1s0f4d1 inet manual

auto vmbr0

iface vmbr0 inet static
        address  192.168.50.2
        netmask  24
        gateway  192.168.50.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0



if those are all of the functions you could use the shorthand syntax (just a tip)
hostpci2: 01:00,pcie=1

Thanks for the suggestion. I was playing around with the individual interfaces to see if removing the last two would help, but then added them in anyway. I have added "01:00,pcie=1" now.

I can see that the Chelsio NIC's interfaces are all in the same group, whereas the 4 port Intel NIC has a different group for each interfaces:

Code: 
dmesg | grep -i iommu

[    0.000000] Command line: initrd=\EFI\proxmox\5.3.13-2-pve\initrd.img-5.3.13-2-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on rootdelay=10
[    0.080115] Kernel command line: initrd=\EFI\proxmox\5.3.13-2-pve\initrd.img-5.3.13-2-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on rootdelay=10
[    0.080164] DMAR: IOMMU enabled
[    0.176498] DMAR-IR: IOAPIC id 8 under DRHD base  0xfed91000 IOMMU 1
[    0.855317] pci 0000:00:00.0: Adding to iommu group 0
[    0.855367] pci 0000:00:01.0: Adding to iommu group 1
[    0.857742] pci 0000:00:02.0: Adding to iommu group 2
[    0.857785] pci 0000:00:03.0: Adding to iommu group 3
[    0.857857] pci 0000:00:14.0: Adding to iommu group 4
[    0.857908] pci 0000:00:16.0: Adding to iommu group 5
[    0.857946] pci 0000:00:19.0: Adding to iommu group 6
[    0.858021] pci 0000:00:1a.0: Adding to iommu group 7
[    0.858060] pci 0000:00:1b.0: Adding to iommu group 8
[    0.858111] pci 0000:00:1c.0: Adding to iommu group 9
[    0.858153] pci 0000:00:1c.1: Adding to iommu group 10
[    0.858207] pci 0000:00:1c.3: Adding to iommu group 11
[    0.858246] pci 0000:00:1c.4: Adding to iommu group 12
[    0.858323] pci 0000:00:1d.0: Adding to iommu group 13
[    0.858373] pci 0000:00:1f.0: Adding to iommu group 14
[    0.858383] pci 0000:00:1f.2: Adding to iommu group 14
[    0.858392] pci 0000:00:1f.3: Adding to iommu group 14
[    0.858401] pci 0000:01:00.0: Adding to iommu group 1
[    0.858407] pci 0000:01:00.1: Adding to iommu group 1
[    0.858413] pci 0000:01:00.2: Adding to iommu group 1
[    0.858418] pci 0000:01:00.3: Adding to iommu group 1
[    0.858423] pci 0000:01:00.4: Adding to iommu group 1
[    0.858429] pci 0000:01:00.5: Adding to iommu group 1
[    0.858434] pci 0000:01:00.6: Adding to iommu group 1
[    0.858486] pci 0000:03:00.0: Adding to iommu group 15
[    0.858527] pci 0000:04:00.0: Adding to iommu group 16
[    0.858579] pci 0000:06:00.0: Adding to iommu group 17
[    0.858617] pci 0000:06:00.1: Adding to iommu group 18
[    0.858665] pci 0000:06:00.2: Adding to iommu group 19
[    0.858702] pci 0000:06:00.3: Adding to iommu group 20
Click to expand...
 
can you also post the output of 'lspci -k' ?
[/QUOTE]

Code: 
# lspci -k:
00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)
        Subsystem: Lenovo 4th Gen Core Processor DRAM Controller
        Kernel driver in use: hsw_uncore
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
        Kernel driver in use: pcieport
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
        Subsystem: Lenovo Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller
        Kernel driver in use: i915
        Kernel modules: i915
00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)
        Subsystem: Lenovo Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 04)
        Subsystem: Lenovo 8 Series/C220 Series Chipset Family USB xHCI
        Kernel driver in use: xhci_hcd
00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
        Subsystem: Lenovo 8 Series/C220 Series Chipset Family MEI Controller
        Kernel driver in use: mei_me
        Kernel modules: mei_me
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-LM (rev 04)
        Subsystem: Lenovo Ethernet Connection I217-LM
        Kernel driver in use: e1000e
        Kernel modules: e1000e
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 04)
        Subsystem: Lenovo 8 Series/C220 Series Chipset Family USB EHCI
        Kernel driver in use: ehci-pci
00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 04)
        Subsystem: Lenovo 8 Series/C220 Series Chipset High Definition Audio Controller
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4)
        Kernel driver in use: pcieport
00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 (rev d4)
        Kernel driver in use: pcieport
00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d4)
        Kernel driver in use: pcieport
00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d4)
        Kernel driver in use: pcieport
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 04)
        Subsystem: Lenovo 8 Series/C220 Series Chipset Family USB EHCI
        Kernel driver in use: ehci-pci
00:1f.0 ISA bridge: Intel Corporation Q87 Express LPC Controller (rev 04)
        Subsystem: Lenovo Q87 Express LPC Controller
        Kernel driver in use: lpc_ich
        Kernel modules: lpc_ich
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 04)
        Subsystem: Lenovo 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode]
        Kernel driver in use: ahci
        Kernel modules: ahci
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 04)
        Subsystem: Lenovo 8 Series/C220 Series Chipset Family SMBus Controller
        Kernel driver in use: i801_smbus
        Kernel modules: i2c_i801
01:00.0 Ethernet controller: Chelsio Communications Inc T520-BT Unified Wire Ethernet Controller
        Subsystem: Chelsio Communications Inc T520-BT Unified Wire Ethernet Controller
        Kernel driver in use: vfio-pci
        Kernel modules: cxgb4
01:00.1 Ethernet controller: Chelsio Communications Inc T520-BT Unified Wire Ethernet Controller
        Subsystem: Chelsio Communications Inc T520-BT Unified Wire Ethernet Controller
        Kernel driver in use: vfio-pci
        Kernel modules: cxgb4
01:00.2 Ethernet controller: Chelsio Communications Inc T520-BT Unified Wire Ethernet Controller
        Subsystem: Chelsio Communications Inc T520-BT Unified Wire Ethernet Controller
        Kernel driver in use: vfio-pci
        Kernel modules: cxgb4
01:00.3 Ethernet controller: Chelsio Communications Inc T520-BT Unified Wire Ethernet Controller
        Subsystem: Chelsio Communications Inc T520-BT Unified Wire Ethernet Controller
        Kernel driver in use: vfio-pci
        Kernel modules: cxgb4
01:00.4 Ethernet controller: Chelsio Communications Inc T520-BT Unified Wire Ethernet Controller
        Subsystem: Chelsio Communications Inc T520-BT Unified Wire Ethernet Controller
        Kernel driver in use: vfio-pci
        Kernel modules: cxgb4
01:00.5 SCSI storage controller: Chelsio Communications Inc T520-BT Unified Wire Storage Controller
        Subsystem: Chelsio Communications Inc T520-BT Unified Wire Storage Controller
        Kernel driver in use: vfio-pci
01:00.6 Fibre Channel: Chelsio Communications Inc T520-BT Unified Wire Storage Controller
        Subsystem: Chelsio Communications Inc T520-BT Unified Wire Storage Controller
        Kernel driver in use: vfio-pci
        Kernel modules: csiostor
03:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
        Subsystem: Intel Corporation Gigabit CT Desktop Adapter
        Kernel driver in use: e1000e
        Kernel modules: e1000e
04:00.0 PCI bridge: Integrated Technology Express, Inc. IT8893E PCIe to PCI Bridge (rev 41)
06:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter I350-T4
        Kernel driver in use: vfio-pci
        Kernel modules: igb
06:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter I350-T4
        Kernel driver in use: vfio-pci
        Kernel modules: igb
06:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter I350-T4
        Kernel driver in use: vfio-pci
        Kernel modules: igb
06:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter I350-T4
        Kernel driver in use: vfio-pci
        Kernel modules: igb
 
Splise said:
It looks like Proxmox is not recognizing the second port of the NIC from what I can tell.
Click to expand...
but this does not matter when you want to pass it through

i am slightly confused because the nic seems to have 5 ethernet devices, in addition to a scsi and fibre channel device
how do those relate to the two ports? ( maybe dmesg output is helpful here)

also pci listing and dmesg(if existing) from the guest would be interesting
 
dcsapak said:
but this does not matter when you want to pass it through

Fair enough

i am slightly confused because the nic seems to have 5 ethernet devices, in addition to a scsi and fibre channel device
how do those relate to the two ports? ( maybe dmesg output is helpful here)

I am also confused by it. As you mentioned, I have identified the scsi and fibre channel ports, but am unsure on the others. As it's a dual port NIC, it doesn't make a lot of sense to me. Here are the errors I have found so far via dmesg on Proxmox:

Code: 
dmesg | grep cxgb4
[    1.200418] cxgb4 0000:01:00.4: enabling device (0000 -> 0002)
[    1.209709] csiostor 0000:01:00.6: Direct firmware load for cxgb4/t5fw.bin failed with error -2
[    1.210539] csiostor 0000:01:00.6: could not find firmware image cxgb4/t5fw.bin, err: -125
[    1.778580] csiostor 0000:01:00.6: Direct firmware load for cxgb4/t5-config.txt failed with error -2
[    1.778586] csiostor 0000:01:00.6: could not find config file cxgb4/t5-config.txt, err: 0
[   13.264333] cxgb4 0000:01:00.4: command 0x4 in mailbox 4 timed out
[   13.265552] cxgb4 0000:01:00.4: encountered fatal error, adapter stopped
[   13.276333] cxgb4 0000:01:00.4: could not connect to FW, error -71
[   13.277739] cxgb4: probe of 0000:01:00.4 failed with error -71

Code: 
csiostor: Chelsio FCoE driver 1.0.0-ko
[    1.198913] Chelsio T4/T5/T6 Network Driver - version 2.0.0-ko
[    1.199630] csiostor 0000:01:00.6: Firmware revision: 1.24.11.0
[    1.200418] cxgb4 0000:01:00.4: enabling device (0000 -> 0002)
[    1.205728] csiostor 0000:01:00.6: PF: 6, Coming up as MASTER, HW state: Initializing
[    1.209709] csiostor 0000:01:00.6: Direct firmware load for cxgb4/t5fw.bin failed with error -2
[    1.210539] csiostor 0000:01:00.6: could not find firmware image cxgb4/t5fw.bin, err: -125
[    1.211426] dca service started, version 1.12.1
[    1.777451] ACPI Warning: SystemIO range 0x0000000000001828-0x000000000000182F conflicts with OpRegion 0x0000000000001800-0x000000000000187F (\PMIO)
[    1.777455] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[    1.777455]  (20190703/utaddress-213)
[    1.777457] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver
[    1.778509] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    1.778580] csiostor 0000:01:00.6: Direct firmware load for cxgb4/t5-config.txt failed with error -2
[    1.778586] csiostor 0000:01:00.6: could not find config file cxgb4/t5-config.txt, err: 0

also pci listing and dmesg(if existing) from the guest would be interesting
I have been testing different configurations, but do not have a working VM at this point. I will post the requested info if I am able to gather it.
Click to expand...
 
Splise said:
And this as well:
Code: 
0.277068] pci_bus 0000:00: max bus depth: 2 pci_try_num: 3
[    0.277078] pci 0000:01:00.0: BAR 9: no space for [mem size 0x00080000 64bit]
[    0.277080] pci 0000:01:00.0: BAR 9: failed to assign [mem size 0x00080000 64bit]
[    0.277082] pci 0000:01:00.1: BAR 9: no space for [mem size 0x00080000 64bit]
[    0.277084] pci 0000:01:00.1: BAR 9: failed to assign [mem size 0x00080000 64bit]
[    0.277086] pci 0000:01:00.2: BAR 9: no space for [mem size 0x00080000 64bit]
[    0.277088] pci 0000:01:00.2: BAR 9: failed to assign [mem size 0x00080000 64bit]
[    0.277090] pci 0000:01:00.3: BAR 9: no space for [mem size 0x00080000 64bit]
[    0.277092] pci 0000:01:00.3: BAR 9: failed to assign [mem size 0x00080000 64bit]
[    0.277095] pci 0000:01:00.0: BAR 11: assigned [mem 0xf688e000-0xf68adfff 64bit]
[    0.277099] pci 0000:01:00.1: BAR 11: assigned [mem 0xf68ae000-0xf68cdfff 64bit]
[    0.277103] pci 0000:01:00.2: BAR 11: assigned [mem 0xf68ce000-0xf68edfff 64bit]
[    0.277106] pci 0000:01:00.3: BAR 11: no space for [mem size 0x00020000 64bit]
[    0.277108] pci 0000:01:00.3: BAR 11: trying firmware assignment [mem 0x00001000-0x00020fff 64bit]
[    0.277110] pci 0000:01:00.3: BAR 11: [mem 0x00001000-0x00020fff 64bit] conflicts with System RAM [mem 0x00001000-0x00057fff]
[    0.277112] pci 0000:01:00.3: BAR 11: failed to assign [mem size 0x00020000 64bit]
[    0.277114] pci 0000:01:00.0: BAR 7: assigned [mem 0xf68ee000-0xf68fdfff 64bit]
[    0.277117] pci 0000:01:00.1: BAR 7: no space for [mem size 0x00010000 64bit]
[    0.277118] pci 0000:01:00.1: BAR 7: failed to assign [mem size 0x00010000 64bit]
[    0.277121] pci 0000:01:00.2: BAR 7: no space for [mem size 0x00010000 64bit]
[    0.277122] pci 0000:01:00.2: BAR 7: failed to assign [mem size 0x00010000 64bit]
[    0.277124] pci 0000:01:00.3: BAR 7: no space for [mem size 0x00010000 64bit]
[    0.277126] pci 0000:01:00.3: BAR 7: failed to assign [mem size 0x00010000 64bit]
Click to expand...
 
are those all from the host? if yes the failed driver loading is not relevant if you purely want to passthrough

the BAR failures are not so good though... did you enable 'above 4-g' (or something similar) decoding in the bios of your host?
or are those from the guest ?
 
All of the logs are form the Host at this point. Sorry, just trying to provide as much info as possible. I will check the 4-g, but I don't remember seeing that option.
 
Splise said:
All of the logs are form the Host at this point. Sorry, just trying to provide as much info as possible. I will check the 4-g, but I don't remember seeing that option.
Click to expand...
Don't see anything for 4-g. But I cam able to see the driver health status for the Chelsio NIC in the BIOS, and both ports show up, and the device is "healthy"
 
This is really unfortunate. I was hoping to create a guide for my peers on how to configure Proxmox for use with a PA VM firewall. I have spent weeks trying to get this working and read hundreds of articles and forums, but I just can't afford to spend this kind of time just to get a VM spun up.

Thank you for your help Dominik, at least you responded!! I thought the community would be a little more involved, even if my questions are not up to the expert's standards.

Proxmox/ZFS looks very promising. However, there appear to be a number of issues when attempting to pass through PCIe devices, and a number of potential resolutions. When there are hundreds of combinations that may work, it's difficult to find the correct sequence without tainting the prior results (and I am recording them). I have tried moving the cards around, tweaking the BIOS settings, attempting close to a hundred variations of the config on the host and VM. Even adding something as simple as "intel_iommu=on" can be an issue, as there are at least 4 different places it can be specified, but that doesn't mean it should be.

There's lots of information regarding GPU passthrough, but very little on NIC passthrough.

Anyway, if I find something out I will post it. But I think I will have to move to another hypervisor at this point, which is really disappointing.
 
Splise said:
Thank you for your help Dominik, at least you responded!! I thought the community would be a little more involved, even if my questions are not up to the expert's standards.
Click to expand...
it has probably more to do with the choice of hardware and nobody else trying that card with passthrough...

Splise said:
There's lots of information regarding GPU passthrough, but very little on NIC passthrough.
Click to expand...
not much difference, but most of the time when there are problems, the reason is bad firmware/bios implementation on the card/mainboard or bad drivers...
 
You must log in or register to reply here.
Top Bottom