[SOLVED] PCI Passthrough NIC stays Down

[defmsy]

Hi,

Context:​

I have a bare metal server hosted by OVH.

This server has Proxmox VE 8.1.10 installed.

This server has 4 NICs.

# lspci -nnk | grep -A 5 "Ethernet"
03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T [8086:15ad]
    DeviceName:  Intel X557-AT2 Ethernet #1
    Subsystem: Super Micro Computer Inc Ethernet Connection X552/X557-AT 10GBASE-T [15d9:15ad]
    Kernel driver in use: ixgbe
    Kernel modules: ixgbe
03:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T [8086:15ad]
    DeviceName:  Intel X557-AT2 Ethernet #2
    Subsystem: Super Micro Computer Inc Ethernet Connection X552/X557-AT 10GBASE-T [15d9:15ad]
    Kernel driver in use: vfio-pci
    Kernel modules: ixgbe
07:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
    DeviceName:  Intel i350 Ethernet #1
    Subsystem: Super Micro Computer Inc I350 Gigabit Network Connection [15d9:1521]
    Kernel driver in use: igb
    Kernel modules: igb
07:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
    DeviceName:  Intel i350 Ethernet #2
    Subsystem: Super Micro Computer Inc I350 Gigabit Network Connection [15d9:1521]
    Kernel driver in use: igb
    Kernel modules: igb

I have two ip addresses for this server. I will reference them as <IP_MGNT> and <IP_PROXY>.

<IP_MGNT> is my management ip address directly connected to my server and will be used to manage my Proxmox server.

<IP_PROXY> is the ip address that I want to bind to an HA Proxy VM to expose all my services to the internet through reverse-proxy.

I have followed instructions to configure IOMMU on my server.

What I did until now:​

I modified /etc/default/grub to enable iommu by adding intel_iommu=on

GRUB_CMDLINE_LINUX="nomodeset iommu=pt console=tty0 console=ttyS1,115200n8 intel_iommu=on"

I execute update-grub

I modified /etc/modules with the following content

# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

I rebooted my server

I ran the command dmesg | grep -e DMAR -e IOMMU to check if IOMMU was enabled

[    0.147433] DMAR: IOMMU enabled

I checked and my NICs have different iommu groups.

I modified /etc/modprobe.d/vfio.conf with the following content

options vfio-pci ids=03:00.1

I modified /etc/network/interfaces to remove config related to this NIC

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
    address <IP_MGNT>/24
    gateway <GATEWAY>
#Management

#iface eno2 inet manual <= removed

iface eno3 inet manual

iface eno4 inet manual

auto vmbr0
iface vmbr0 inet static
    address 10.10.10.1/24
    bridge-ports none
    bridge-stp off
    bridge-fd 0
#Default

I rebooted my server

I added the NIC to the VM through the UI

In /etc/pve/qemu-server/<VM_ID>.conf I have the line:

hostpci0: 0000:03:00.1

In my VM (Ubuntu 22.04), I ran ip link show

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether bc:24:11:74:fa:38 brd ff:ff:ff:ff:ff:ff
    altname enp0s18
3: ens16: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 0c:c4:7a:7b:5c:63 brd ff:ff:ff:ff:ff:ff
    altname enp0s16

I modified my netplan configuration, applied and reboot my VM

# This is the network config written by 'subiquity'
network:
  renderer: networkd
  ethernets:
    ens18:
      addresses:
        - 10.10.10.2/24
    ens16:
      addresses:
        - <IP_PROXY>/24
      nameservers:
        addresses:
          - 4.2.2.2
          - 8.8.8.8
      routes:
        - to: default
          via: <GATEWAY>
  version: 2

Result:​

I can see on my host that Kernel driver in use: vfio-pci but my network interface stays down in the VM.
What did I made wrong?

 

[celemine1gig]

...
03:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T [8086:15ad]
...

I modified /etc/modprobe.d/vfio.conf with the following content

options vfio-pci ids= 03:00.1

...

The entry "options vfio-pci ids=03:00.1", that you created, is simply wrong. You need the actual PCI-ID (see here https://admin.pci-ids.ucw.cz/read/PC/8086) and NOT the bus number and function here.
In your case, this would be "options vfio-pci ids=8086:15ad". See your own lspci quote.

 

[defmsy]

Hi @celemine1gig ,

I fixed /etc/modprobe.d/vfio.conf:

options vfio-pci ids=8086:15ad

then I ran

sudo update-initramfs -u

and I rebooted my server

Unfortunately, nothing changed, the NIC is still down on the VM.

 

[celemine1gig]

What does "dmesg|grep -i vfio" on the proxmox host say? And is the device listed inside the VM (lspci), or is it completely missing?

 

[defmsy]

On the Host:

dmesg | grep -i vfio
[   14.917682] VFIO - User Level meta-driver version: 0.3
[   14.996170] vfio_pci: add [8086:15ad[ffffffff:ffffffff]] class 0x000000/00000000
[ 1092.737737] vfio-pci 0000:03:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 1096.180501] vfio-pci 0000:03:00.1: vfio_bar_restore: reset recovery - restoring BARs
[ 2164.210737] vfio-pci 0000:03:00.0: vfio_bar_restore: reset recovery - restoring BARs
[ 2167.655527] vfio-pci 0000:03:00.1: vfio_bar_restore: reset recovery - restoring BARs

On the VM:

lspci

00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)


00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]


00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]


00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)


00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)


00:02.0 VGA compatible controller: Device 1234:1111 (rev 02)


00:03.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon


00:05.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge


00:10.0 Ethernet controller: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T


00:10.1 Ethernet controller: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T


00:12.0 Ethernet controller: Red Hat, Inc. Virtio network device


00:1e.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge


00:1f.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge


01:01.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI

 

[celemine1gig]

Looking good, according to the posted output. Only thing I see is, that you originally only wanted to pass-through one Ethernet interface, right? Just function 1, but not function 0?
According to your output, both have been passed through.

Did you by any chance enable the "all functions" option in the Proxmox GUI?

 

[defmsy]

It was enabled. I disabled it but the NIC is still down.

 

[celemine1gig]

Try to pass-through both, function 0 and 1, after having unloaded the driver module from the Proxmox host. Let's see if the issue is the fact, that you are trying to pass through a device, while the host's driver is still active and is managing the device.

 

[defmsy]

Hi @celemine1gig,

Thanks for your help.

It's the network card which was the issue.

I finally abandoned this solution and switched to using a single card.

If it can help someone else, here is my alternative solution.

/etc/network/interfaces

auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
    address <host_ip_address>
    gateway <gateway>
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0
#Public

auto vmbr1
iface vmbr1 inet static
    address 10.10.10.1/24
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    post-up echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
    post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eno1 -j MASQUERADE
#Private

You need to add the bridge to the VM. In my case, the public IP had a virtual MAC address attached that I had to configure when adding the bridge to the VM.

Then, I configured my VM (Ubuntu 22.04) like that:

network:
  renderer: networkd
  ethernets:
    ens18:
      addresses:
        - <secondary_public_ip>/32
      nameservers:
        addresses:
          - 8.8.4.4
          - 8.8.8.8
      routes:
        - to: default
          via: <host_gateway>
          on-link: true
  version: 2