[SOLVED] Network interface/driver crashing

[tfcbx]

I have a 10 GB dual port Intel ethernet card and a single 2.5 GB Realtek interface (built-in on the motherboard).

# ethtool -i  enp3s0f0
driver: ixgbe
version: 6.2.16-15-pve
firmware-version: 0x80000760, 1.1568.0
expansion-rom-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

# ethtool -i  enp5s0
driver: r8169
version: 6.2.16-15-pve
firmware-version: rtl8125b-2_0.0.2 07/13/20
expansion-rom-version:
bus-info: 0000:05:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

Passthrough is enabled and the ports on the ethernet card have individual iommu groups.

# dmesg | grep -e DMAR -e IOMMU
[    0.839553] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.845682] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    0.846096] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

# pvesh get /nodes/pve/hardware/pci --pci-class-blacklist ""

├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────┼──────┼──────────────────┼────────────────────────────────────────────┼──────────────────┼────────────────────────────────────┼────────────────────────────────────┤
│ 0x020000 │ 0x1563 │ 0000:03:00.0 │         19 │ 0x8086 │ Ethernet Controller X550                                │      │ 0x0022           │ Ethernet Converged Network Adapter X550-T2 │ 0x8086           │ Intel Corporation                  │ Intel Corporation                  │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────┼──────┼──────────────────┼────────────────────────────────────────────┼──────────────────┼────────────────────────────────────┼────────────────────────────────────┤
│ 0x020000 │ 0x1563 │ 0000:03:00.1 │         20 │ 0x8086 │ Ethernet Controller X550                                │      │ 0x0022           │ Ethernet Converged Network Adapter X550-T2 │ 0x8086           │ Intel Corporation                  │ Intel Corporation                  │
├──────────┼────────┼──────────────┼────────────┼────────┼─────────────────────────────────────────────────────────┼──────┼──────────────────┼────────────────────────────────────────────┼──────────────────┼────────────────────────────────────┼────────────────────────────────────┤
│ 0x020000 │ 0x3000 │ 0000:05:00.0 │         21 │ 0x10ec │ Killer E3000 2.5GbE Controller                          │      │ 0x3000           │                                            │ 0x1849           │ ASRock Incorporation               │ Realtek Semiconductor Co., Ltd.

I've added the two devices to the VM (happens to be openwrt).

# cat 100.conf
balloon: 512
bios: ovmf
boot: order=scsi0
cores: 2
cpu: x86-64-v2-AES
efidisk0: local:100/vm-100-disk-0.qcow2,efitype=4m,pre-enrolled-keys=1,size=528K
hostpci0: 0000:05:00
hostpci1: 0000:03:00
ide2: none,media=cdrom
machine: q35
memory: 2048
meta: creation-qemu=8.0.2,ctime=1697682281
name: openwrt23
numa: 0
ostype: l26
scsi0: local:100/vm-100-disk-1.raw,iothread=1,size=126123520
scsihw: virtio-scsi-single
smbios1: uuid=b53cf649-6228-473c-8236-7226b85c6ab4
sockets: 1
vmgenid: 05e6599f-1de2-4d75-84de-ff6015a834c1

Every time I attempt to start the VM the interfaces "crash" on the host. So they are present on a fresh system boot and then don't appear after attempting to start the VM.

# ip link
3: enp3s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether macaddr brd ff:ff:ff:ff:ff:ff
4: enp3s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether macaddr brd ff:ff:ff:ff:ff:ff

At this point I'm pretty much out of debugging ideas. Any thought on how to proceed? Should this work assigning on one interface from a dual port card to a VM?

Thanks for the help!

 

[tfcbx]

Hum, it seems as though only the full NIC (not sure of the correct term) can be passed through. So even though it appears that I'm picking the "0000.03.00.0" interface

I'm really picking the entire card based on the conf file.

$ sudo cat 100.conf
balloon: 512
bios: ovmf
boot: order=scsi0
cores: 2
cpu: x86-64-v2-AES
efidisk0: local:100/vm-100-disk-0.qcow2,efitype=4m,pre-enrolled-keys=1,size=528K
hostpci0: 0000:03:00
ide2: none,media=cdrom
machine: q35
memory: 2048
meta: creation-qemu=8.0.2,ctime=1697682281
name: openwrt23
numa: 0
ostype: l26
scsi0: local:100/vm-100-disk-1.raw,iothread=1,size=126123520
scsihw: virtio-scsi-single
smbios1: uuid=b53cf649-6228-473c-8236-7226b85c6ab4
sockets: 1
vmgenid: 05e6599f-1de2-4d75-84de-ff6015a834c1

The VM boots fine in this case.

 

[Neobin]

Try without setting: "All Functions" on the Intel NIC.

Also: Do you use ACS-override and what NIC is your PVE actually assigned to?

 

[tfcbx]

@Neobin Thanks so much for your response!

"All Functions" appears to have been the culprit. A bad case of random clicking.

The final configuration has enp3s0f1 (0000:03:00.1) for the vmbr0 interface, enp3s0f0 (0000:03:00.0) for the LAN, and enp5s0 (0000:05:00.0) for the WAN.

I'm not familiar with ACS-override.