Passthrough issue with 10Gtek 82576-2T-X1

O

onixid

Member
May 16, 2020
13
0
21
44
Hi,
I just installed a 10Gtek 82576-2T-X1 on a HP Microserver Gen8, Proxmox (version 7.2-3) is showing the card correctly.
I followed the passthrough guide to configure the system and the current configuration is as follows:


Code: 
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller [8086:0158] (rev 09)
IOMMU Group 10 00:1f.0 ISA bridge [0601]: Intel Corporation C204 Chipset LPC Controller [8086:1c54] (rev 05)
IOMMU Group 10 00:1f.2 SATA controller [0106]: Intel Corporation 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller [8086:1c02] (rev 05)
IOMMU Group 11 07:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01)
IOMMU Group 12 07:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01)
IOMMU Group 13 03:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe [14e4:165f]
IOMMU Group 14 03:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe [14e4:165f]
IOMMU Group 15 04:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014] (rev 03)
IOMMU Group 16 01:00.0 System peripheral [0880]: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support [103c:3306] (rev 05)
IOMMU Group 17 01:00.1 VGA compatible controller [0300]: Matrox Electronics Systems Ltd. MGA G200EH [102b:0533]
IOMMU Group 18 01:00.2 System peripheral [0880]: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging [103c:3307] (rev 05)
IOMMU Group 19 01:00.4 USB controller [0c03]: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller [103c:3300] (rev 02)
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)
IOMMU Group 2 00:06.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:015d] (rev 09)
IOMMU Group 3 00:1a.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 [8086:1c2d] (rev 05)
IOMMU Group 4 00:1c.0 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 [8086:1c10] (rev b5)
IOMMU Group 5 00:1c.4 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 [8086:1c18] (rev b5)
IOMMU Group 6 00:1c.6 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 [8086:1c1c] (rev b5)
IOMMU Group 7 00:1c.7 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 [8086:1c1e] (rev b5)
IOMMU Group 8 00:1d.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 [8086:1c26] (rev 05)
IOMMU Group 9 00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev a5)

cat /etc/modules
Code: 
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

cat /etc/modprobe.d/igb.conf
Code: 
options igb max_vfs=4

I also added the following in /etc/default/grub:
Code: 
# If you change this file, run 'update-grub' afterwards to update
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction,id:8086:10c9"

The VM is configured as follows:
Code: 
boot: order=scsi0;net0
cores: 2
machine: q35
memory: 8192
name: OMV
net0: virtio=7A:5C:C0:93:8A:F7,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: local:100/vm-100-disk-0.qcow2,size=21G
scsihw: virtio-scsi-pci
smbios1: uuid=be80ed6c-d528-45e0-8bdf-8f3b96d158f5
sockets: 2
startup: order=2
usb0: host=1058:25a2,usb3=1
virtio1: /dev/disk/by-id/ata-WDC_WD120EDBZ-11B1HA0_D5H9HT8F,size=11176G,aio=native,serial=D5H9HT8F
virtio2: /dev/disk/by-id/ata-WDC_WD120EDAZ-11F3RA0_5PK2X52B,size=11176G,aio=native,serial=5PK2X52B
vmgenid: 2ac6e010-0881-4df0-8986-21bdc43fd3a6
hostpci0: 0000:07:00.0,pcie=1,driver=vfio
hostpci1: 0000:07:00.1,pcie=1,driver=vfio

The result of all this is that the VM doesn't see the 2 network interfaces.
I tried tweaking with the hostpci parameters, but most of the time changing/removing anything would result in an error that would prevent the VM from starting.
Can anyone please help me to untackle this situation?
 
Last edited:
remove the 'driver=vfio' part of the hostpci, this is unsupported syntax and thus the lines cannot be parsed
 
If I remove that it won't start completely and it throws this error:

Code: 
kvm: -device vfio-pci,host=0000:07:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0: vfio 0000:07:00.0: failed to setup container for group 11: Failed to set iommu for container: Operation not permitted
TASK ERROR: start failed: QEMU exited with code 1
 
can you post the complete output of the `dmesg` cli command?
 
onixid said:
If I remove that it won't start completely and it throws this error:

Code: 
kvm: -device vfio-pci,host=0000:07:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0: vfio 0000:07:00.0: failed to setup container for group 11: Failed to set iommu for container: Operation not permitted
TASK ERROR: start failed: QEMU exited with code 1
Click to expand...
This could be meaningful:

Code: 
[15265.270900] perf: interrupt took too long (2627 > 2500), lowering kernel.perf_event_max_sample_rate to 76000
[22493.329383] perf: interrupt took too long (3298 > 3283), lowering kernel.perf_event_max_sample_rate to 60500
[32939.353103] perf: interrupt took too long (4135 > 4122), lowering kernel.perf_event_max_sample_rate to 48250
[44388.099489] hrtimer: interrupt took 20547 ns
[45105.015569] perf: interrupt took too long (5172 > 5168), lowering kernel.perf_event_max_sample_rate to 38500
[54699.605553]  sda: sda1 sda9
[54699.653309]  sdb: sdb1 sdb9
[54700.141588] fwbr100i0: port 2(tap100i0) entered disabled state
[54700.165288] fwbr100i0: port 1(fwln100i0) entered disabled state
[54700.165931] vmbr0: port 4(fwpr100p0) entered disabled state
[54700.166100] device fwln100i0 left promiscuous mode
[54700.166103] fwbr100i0: port 1(fwln100i0) entered disabled state
[54700.179864] device fwpr100p0 left promiscuous mode
[54700.179873] vmbr0: port 4(fwpr100p0) entered disabled state
[54700.509262] usb 5-2: reset SuperSpeed USB device number 2 using xhci_hcd
[54700.531500] usb-storage 5-2:1.0: USB Mass Storage device detected
[54700.532012] scsi host6: usb-storage 5-2:1.0
[54701.564021] scsi 6:0:0:0: Direct-Access     WD       Elements 25A2    1026 PQ: 0 ANSI: 6
[54701.564447] sd 6:0:0:0: Attached scsi generic sg2 type 0
[54701.564710] sd 6:0:0:0: [sdc] Very big device. Trying to use READ CAPACITY(16).
[54701.564862] sd 6:0:0:0: [sdc] 7813969920 512-byte logical blocks: (4.00 TB/3.64 TiB)
[54701.564869] sd 6:0:0:0: [sdc] 4096-byte physical blocks
[54701.566087] sd 6:0:0:0: [sdc] Write Protect is off
[54701.566094] sd 6:0:0:0: [sdc] Mode Sense: 47 00 10 08
[54701.567724] sd 6:0:0:0: [sdc] No Caching mode page found
[54701.567788] sd 6:0:0:0: [sdc] Assuming drive cache: write through
[54701.667947]  sdc: sdc1
[54701.670844] sd 6:0:0:0: [sdc] Attached SCSI disk
[54707.581270] device tap100i0 entered promiscuous mode
[54707.659402] vmbr0: port 4(fwpr100p0) entered blocking state
[54707.659409] vmbr0: port 4(fwpr100p0) entered disabled state
[54707.677326] device fwpr100p0 entered promiscuous mode
[54707.677394] vmbr0: port 4(fwpr100p0) entered blocking state
[54707.677397] vmbr0: port 4(fwpr100p0) entered forwarding state
[54707.687389] fwbr100i0: port 1(fwln100i0) entered blocking state
[54707.687396] fwbr100i0: port 1(fwln100i0) entered disabled state
[54707.699753] device fwln100i0 entered promiscuous mode
[54707.699851] fwbr100i0: port 1(fwln100i0) entered blocking state
[54707.699855] fwbr100i0: port 1(fwln100i0) entered forwarding state
[54707.706293] fwbr100i0: port 2(tap100i0) entered blocking state
[54707.706299] fwbr100i0: port 2(tap100i0) entered disabled state
[54707.706515] fwbr100i0: port 2(tap100i0) entered blocking state
[54707.706519] fwbr100i0: port 2(tap100i0) entered forwarding state
[54707.772463] vfio-pci 0000:07:00.0: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.
[54708.022959] fwbr100i0: port 2(tap100i0) entered disabled state
[54708.069036] fwbr100i0: port 1(fwln100i0) entered disabled state
[54708.069244] vmbr0: port 4(fwpr100p0) entered disabled state
[54708.069495] device fwln100i0 left promiscuous mode
[54708.069500] fwbr100i0: port 1(fwln100i0) entered disabled state
[54708.095946] device fwpr100p0 left promiscuous mode
[54708.095956] vmbr0: port 4(fwpr100p0) entered disabled state
[54725.412041] device tap100i0 entered promiscuous mode
[54725.535078] vmbr0: port 4(fwpr100p0) entered blocking state
[54725.535091] vmbr0: port 4(fwpr100p0) entered disabled state
[54725.535440] device fwpr100p0 entered promiscuous mode
[54725.535564] vmbr0: port 4(fwpr100p0) entered blocking state
[54725.535573] vmbr0: port 4(fwpr100p0) entered forwarding state
[54725.560104] fwbr100i0: port 1(fwln100i0) entered blocking state
[54725.560118] fwbr100i0: port 1(fwln100i0) entered disabled state
[54725.560404] device fwln100i0 entered promiscuous mode
[54725.560668] fwbr100i0: port 1(fwln100i0) entered blocking state
[54725.560677] fwbr100i0: port 1(fwln100i0) entered forwarding state
[54725.580751] fwbr100i0: port 2(tap100i0) entered blocking state
[54725.580764] fwbr100i0: port 2(tap100i0) entered disabled state
[54725.590169] fwbr100i0: port 2(tap100i0) entered blocking state
[54725.590183] fwbr100i0: port 2(tap100i0) entered forwarding state
[54744.876972] usb 5-2: reset SuperSpeed USB device number 2 using xhci_hcd
 
onixid said:
[54707.772463] vfio-pci 0000:07:00.0: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.
Click to expand...
yes it's a bios issue, maybe check for a bios upgrade
 
You must log in or register to reply here.
Top Bottom