felipemelloni

Member
Aug 5, 2014
4
0
21
Hello everyone!
I've been using Proxmox for some years and I'm very satisfied with the platform's features and stability.
New features are constantly being incorporated into the system, making it one of the best hypervisors on the market today.
This is nothing new. But now I'm having a problem.
I've been trying for a few days to perform a configuration that at first glance seemed simple, but it's been taking me a lot of time.
Thank you in advance for anyone who can help with the following question.

I'm trying to Passthrough a two-port HBA card (Emulex 01CV842 16Gb FC Dual-port HBA - LPe31002-M6-L - Emulex-Gen6-DOC) for a Windows 2016 Server. Proxmox (Version 7.1-7) and this card are installed in a Lenovo ThinkSystem SR650 Server.
I performed all those steps in the documentation that, in my opinion, make sense for this configuration. (PVE-Wiki-PCI(e))

When checking the status of the devices via Device Manager (Windows), one of the devices is shown with a yellow exclamation, also displaying the following message: "This device cannot start. (Code 10)".

1644887990250.png

With that, I have communication with the storage only through a single port (no redundancy).
This is reflected in the device administration software, where only one of the ports is displayed.

PCI device configuration via Proxmox web interface:

1644888048363.png
1644888063794.png

All possible drivers and firmware have already been updated.

I have also verified that it is not a problem with the device. I tested with another identical device installed in another slot. Same problem.

As you can see, both Ids are part of the same IOMM Group. I understand then that no configuration is necessary for segregation of these groups, and both (0000:2f:00.0 and 0000:2f:00.1) can be forwarded to the VM without major problems. (I hope I'm right)

Some additional information that may be important:

Code:
cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Code:
cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
GRUB_CMDLINE_LINUX=""
...

Note: I have also performed tests with this configuration:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1"


Code:
dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
[    0.011382] ACPI: DMAR 0x00000000931C0128 000280 (v01 LENOVO THINKSYS 00000001 INTL 20091013)
[    0.011438] ACPI: Reserving DMAR table memory at [mem 0x931c0128-0x931c03a7]
[    0.486376] DMAR: IOMMU enabled
...

Code:
cat /etc/modprobe.d/pve-blacklist.conf
# This file contains a list of modules which are not supported by Proxmox VE

# nidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
blacklist lpfc

Code:
cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=10df:e300,10df:e300

Code:
lspci -nn | grep Lpe
2f:00.0 Fibre Channel [0c04]: Emulex Corporation LPe31000/LPe32000 Series 16Gb/32Gb Fibre Channel Adapter [10df:e300] (rev 01)
2f:00.1 Fibre Channel [0c04]: Emulex Corporation LPe31000/LPe32000 Series 16Gb/32Gb Fibre Channel Adapter [10df:e300] (rev 01)

Code:
cat /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

Code:
dmesg | grep 'remapping'
[    1.095175] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    1.097018] DMAR-IR: Enabled IRQ remapping in x2apic mode

Code:
cat /etc/pve/nodes/pve01/qemu-server/100.conf
bios: ovmf
boot: order=scsi0;ide2;net0;ide0
cores: 8
cpu: host
efidisk0: local-lvm:vm-100-disk-1,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:2f:00,pcie=1
ide0: local:iso/virtio-win-0.1.215.iso,media=cdrom,size=528308K
ide2: local:iso/SW_DVD9_Win_Server_STD_CORE_2016_64Bit_English_-4_DC_STD_MLF_X21-70526.ISO,media=cdrom
machine: pc-q35-6.1
memory: 8192
meta: creation-qemu=6.1.0,ctime=1644793789
name: VM-TEST-DB
net0: virtio=B2:99:8F:BB:14:30,bridge=vmbr0,firewall=1
numa: 1
ostype: win10
scsi0: local-lvm:vm-100-disk-0,size=50G
scsihw: virtio-scsi-pci
smbios1: uuid=b4b0a989-1ae7-403a-897c-c29f8d2ea889
sockets: 1
vcpus: 8

In addition to the code 10 error on Windows, after performing the Pci passthrough, some errors started to be displayed in the Proxmox Syslog:

Feb 14 22:33:42 pve01 QEMU[8345]: kvm: vfio 0000:2f:00.0: Failed to set up TRIGGER eventfd signaling for interrupt MSIX-489: VFIO_DEVICE_SET_IRQS failure: Invalid argument
Feb 14 22:33:42 pve01 QEMU[8345]: kvm: vfio: Error: event_notifier_init failed
Feb 14 22:33:42 pve01 QEMU[8345]: kvm: vfio 0000:2f:00.0: Failed to set up TRIGGER eventfd signaling for interrupt MSIX-490: VFIO_DEVICE_SET_IRQS failure: Invalid argument
Feb 14 22:33:42 pve01 QEMU[8345]: kvm: vfio: Error: event_notifier_init failed
...
Feb 14 22:33:45 pve01 QEMU[8345]: kvm: vfio 0000:2f:00.1: Failed to set up TRIGGER eventfd signaling for interrupt MSIX-1: VFIO_DEVICE_SET_IRQS failure: Invalid argument
Feb 14 22:33:45 pve01 QEMU[8345]: kvm: vfio: Error: event_notifier_init failed
Feb 14 22:33:45 pve01 QEMU[8345]: kvm: vfio 0000:2f:00.1: Failed to set up TRIGGER eventfd signaling for interrupt MSIX-2: VFIO_DEVICE_SET_IRQS failure: Invalid argument
Feb 14 22:33:45 pve01 QEMU[8345]: kvm: vfio: Error: event_notifier_init failed
...
Feb 14 22:33:46 pve01 QEMU[8345]: kvm: virtio-scsi: Failed to set guest notifiers (-24), ensure -accel kvm is set.
Feb 14 22:33:46 pve01 QEMU[8345]: kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).
Feb 14 22:33:47 pve01 QEMU[8345]: kvm: Error binding guest notifier: 24
Feb 14 22:33:47 pve01 QEMU[8345]: kvm: unable to start vhost net: 24: falling back on userspace virtio
Feb 14 22:33:47 pve01 QEMU[8345]: kvm: virtio_bus_set_host_notifier: unable to init event notifier: Too many open files (-24)
Feb 14 22:33:47 pve01 QEMU[8345]: kvm: virtio_bus_start_ioeventfd: failed. Fallback to userspace (slower).
Feb 14 22:33:56 pve01 pvestatd[1473]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - got timeout
Feb 14 22:33:56 pve01 pvestatd[1473]: status update time (6.244 seconds)
Feb 14 22:34:07 pve01 pvestatd[1473]: VM 100 qmp command failed - VM 100 qmp command 'query-proxmox-support' failed - unable to connect to VM 100 qmp socket - timeout after 31 retries
Feb 14 22:34:07 pve01 pvestatd[1473]: status update time (6.210 seconds)
...

Sorry for the long post, but I tried to add all the necessary information.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!