[SOLVED] VM Failed to Start: Error Cannot Bind 0000:04:00.0

[Nollimox]

Is there a way to force a bind of a pci device? Does that failure suggest at least it has the proper hostpci2?
There are three PCIe devices...slot 1 (hostpci0) is Nvidia NVS510 - (Proxmox), slot 2 (hostpci1) is Intel i350x4 and slot 3 (hostpci2) Intel QAT 8950 (0000:04:00.0)

Slot 2 and slot 3 are passthrough to pfSense which has the drivers for both. How to resolve...

This thread is related despite been solved: https://forum.proxmox.com/threads/pcie-passthrough-to-specific-vm.134615/
BTW, I had ran: update-initramfs -u -k all several times...

Where can I find more detail logs on the supposed error?

 

[Nollimox]

@leesteken I would like to blacklist the drivername...which one is the driver name, co-processor or DH895XCC?

 

[leesteken]

@leesteken I would like to blacklist the drivername...which one is the driver name, co-processor or DH895XCC?

View attachment 56412

There is no driver name in that screenshot. What is the output of lspci -knns 04:00? The driver name is after Kernel modules: .

 

[leesteken]

Where can I find more detail logs on the supposed error?

View attachment 56357

Try double clicking the line to show a window with more information (but sometimes it does not show much).
Run journalctl and scroll (using the arrow keys) to the time of starting the VM.

 

[Nollimox]

lspci -knns 04:00?

root@nolliprivatecloud:~# lspci -knns 04:00
0000:04:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [8086:0435]
Subsystem: Dell DH895XCC Series QAT [1028:0738]
Kernel driver in use: dh895xcc
Kernel modules: qat_dh895xcc
root@nolliprivatecloud:~#

Does this subsystem also needed to be added to the pve.blacklist.conf? What about the kernel model, does that needs to be there also?

 

[leesteken]

0000:04:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [8086:0435]
Subsystem: Dell DH895XCC Series QAT [1028:0738]
Kernel driver in use: dh895xcc
Kernel modules: qat_dh895xcc

Does this subsystem also needed to be added to the pve.blacklist.conf? What about the kernel model, does that needs to be there also?

The kernel module is qat_dh895xcc. That is what you use to prevent the driver from loading: blacklist qat_dh895xcc in a .conf file in the /etc/modprobe.d directory and run update-initramfs -u.

However, I don't think this will help you much with the original issue. That why I suggested looking at the relevant error messages.

 

[Nollimox]

The kernel module is qat_dh895xcc. That is what you use to prevent the driver from loading: blacklist qat_dh895xcc in a .conf file in the /etc/modprobe.d directory and run update-initramfs -u.

However, I don't think this will help you much with the original issue. That why I suggested looking at the relevant error messages.

Before you had responded, I tried adding all related to blacklist:

GNU nano 7.2 /etc/modprobe.d/pve-blacklist.conf
# This file contains a list of modules which are not supported by Proxmox VE

# nvidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
blacklist vfio-pci ids=8086:1521, 8086:0435, 1028:0738
blacklist dh895xcc
blacklist qat_dh895xcc

However, that produces the same VM failed to start with same error and journalctl still shows:

Oct 04 21:55:48 nolliprivatecloud kernel: dh895xcc 0000:04:00.0: qat_dev0 started 12 acceleration engines

So, I removed the dh895xcc drivername from the pve-blacklist.conf and just kept the the qat_dh895xcc, ran
update-initramfs -u -k all
followed by
proxmox-boot-tool refresh

rebooted Proxmox and launched VM...failed to start Erroe cannot bind 04:00

Checked: journalctl

Oct 04 21:55:48 nolliprivatecloud kernel: dh895xcc 0000:04:00.0: qat_dev0 started 12 acceleration engines

That's when I noticed the logs stuck on the same date.
So, I rebooted...

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu Oct 12 09:48:57 CDT 2023 on pts/0
root@nolliprivatecl

Date is correct, then, ran journalctl again,

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Thu Oct 12 09:48:57 CDT 2023 on pts/0
root@nolliprivatecloud:~# journalctl
Oct 04 21:55:47 nolliprivatecloud kernel: Linux version 6.2.16-3-pve (tom@sbuild) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.>
Oct 04 21:55:47 nolliprivatecloud kernel: Command line: initrd=\EFI\proxmox\6.2.16-3-pve\initrd.img-6.2.16-3-pve root=ZFS=rpool/ROOT/pve-1 boot=zfs
Oct 04 21:55:47 nolliprivatecloud kernel: KERNEL supported cpus:
Oct 04 21:55:47 nolliprivatecloud kernel: Intel GenuineIntel
Oct 04 21:55:47 nolliprivatecloud kernel: AMD AuthenticAMD
Oct 04 21:55:47 nolliprivatecloud kernel: Hygon HygonGenuine
Oct 04 21:55:47 nolliprivatecloud kernel: Centaur CentaurHauls
Oct 04 21:55:47 nolliprivatecloud kernel: zhaoxin Shanghai

Wondered why it went back to that date despite several rebooting?
Any errors? The only thing I noticed looking through the long file is that as soon as I passthrough the NIC, I get:

Oct 04 21:55:47 nolliprivatecloud kernel: pci 10000:00:03.0: PCI bridge to [bus 02]
Oct 04 21:55:47 nolliprivatecloud kernel: pci 10000:00:03.0: bridge window [mem 0xf8400000-0xf85fffff]
Oct 04 21:55:47 nolliprivatecloud kernel: pci 10000:00:03.0: bridge window [mem 0xf8600000-0xf87fffff 64bit pref]
Oct 04 21:55:47 nolliprivatecloud kernel: pcieport 10000:00:02.0: can't derive routing for PCI INT A
Oct 04 21:55:47 nolliprivatecloud kernel: pcieport 10000:00:02.0: PCI INT A: no GSI
Oct 04 21:55:47 nolliprivatecloud kernel: pcieport 10000:00:02.0: pciehp: Slot #91 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlo>
Oct 04 21:55:47 nolliprivatecloud kernel: pcieport 10000:00:03.0: can't derive routing for PCI INT A
Oct 04 21:55:47 nolliprivatecloud kernel: pcieport 10000:00:03.0: PCI INT A: no GSI
Oct 04 21:55:47 nolliprivatecloud kernel: pcieport 10000:00:03.0: pciehp: Slot #92 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlo>
Oct 04 21:55:47 nolliprivatecloud kernel: vmd 0000:bc:05.5: Bound to PCI domain 10000
Oct 04 21:55:47 nolliprivatecloud kernel: pcieport 10000:00:02.0: pciehp: Slot(91): Power fault
Oct 04 21:55:47 nolliprivatecloud kernel: pcieport 10000:00:03.0: pciehp: Slot(92): Power fault
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.0: added PHC on eth0
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.0: Intel(R) Gigabit Ethernet Network Connection
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.0: eth0: (PCIe:5.0Gb/s:Width x4) a0:36:9f:40:a0:90
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.0: eth0: PBA No: G32100-005
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Oct 04 21:55:47 nolliprivatecloud kernel: usb 1-10: new high-speed USB device number 2 using xhci_hcd
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.1: added PHC on eth1
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.1: Intel(R) Gigabit Ethernet Network Connection
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.1: eth1: (PCIe:5.0Gb/s:Width x4) a0:36:9f:40:a0:91
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.1: eth1: PBA No: G32100-005
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.2: added PHC on eth2
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.2: Intel(R) Gigabit Ethernet Network Connection
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.2: eth2: (PCIe:5.0Gb/s:Width x4) a0:36:9f:40:a0:92
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.2: eth2: PBA No: G32100-005
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Oct 04 21:55:47 nolliprivatecloud kernel: tsc: Refined TSC clocksource calibration: 3311.999 MHz
Oct 04 21:55:47 nolliprivatecloud kernel: clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x2fbd936b72f, max_idle_ns: 440795283163 ns
Oct 04 21:55:47 nolliprivatecloud kernel: clocksource: Switched to clocksource tsc
Oct 04 21:55:47 nolliprivatecloud kernel: e1000e 0000:00:1f.6 0000:00:1f.6 (uninitialized): registered PHC clock
Oct 04 21:55:47 nolliprivatecloud kernel: usb 1-10: New USB device found, idVendor=0bda, idProduct=5411, bcdDevice= 1.23
Oct 04 21:55:47 nolliprivatecloud kernel: usb 1-10: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Oct 04 21:55:47 nolliprivatecloud kernel: usb 1-10: Product: 4-Port USB 2.0 Hub
Oct 04 21:55:47 nolliprivatecloud kernel: usb 1-10: Manufacturer: Generic
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.3: added PHC on eth3
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.3: Intel(R) Gigabit Ethernet Network Connection
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.3: eth3: (PCIe:5.0Gb/s:Width x4) a0:36:9f:40:a0:93
Oct 04 21:55:47 nolliprivatecloud kernel: hub 1-10:1.0: USB hub found
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.3: eth3: PBA No: G32100-005

Whereas if I don't passthrough the NIC, it seems happy with vtnet0/vtnet1

 

[Nollimox]

Why root doesn't have permission to view logs?

root@nolliprivatecloud:~# ls /var/log
alternatives.log chrony fontconfig.log lastlog pveam.log pve-firewall.log.3.gz pve-firewall.log.7.gz samba
apt corosync glusterfs lxc pve-firewall.log pve-firewall.log.4.gz pveproxy wtmp
btmp dpkg.log ifupdown2 private pve-firewall.log.1 pve-firewall.log.5.gz README
ceph faillog journal pve pve-firewall.log.2.gz pve-firewall.log.6.gz runit
root@nolliprivatecloud:~# /var/log/lastlog
-bash: /var/log/lastlog: Permission denied
root@nolliprivatecloud:~#

 

[Nollimox]

Well, I double clicked and got this but why? Frustrating, indeed...


What am I doing wrong?

Last login: Thu Oct 12 15:14:18 CDT 2023 on pts/0
root@nolliprivatecloud:~# qm set -hostpci2 0000:04:00.0 pcie=on
400 Parameter verification failed.
vmid: type check ('integer') failed - got 'pcie=on'
qm set <vmid> [OPTIONS]
root@nolliprivatecloud:~# qm set 100 -hostpci2 0000:04:00.0 pcie=on
400 too many arguments
qm set <vmid> [OPTIONS]
root@nolliprivatecloud:~# qm set 100 -hostpci2 04:00 pcie=on
400 too many arguments
qm set <vmid> [OPTIONS]
root@nolliprivatecloud:~# qm set <100> -hostpci2 04:00 pcie=on
-bash: syntax error near unexpected token `100'
root@nolliprivatecloud:~#

And one followed the example...

 

[Nollimox]

Well, according to this, vfio_pci is using 0 device but vfio_pci_core: is using 1

root@nolliprivatecloud:~# lsmod
Module Size Used by
tcp_diag 16384 0
inet_diag 28672 1 tcp_diag
vfio_pci 16384 0
vfio_pci_core 94208 1 vfio_pci
ebtable_filter 16384 0
ebtables 49152 1 ebtable_filter

which one?

GNU nano 7.2 /etc/modprobe.d/pve-blacklist.conf
# This file contains a list of modules which are not supported by Proxmox VE

# nvidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
blacklist vfio-pci ids=8086:0435, 1028:0738
blacklist qat_dh895xcc

So, it seems I need to use a script as here: https://forum.proxmox.com/threads/difficulty-to-bind-vfio-pci-to-hba.56437/

 

[Nollimox]

The script did not work for me sadly and Github has a 25 one...seems that's too much for simplicity: https://github.com/andre-richter/vfio-pci-bind

root@nolliprivatecloud:~# #!/usr/bin/perl
use strict;
use warnings;
print "GUEST HOOK: " . join(' ', @ARGV). "\n";
# First argument is the vmid
my $vmid = shift;
# Second argument is the phase
my $phase = shift;
if ($phase eq 'pre-start') {
# First phase 'pre-start' will be executed before the guest
# is started. Exiting with a code != 0 will abort the start
print "$vmid is starting, doing preparations.\n";
system("echo \”8086 0435\” > /sys/bus/pci/drivers/vfio-pci/new_id &&
echo \"0000:04:00.0\" > /sys/bus/pci/devices/0000:04:00.0/driver/unbind &&
echo \"0000:04:00.0\" > /sys/bus/pci/drivers/vfio-pci/bind &&
echo \”8086 0435\” > /sys/bus/pci/drivers/vfio-pci/remove_id");
# print "preparations failed, aborting."
# exit(1);
} elsif ($phase eq 'post-stop') {
# Last phase 'post-stop' will be executed after the guest stopped.
# This should even be executed in case the guest crashes or stopped
# unexpectedly.
print "$vmid stopped. Doing cleanup.\n";
system("echo 1 > /sys/bus/pci/devices/0000:04:00.0/remove &&
echo 1 > /sys/bus/pci/rescan");
} else {
die "got unknown phase '$phase'\n";
}
exit(0);
-bash: use: command not found
-bash: use: command not found
-bash: syntax error near unexpected token `('
-bash: my: command not found
-bash: my: command not found
-bash: syntax error near unexpected token `{'
Warning: unknown mime-type for " is starting, doing preparations.\n" -- using "application/octet-stream"
Error: no such file " is starting, doing preparations.\n"
-bash: syntax error near unexpected token `$'"echo \\”8086 0435\\” > /sys/bus/pci/drivers/vfio-pci/new_id &&\n echo \\"0000:04:00.0\\" > /sys/bus/pci/devices/0000:04:00.0/driver/unbind &&\n echo \\"0000:04:00.0\\" > /sys/bus/pci/drivers/vfio-pci/bind &&\n echo \\”8086 0435\\” > /sys/bus/pci/drivers/vfio-pci/remove_id"''
-bash: syntax error near unexpected token `}'
Warning: unknown mime-type for " stopped. Doing cleanup.\n" -- using "application/octet-stream"
Error: no such file " stopped. Doing cleanup.\n"
-bash: syntax error near unexpected token `$'"echo 1 > /sys/bus/pci/devices/0000:04:00.0/remove &&\n echo 1 > /sys/bus/pci/rescan"''
-bash: syntax error near unexpected token `}'
-bash: die: command not found
-bash: syntax error near unexpected token `}'
-bash: syntax error near unexpected token `0'
root@nolliprivatecloud:~#

 

[Nollimox]

This one seems to be simpler despite crediting the Github above: https://forum.proxmox.com/threads/script-service-passthrough-helper.85488/

 

[leesteken]

# nvidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
blacklist vfio-pci ids=8086:1521, 8086:0435, 1028:0738

blacklist vfio-pci ids=8086:1521, 8086:0435, 1028:0738 is wrong; use options vfio-pci ids=8086:1521,8086:0435,1028:0738 instead.

blacklist dh895xcc

There is no module named dh895xcc.

blacklist qat_dh895xcc

This looks good. But as I said twice before, I don't think that wll solve your actual issue (which is still unknown).

Any errors? The only thing I noticed looking through the long file is that as soon as I passthrough the NIC, I get:

Oct 04 21:55:47 nolliprivatecloud kernel: pci 10000:00:03.0: PCI bridge to [bus 02]
Oct 04 21:55:47 nolliprivatecloud kernel: pci 10000:00:03.0: bridge window [mem 0xf8400000-0xf85fffff]
Oct 04 21:55:47 nolliprivatecloud kernel: pci 10000:00:03.0: bridge window [mem 0xf8600000-0xf87fffff 64bit pref]
Oct 04 21:55:47 nolliprivatecloud kernel: pcieport 10000:00:02.0: can't derive routing for PCI INT A
Oct 04 21:55:47 nolliprivatecloud kernel: pcieport 10000:00:02.0: PCI INT A: no GSI
Oct 04 21:55:47 nolliprivatecloud kernel: pcieport 10000:00:02.0: pciehp: Slot #91 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlo>
Oct 04 21:55:47 nolliprivatecloud kernel: pcieport 10000:00:03.0: can't derive routing for PCI INT A
Oct 04 21:55:47 nolliprivatecloud kernel: pcieport 10000:00:03.0: PCI INT A: no GSI
Oct 04 21:55:47 nolliprivatecloud kernel: pcieport 10000:00:03.0: pciehp: Slot #92 AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise+ Interlo>
Oct 04 21:55:47 nolliprivatecloud kernel: vmd 0000:bc:05.5: Bound to PCI domain 10000
Oct 04 21:55:47 nolliprivatecloud kernel: pcieport 10000:00:02.0: pciehp: Slot(91): Power fault
Oct 04 21:55:47 nolliprivatecloud kernel: pcieport 10000:00:03.0: pciehp: Slot(92): Power fault
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.0: added PHC on eth0
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.0: Intel(R) Gigabit Ethernet Network Connection
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.0: eth0: (PCIe:5.0Gb/s:Width x4) a0:36:9f:40:a0:90
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.0: eth0: PBA No: G32100-005
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Oct 04 21:55:47 nolliprivatecloud kernel: usb 1-10: new high-speed USB device number 2 using xhci_hcd
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.1: added PHC on eth1
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.1: Intel(R) Gigabit Ethernet Network Connection
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.1: eth1: (PCIe:5.0Gb/s:Width x4) a0:36:9f:40:a0:91
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.1: eth1: PBA No: G32100-005
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.2: added PHC on eth2
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.2: Intel(R) Gigabit Ethernet Network Connection
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.2: eth2: (PCIe:5.0Gb/s:Width x4) a0:36:9f:40:a0:92
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.2: eth2: PBA No: G32100-005
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
Oct 04 21:55:47 nolliprivatecloud kernel: tsc: Refined TSC clocksource calibration: 3311.999 MHz
Oct 04 21:55:47 nolliprivatecloud kernel: clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x2fbd936b72f, max_idle_ns: 440795283163 ns
Oct 04 21:55:47 nolliprivatecloud kernel: clocksource: Switched to clocksource tsc
Oct 04 21:55:47 nolliprivatecloud kernel: e1000e 0000:00:1f.6 0000:00:1f.6 (uninitialized): registered PHC clock
Oct 04 21:55:47 nolliprivatecloud kernel: usb 1-10: New USB device found, idVendor=0bda, idProduct=5411, bcdDevice= 1.23
Oct 04 21:55:47 nolliprivatecloud kernel: usb 1-10: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Oct 04 21:55:47 nolliprivatecloud kernel: usb 1-10: Product: 4-Port USB 2.0 Hub
Oct 04 21:55:47 nolliprivatecloud kernel: usb 1-10: Manufacturer: Generic
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.3: added PHC on eth3
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.3: Intel(R) Gigabit Ethernet Network Connection
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.3: eth3: (PCIe:5.0Gb/s:Width x4) a0:36:9f:40:a0:93
Oct 04 21:55:47 nolliprivatecloud kernel: hub 1-10:1.0: USB hub found
Oct 04 21:55:47 nolliprivatecloud kernel: igb 0000:91:00.3: eth3: PBA No: G32100-005

Whereas if I don't passthrough the NIC, it seems happy with vtnet0/vtnet1

I don't see anything here about 04:00.0. Maybe passthrough of that device or IOMMU group simply does not work? It happens often that passthrough does not work.
Did you show your IOMMU groups already with cat /proc/cmdline; for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done? Make sure not to use pcie_acs_override because that completely invalidates the information.

root@nolliprivatecloud:~# /var/log/lastlog
-bash: /var/log/lastlog: Permission denied

Always use journalctl on Proxmox instead. Use journalctl -b 0 to see only the current boot. Anything in there about 04:00.0 when you start the VM with passthrough of 04:00.0?

Well, I double clicked and got this but why? Frustrating, indeed...

View attachment 56500
What am I doing wrong?

That's what we need to find out with journalctl.

Last login: Thu Oct 12 15:14:18 CDT 2023 on pts/0
root@nolliprivatecloud:~# qm set -hostpci2 0000:04:00.0 pcie=on
400 Parameter verification failed.
vmid: type check ('integer') failed - got 'pcie=on'

You are missing the VM ID number (and are making other syntax errors), but this can be much more easily done via the Proxmox web GUI (and enabling PCI-Express will most likely not fix your issue).

I don't understand your last three posts at all, sorry. Maybe you can share the output of qm config followed by your VM ID number, to check its configuration?

 

[Nollimox]

You are missing the VM ID number (and are making other syntax errors), but this can be much more easily done via the Proxmox web GUI (and enabling PCI-Express will most likely not fix your issue).

In that example, I did left out the VMID; however, when I corrected that, it still resulted in syntax error which seems baffling...so, I removed the 00:04, and that got it enabled:

root@nolliprivatecloud:~# qm set 100 -hostpci2 04:00, pcie=on,
400 too many arguments
qm set <vmid> [OPTIONS]
root@nolliprivatecloud:~#

root@nolliprivatecloud:~# qm set 100 -hostpci2 pcie=on
update VM 100: -hostpci2 pcie=on
root@nolliprivatecloud:~#

Error from logs:
Oct 13 07:53:14 nolliprivatecloud pvedaemon[5648]: start VM 100: UPID:nolliprivatecloud:00001610:0000AA5A:65293DBA:qmstart:100:root@pam:
Oct 13 07:53:14 nolliprivatecloud pvedaemon[2232]: <root@pam> starting task UPID:nolliprivatecloud:00001610:0000AA5A:65293DBA:qmstart:100:root@pam:
Oct 13 07:53:14 nolliprivatecloud pvedaemon[5648]: Either 'host' or 'mapping' must be set.
Oct 13 07:53:14 nolliprivatecloud pvedaemon[2232]: <root@pam> end task UPID:nolliprivatecloud:00001610:0000AA5A:65293DBA:qmstart:100:root@pam: Either>
Oct 13 07:53:31 nolliprivatecloud pvedaemon[5790]: starting termproxy UPID:nolliprivatecloud:0000169E:0000B0FB:65293DCB:vncshell::root@pam:
Oct 13 07:53:31 nolliprivatecloud pvedaemon[2235]: <root@pam> starting task UPID:nolliprivatecloud:0000169E:0000B0FB:65293DCB:vncshell::root@pam:
Oct 13 07:53:31 nolliprivatecloud pvedaemon[2236]: <root@pam> successful auth for user 'root@pam'
Oct 13 07:53:31 nolliprivatecloud login[5798]: pam_unix(login:session): session opened for user root(uid=0) by (uid=0)

Oct 13 08:20:01 nolliprivatecloud pvedaemon[2232]: <root@pam> successful auth for user 'root@pam'
Oct 13 08:21:58 nolliprivatecloud pveproxy[2249]: worker 2250 finished
Oct 13 08:21:58 nolliprivatecloud pveproxy[2249]: starting 1 worker(s)
Oct 13 08:21:58 nolliprivatecloud pveproxy[2249]: worker 18625 started
Oct 13 08:22:00 nolliprivatecloud pveproxy[18624]: got inotify poll request in wrong process - disabling inotify
Oct 13 08:35:01 nolliprivatecloud pvedaemon[2236]: <root@pam> successful auth for user 'root@pam'
Oct 13 08:39:21 nolliprivatecloud pveproxy[2251]: worker exit
Oct 13 08:39:21 nolliprivatecloud pveproxy[2249]: worker 2251 finished
Oct 13 08:39:21 nolliprivatecloud pveproxy[2249]: starting 1 worker(s)
Oct 13 08:39:21 nolliprivatecloud pveproxy[2249]: worker 26387 started
lines 2043-2083/2083 (END)

Wondered about this:

Oct 13 07:46:02 nolliprivatecloud kernel: Booting paravirtualized kernel on bare hardware
Oct 13 07:46:02 nolliprivatecloud kernel: clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
Oct 13 07:46:02 nolliprivatecloud kernel: setup_percpu: NR_CPUS:8192 nr_cpumask_bits:20 nr_cpu_ids:20 nr_node_ids:1
Oct 13 07:46:02 nolliprivatecloud kernel: percpu: Embedded 62 pages/cpu s217088 r8192 d28672 u262144
Oct 13 07:46:02 nolliprivatecloud kernel: pcpu-alloc: s217088 r8192 d28672 u262144 alloc=1*2097152
Oct 13 07:46:02 nolliprivatecloud kernel: pcpu-alloc: [0] 00 01 02 03 04 05 06 07 [0] 08 09 10 11 12 13 14 15
Oct 13 07:46:02 nolliprivatecloud kernel: pcpu-alloc: [0] 16 17 18 19 -- -- -- --
Oct 13 07:46:02 nolliprivatecloud kernel: Fallback order for Node 0: 0
Oct 13 07:46:02 nolliprivatecloud kernel: Built 1 zonelists, mobility grouping on. Total pages: 32946752
Oct 13 07:46:02 nolliprivatecloud kernel: Policy zone: Normal
Oct 13 07:46:02 nolliprivatecloud kernel: Kernel command line: initrd=\EFI\proxmox\6.2.16-3-pve\initrd.img-6.2.16-3-pve root=ZFS=rpool/ROOT/pve-1 boo>
Oct 13 07:46:02 nolliprivatecloud kernel: DMAR: IOMMU enabled
Oct 13 07:46:02 nolliprivatecloud kernel: Unknown kernel command line parameters "boot=zfs", will be passed to user space.
Oct 13 07:46:02 nolliprivatecloud kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes, linear)

This seems to confirmed the the error: Either host or mapping must be set

root@nolliprivatecloud:~# lspci -vvs 0000:04:00.0 | grep driver
root@nolliprivatecloud:~# lspci -vvs 0000:04:00.0 | grep driver
root@nolliprivatecloud:~# lspci -vvs 04:00. | grep driver
root@nolliprivatecloud:~# lspci -vvs 04:00 | grep driver
root@nolliprivatecloud:~#

So, how do I set mapping from the cmdline as the GUI does not allow me to select the device:



Does knowing the BAR number helps in doing the cmdline as below?

Oct 13 07:46:02 nolliprivatecloud kernel: pci 0000:00:1c.5: BAR 13: assigned [io 0x1000-0x1fff]
Oct 13 07:46:02 nolliprivatecloud kernel: pci 0000:00:1c.0: PCI bridge to [bus 01]
Oct 13 07:46:02 nolliprivatecloud kernel: pci 0000:02:00.0: BAR 13: assigned [io 0x1000-0x1fff]
Oct 13 07:46:02 nolliprivatecloud kernel: pci 0000:03:00.0: BAR 13: assigned [io 0x1000-0x1fff]
Oct 13 07:46:02 nolliprivatecloud kernel: pci 0000:04:00.0: BAR 7: assigned [mem 0x92d80000-0x92d9ffff 64bit]
Oct 13 07:46:02 nolliprivatecloud kernel: pci 0000:04:00.0: BAR 9: assigned [mem 0x92da0000-0x92dbffff 64bit]
Oct 13 07:46:02 nolliprivatecloud kernel: pci 0000:03:00.0: PCI bridge to [bus 04]
Oct 13 07:46:02 nolliprivatecloud kernel: pci 0000:03:00.0: bridge window [io 0x

 

[leesteken]

In that example, I did left out the VMID; however, when I corrected that, it still resulted in syntax error which seems baffling...so, I removed the 00:04, and that got it enabled:

root@nolliprivatecloud:~# qm set 100 -hostpci2 04:00, pcie=on,
400 too many arguments
qm set <vmid> [OPTIONS]

Too many spaces. Maybe it would be easier for you to edit the configuration file? But there also, don't use too many spaces.

Error from logs:
Oct 13 07:53:14 nolliprivatecloud pvedaemon[5648]: start VM 100: UPID:nolliprivatecloud:00001610:0000AA5A:65293DBA:qmstart:100:root@pam:
Oct 13 07:53:14 nolliprivatecloud pvedaemon[2232]: <root@pam> starting task UPID:nolliprivatecloud:00001610:0000AA5A:65293DBA:qmstart:100:root@pam:
Oct 13 07:53:14 nolliprivatecloud pvedaemon[5648]: Either 'host' or 'mapping' must be set.

There appears to be a syntax error in your VM configuration file. Please show the VM configuration file: qm config 100. (Did I ask for this before already?)

Wondered about this:

Oct 13 07:46:02 nolliprivatecloud kernel: Booting paravirtualized kernel on bare hardware
Oct 13 07:46:02 nolliprivatecloud kernel: clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
Oct 13 07:46:02 nolliprivatecloud kernel: setup_percpu: NR_CPUS:8192 nr_cpumask_bits:20 nr_cpu_ids:20 nr_node_ids:1
Oct 13 07:46:02 nolliprivatecloud kernel: percpu: Embedded 62 pages/cpu s217088 r8192 d28672 u262144
Oct 13 07:46:02 nolliprivatecloud kernel: pcpu-alloc: s217088 r8192 d28672 u262144 alloc=1*2097152
Oct 13 07:46:02 nolliprivatecloud kernel: pcpu-alloc: [0] 00 01 02 03 04 05 06 07 [0] 08 09 10 11 12 13 14 15
Oct 13 07:46:02 nolliprivatecloud kernel: pcpu-alloc: [0] 16 17 18 19 -- -- -- --
Oct 13 07:46:02 nolliprivatecloud kernel: Fallback order for Node 0: 0
Oct 13 07:46:02 nolliprivatecloud kernel: Built 1 zonelists, mobility grouping on. Total pages: 32946752
Oct 13 07:46:02 nolliprivatecloud kernel: Policy zone: Normal
Oct 13 07:46:02 nolliprivatecloud kernel: Kernel command line: initrd=\EFI\proxmox\6.2.16-3-pve\initrd.img-6.2.16-3-pve root=ZFS=rpool/ROOT/pve-1 boo>
Oct 13 07:46:02 nolliprivatecloud kernel: DMAR: IOMMU enabled
Oct 13 07:46:02 nolliprivatecloud kernel: Unknown kernel command line parameters "boot=zfs", will be passed to user space.
Oct 13 07:46:02 nolliprivatecloud kernel: Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes, linear)

I don't see any problems there; this is normal with Proxmox on ZFS.

This seems to confirmed the the error: Either host or mapping must be set

Please, please, please share the VM configuration file. And maybe some of the other information that was asked...
We are still no closed to the actual error message of the actual problem, except now your VM configuration file is broken.

 

[Nollimox]

Please, please, please share the VM configuration file.

root@nolliprivatecloud:~# qm config 100
agent: 1
bios: ovmf
boot: order=virtio0;ide2;net0
cores: 4
cpu: host
efidisk0: local-zfs:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci2: pcie=on
ide2: local:iso/pfSense-CE-2.7.0-RELEASE-amd64.iso,media=cdrom,size=747284K
machine: q35
memory: 24000
meta: creation-qemu=8.0.2,ctime=1696549275
name: pfSense
net0: virtio=36:62:BF:C7:26:75,bridge=vmbr1
net1: virtio=42:F8:EA:16:16:A8,bridge=vmbr2
net2: virtio=A6:9F:BE:E2:85:67,bridge=vmbr3
net3: virtio=76:1F:60:C5:7D:A3,bridge=vmbr4
numa: 0
ostype: other
scsihw: virtio-scsi-single
smbios1: uuid=8699b88a-5891-44b3-a2ac-6a4e630df71a
sockets: 1
usb0: host=1-7,usb3=1
vga: qxl
virtio0: local-zfs:vm-100-disk-1,iothread=1,size=63G
vmgenid: 0c1ef6cf-4cff-402a-aa60-a2fdd8f6d5ae
root@nolliprivatecloud:~#

Looks like when I added: #qm set -hostpci2 pcie=on...that removed 04:00

Here it is when first configured
GNU nano 7.2 /etc/pve/qemu-server/100.conf
agent: 1
bios: ovmf
boot: order=virtio0;ide2;net0
cores: 4
cpu: host
efidisk0: local-zfs:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci1: 0000:91:00.0
hostpci2: 0000:04:00.0
ide2: local:iso/pfSense-CE-2.7.0-RELEASE-amd64.iso,media=cdrom,size=747284K
machine: q35
memory: 24000
meta: creation-qemu=8.0.2,ctime=1696549275
name: pfSense
net0: virtio=36:62:BF:C7:26:75,bridge=vmbr1,firewall=1
net1: virtio=42:F8:EA:16:16:A8,bridge=vmbr2,firewall=1
numa: 0
ostype: other
scsihw: virtio-scsi-single
smbios1: uuid=8699b88a-5891-44b3-a2ac-6a4e630df71a
sockets: 1
usb0: host=1-7,usb3=1
vga: qxl
virtio0: local-zfs:vm-100-disk-1,iothread=1,size=63G
vmgenid: 0c1ef6cf-4cff-402a-aa60-a2fdd8f6d5ae

when I set it as previously shown, I get error:
root@nolliprivatecloud:~# qm set 100 -hostpci2 04:00, pcie=on,
400 too many arguments
qm set <vmid> [OPTIONS]
root@nolliprivatecloud:~#

 

[leesteken]

root@nolliprivatecloud:~# qm config 100
agent: 1
bios: ovmf
boot: order=virtio0;ide2;net0
cores: 4
cpu: host
efidisk0: local-zfs:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci2: pcie=on
ide2: local:iso/pfSense-CE-2.7.0-RELEASE-amd64.iso,media=cdrom,size=747284K
machine: q35
memory: 24000
meta: creation-qemu=8.0.2,ctime=1696549275
name: pfSense
net0: virtio=36:62:BF:C7:26:75,bridge=vmbr1
net1: virtio=42:F8:EA:16:16:A8,bridge=vmbr2
net2: virtio=A6:9F:BE:E2:85:67,bridge=vmbr3
net3: virtio=76:1F:60:C5:7D:A3,bridge=vmbr4
numa: 0
ostype: other
scsihw: virtio-scsi-single
smbios1: uuid=8699b88a-5891-44b3-a2ac-6a4e630df71a
sockets: 1
usb0: host=1-7,usb3=1
vga: qxl
virtio0: local-zfs:vm-100-disk-1,iothread=1,size=63G
vmgenid: 0c1ef6cf-4cff-402a-aa60-a2fdd8f6d5ae
root@nolliprivatecloud:~#

As you correctly pointed out hostpci2 is wrong.

Looks like when I added: #qm set -hostpci2 pcie=on...that removed 04:00

Here it is when first configured
GNU nano 7.2 /etc/pve/qemu-server/100.conf
agent: 1
bios: ovmf
boot: order=virtio0;ide2;net0
cores: 4
cpu: host
efidisk0: local-zfs:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci1: 0000:91:00.0
hostpci2: 0000:04:00.0
ide2: local:iso/pfSense-CE-2.7.0-RELEASE-amd64.iso,media=cdrom,size=747284K
machine: q35
memory: 24000
meta: creation-qemu=8.0.2,ctime=1696549275
name: pfSense
net0: virtio=36:62:BF:C7:26:75,bridge=vmbr1,firewall=1
net1: virtio=42:F8:EA:16:16:A8,bridge=vmbr2,firewall=1
numa: 0
ostype: other
scsihw: virtio-scsi-single
smbios1: uuid=8699b88a-5891-44b3-a2ac-6a4e630df71a
sockets: 1
usb0: host=1-7,usb3=1
vga: qxl
virtio0: local-zfs:vm-100-disk-1,iothread=1,size=63G
vmgenid: 0c1ef6cf-4cff-402a-aa60-a2fdd8f6d5ae

This looks better.

when I set it as previously shown, I get error:
root@nolliprivatecloud:~# qm set 100 -hostpci2 04:00, pcie=on,
400 too many arguments
qm set <vmid> [OPTIONS]
root@nolliprivatecloud:~#

Yes because your command is wrong because there is a space between 04:00, and pcie=on,. As I said before:

Too many spaces. Maybe it would be easier for you to edit the configuration file? But there also, don't use too many spaces.

I do hope someone here can help you, as I feel we are going around in circles. The pcie=on will not help you and we need to find out the actual error, which is not possible while you keep breaking other things. It feels to me that it is impossible to get the relevant information from you, sorry.

 

[Nollimox]

it would be easier for you to edit the configuration file?

That's exactly what I ended doing:

GNU nano 7.2 /etc/pve/qemu-server/100.conf
agent: 1
bios: ovmf
boot: order=virtio0;ide2;net0
cores: 4
cpu: host
efidisk0: local-zfs:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci2: 00:04, pcie=on
ide2: local:iso/pfSense-CE-2.7.0-RELEASE-amd64.iso,media=cdrom,size=747284K
machine: q35
memory: 24000
meta: creation-qemu=8.0.2,ctime=1696549275
name: pfSense
net0: virtio=36:62:BF:C7:26:75,bridge=vmbr1
net1: virtio=42:F8:EA:16:16:A8,bridge=vmbr2
net2: virtio=A6:9F:BE:E2:85:67,bridge=vmbr3
net3: virtio=76:1F:60:C5:7D:A3,bridge=vmbr4
numa: 0
ostype: other
scsihw: virtio-scsi-single
smbios1: uuid=8699b88a-5891-44b3-a2ac-6a4e630df71a
sockets: 1
usb0: host=1-7,usb3=1
vga: qxl
virtio0: local-zfs:vm-100-disk-1,iothread=1,size=63G
vmgenid: 0c1ef6cf-4cff-402a-aa60-a2fdd8f6d5ae

So, now, the VM starts



However, still not binding:
root@nolliprivatecloud:~# lspci -vvs 0000:04:00.0 | grep driver
root@nolliprivatecloud:~#

So, it seems I need to use a script as mentioned here in post #12:
https://forum.proxmox.com/threads/script-service-passthrough-helper.85488/

 

[Nollimox]

The pve-blacklist.conf is not working because lspci -nnk still shows:

0000:04:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [8086:0435]
Subsystem: Dell DH895XCC Series QAT [1028:0738]
Kernel modules: qat_dh895xcc

 

[leesteken]

The pve-blacklist.conf is not working because lspci -nnk still shows:

0000:04:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [8086:0435]
Subsystem: Dell DH895XCC Series QAT [1028:0738]
Kernel modules: qat_dh895xcc

It is working because there is no line with Driver in use: .