[SOLVED] Pve-blacklist.conf Not Working

Nollimox

Member
Mar 9, 2023
267
21
18
This thread below is marked solved because I got the VM to start and is running.
https://forum.proxmox.com/threads/vm-failed-to-start-error-cannot-bind-0000-04-00-0.134671/

However, I discovered that the balcklist is not carrying out the intended function. Here is the file content.

GNU nano 7.2 /etc/modprobe.d/pve-blacklist.conf *
# This file contains a list of modules which are not supported by Proxmox VE
# nvidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
blacklist options vfio-pci ids=8086:0435, 1028:0738
blacklist qat_dh895xcc

Yet running the command: lspci -nnk still shows:

0000:04:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [8086:0435]
Subsystem: Dell DH895XCC Series QAT [1028:0738]
Kernel modules: qat_dh895xcc

While lsmod shows:

vfio_pci 16384 0
vfio_pci_core 94208 1 vfio_pci
irqbypass 16384 16 vfio_pci_core,kvm
vfio_iommu_type1 49152 0
vfio 57344 3 vfio_pci_core,vfio_iommu_type1,vfio_pci

So, until that's resolved, the script won't work...why it's not been removed from the kernel?
 
Last edited:
# This file contains a list of modules which are not supported by Proxmox VE
# nvidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
blacklist options vfio-pci ids=8086:0435, 1028:0738
blacklist options vfio-pci ids=8086:0435, 1028:0738 is wrong in various ways: https://forum.proxmox.com/threads/v...r-cannot-bind-0000-04-00-0.134671/post-596067
blacklist qat_dh895xcc

Yet running the command: lspci -nnk still shows:

0000:04:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [8086:0435]
Subsystem: Dell DH895XCC Series QAT [1028:0738]
Kernel modules: qat_dh895xcc
Blacklisting is working because there is no line with Driver in use: : https://forum.proxmox.com/threads/v...r-cannot-bind-0000-04-00-0.134671/post-596338
 
Okay, I had made the changes:

GNU nano 7.2 /etc/modprobe.d/pve-blacklist.conf
# This file contains a list of modules which are not supported by Proxmox VE
# nvidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
blacklist qat_dh895xcc
options vfio-pci ids=8086:0435, 1028:0738

Did update-initramfs -u -k all
rebooted and it made no difference, lspci -nnk still shows the kernel modules is still using the driver instead of vfio_pci

0000:04:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [8086:0435]
Subsystem: Dell DH895XCC Series QAT [1028:0738]
Kernel modules: qat_dh895xcc
0000:06:00.0 PCI bridge [0604]: Texas Instruments XIO2001 PCI Express-to-PCI Bridge [104c:8240]
 
The kernel modules are still not releasing the pci device...

root@nolliprivatecloud:~# lspci -knns 04:00
0000:04:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [8086:0435]
Subsystem: Dell DH895XCC Series QAT [1028:0738]
Kernel modules: qat_dh895xcc
root@nolliprivatecloud:~#

GNU nano 7.2 /etc/pve/qemu-server/100.conf
agent: 1
bios: ovmf
boot: order=virtio0;ide2;net0
cores: 4
cpu: host
efidisk0: local-zfs:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci2: 00:04, pcie=on
ide2: local:iso/pfSense-CE-2.7.0-RELEASE-amd64.iso,media=cdrom,size=747284K
machine: q35
memory: 24000
meta: creation-qemu=8.0.2,ctime=1696549275
name: pfSense
net0: virtio=36:62:BF:C7:26:75,bridge=vmbr1
net1: virtio=42:F8:EA:16:16:A8,bridge=vmbr2
net2: virtio=A6:9F:BE:E2:85:67,bridge=vmbr3
net3: virtio=76:1F:60:C5:7D:A3,bridge=vmbr4
numa: 0
ostype: other
scsihw: virtio-scsi-single
smbios1: uuid=8699b88a-5891-44b3-a2ac-6a4e630df71a
sockets: 1
usb0: host=1-7,usb3=1
vga: qxl
virtio0: local-zfs:vm-100-disk-1,iothread=1,size=63G
vmgenid: 0c1ef6cf-4cff-402a-aa60-a2fdd8f6d5ae

The hostpci is present above yet it's not present on the GUI...

Screen Shot 2023-10-14 at 7.54.24 PM.png

So, I removed the hostpci2 04:00 from 100.conf then added it via the shell;


root@nolliprivatecloud:~# qm set 100 -hostpci2 04:00,pcie=on
update VM 100: -hostpci2 04:00,pcie=on
root@nolliprivatecloud:~#

Now, it shows up on the GUI:
Screen Shot 2023-10-14 at 8.51.20 PM.png

So, I''ll update
update-initramfs -u -k all and reboot only to get the error: TASK ERROR: Cannot bind 0000:04:00.0 to vfio
and the VM fails to boot. So, it seems that when I edited the VMID.conf, and added the hostpci, the VM boots, but if I set vis the qm command, I get the error....I'll try without the pcie on.

Screen Shot 2023-10-14 at 9.07.41 PM.png

You can also see in the GUI that there's a red line around the raw interface

Screen Shot 2023-10-14 at 9.32.31 PM.png
 
Last edited:
Now, we have a new error message:
Oct 14 21:44:40 nolliprivatecloud pvedaemon[2714]: Cannot bind 0000:04:00.0 to vfio
Oct 14 21:44:40 nolliprivatecloud kernel: vfio-pci 0000:04:00.0: 8086:0435 exists in vfio-pci device denylist, driver probing disallowed.
Oct 14 21:44:40 nolliprivatecloud kernel: vfio-pci: probe of 0000:04:00.0 failed with error -22
Oct 14 21:44:40 nolliprivatecloud pvedaemon[2171]: <root@pam> end task UPID:nolliprivatecloud:00000A9A:000016DB:652B5217:qmstart:100:root@pam: Cannot>
Oct 14 21:46:34 nolliprivatecloud pvedaemon[3632]: starting termproxy UPID:nolliprivatecloud:00000E30:000043B0:652B528A:vncshell::root@pam:
Oct 14 21:46:34 nolliprivatecloud pvedaemon[2172]: <root@pam> starting task UPID:nolliprivatecloud:00000E30:000043B0:652B528A:vncshell::root@pam:

How did it got on the vfio_pci denylist and how to remove it?
 
Where could this vfio-pci device denylist, be? I removed the script to be sure it wasn't contributing to the problem.

The only place with "vfio_pci" is;
root@nolliprivatecloud:~# grep -rnw /etc/ -e "vfio-pci"
/etc/modprobe.d/pve-blacklist.conf:5: options vfio-pci ids=8086:0435, 1028:0738
root@nolliprivatecloud:~#

So, it seems that the option doesn't belong there?
 
Last edited:
GNU nano 7.2 /etc/modprobe.d/vfio-pci.conf
options vfio-pci disable_denylist=1

And now pfSense is happy and no script was necessary. This means PCI-passthrough document needs updating to meet modern kernel requirement.

Screen Shot 2023-10-15 at 10.04.01 AM.png
 
We should be using "pve-blacklist.conf" or "blacklist.conf" in modprobe.d ?

I would recommend to not use the default: pve-blacklist.conf one, because it could potentially be, respectively wants to be, overridden with a future update.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!