[SOLVED] Pve-blacklist.conf Not Working

N

Nollimox

Member
Mar 9, 2023
273
24
23
This thread below is marked solved because I got the VM to start and is running.
https://forum.proxmox.com/threads/vm-failed-to-start-error-cannot-bind-0000-04-00-0.134671/

However, I discovered that the balcklist is not carrying out the intended function. Here is the file content.

GNU nano 7.2 /etc/modprobe.d/pve-blacklist.conf *
# This file contains a list of modules which are not supported by Proxmox VE
# nvidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
blacklist options vfio-pci ids=8086:0435, 1028:0738
blacklist qat_dh895xcc

Yet running the command: lspci -nnk still shows:

0000:04:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [8086:0435]
Subsystem: Dell DH895XCC Series QAT [1028:0738]
Kernel modules: qat_dh895xcc

While lsmod shows:

vfio_pci 16384 0
vfio_pci_core 94208 1 vfio_pci
irqbypass 16384 16 vfio_pci_core,kvm
vfio_iommu_type1 49152 0
vfio 57344 3 vfio_pci_core,vfio_iommu_type1,vfio_pci

So, until that's resolved, the script won't work...why it's not been removed from the kernel?
 
Last edited:
Nollimox said:
# This file contains a list of modules which are not supported by Proxmox VE
# nvidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
blacklist options vfio-pci ids=8086:0435, 1028:0738
Click to expand...
blacklist options vfio-pci ids=8086:0435, 1028:0738 is wrong in various ways: https://forum.proxmox.com/threads/v...r-cannot-bind-0000-04-00-0.134671/post-596067
Nollimox said:
blacklist qat_dh895xcc

Yet running the command: lspci -nnk still shows:

0000:04:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [8086:0435]
Subsystem: Dell DH895XCC Series QAT [1028:0738]
Kernel modules: qat_dh895xcc
Click to expand...
Blacklisting is working because there is no line with Driver in use: : https://forum.proxmox.com/threads/v...r-cannot-bind-0000-04-00-0.134671/post-596338
 
Okay, I had made the changes:

GNU nano 7.2 /etc/modprobe.d/pve-blacklist.conf
# This file contains a list of modules which are not supported by Proxmox VE
# nvidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
blacklist qat_dh895xcc
options vfio-pci ids=8086:0435, 1028:0738

Did update-initramfs -u -k all
rebooted and it made no difference, lspci -nnk still shows the kernel modules is still using the driver instead of vfio_pci

0000:04:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [8086:0435]
Subsystem: Dell DH895XCC Series QAT [1028:0738]
Kernel modules: qat_dh895xcc
0000:06:00.0 PCI bridge [0604]: Texas Instruments XIO2001 PCI Express-to-PCI Bridge [104c:8240]
 
The kernel modules are still not releasing the pci device...

root@nolliprivatecloud:~# lspci -knns 04:00
0000:04:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [8086:0435]
Subsystem: Dell DH895XCC Series QAT [1028:0738]
Kernel modules: qat_dh895xcc
root@nolliprivatecloud:~#

GNU nano 7.2 /etc/pve/qemu-server/100.conf
agent: 1
bios: ovmf
boot: order=virtio0;ide2;net0
cores: 4
cpu: host
efidisk0: local-zfs:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci2: 00:04, pcie=on
ide2: local:iso/pfSense-CE-2.7.0-RELEASE-amd64.iso,media=cdrom,size=747284K
machine: q35
memory: 24000
meta: creation-qemu=8.0.2,ctime=1696549275
name: pfSense
net0: virtio=36:62:BF:C7:26:75,bridge=vmbr1
net1: virtio=42:F8:EA:16:16:A8,bridge=vmbr2
net2: virtio=A6:9F:BE:E2:85:67,bridge=vmbr3
net3: virtio=76:1F:60:C5:7D:A3,bridge=vmbr4
numa: 0
ostype: other
scsihw: virtio-scsi-single
smbios1: uuid=8699b88a-5891-44b3-a2ac-6a4e630df71a
sockets: 1
usb0: host=1-7,usb3=1
vga: qxl
virtio0: local-zfs:vm-100-disk-1,iothread=1,size=63G
vmgenid: 0c1ef6cf-4cff-402a-aa60-a2fdd8f6d5ae

The hostpci is present above yet it's not present on the GUI...

Screen Shot 2023-10-14 at 7.54.24 PM.png

So, I removed the hostpci2 04:00 from 100.conf then added it via the shell;

root@nolliprivatecloud:~# qm set 100 -hostpci2 04:00,pcie=on
update VM 100: -hostpci2 04:00,pcie=on
root@nolliprivatecloud:~#

Now, it shows up on the GUI:
Screen Shot 2023-10-14 at 8.51.20 PM.png

So, I''ll update update-initramfs -u -k all and reboot only to get the error: TASK ERROR: Cannot bind 0000:04:00.0 to vfio
and the VM fails to boot. So, it seems that when I edited the VMID.conf, and added the hostpci, the VM boots, but if I set vis the qm command, I get the error....I'll try without the pcie on.

Screen Shot 2023-10-14 at 9.07.41 PM.png

You can also see in the GUI that there's a red line around the raw interface

Screen Shot 2023-10-14 at 9.32.31 PM.png
 
Last edited:
Now, we have a new error message:
Oct 14 21:44:40 nolliprivatecloud pvedaemon[2714]: Cannot bind 0000:04:00.0 to vfio
Oct 14 21:44:40 nolliprivatecloud kernel: vfio-pci 0000:04:00.0: 8086:0435 exists in vfio-pci device denylist, driver probing disallowed.
Oct 14 21:44:40 nolliprivatecloud kernel: vfio-pci: probe of 0000:04:00.0 failed with error -22
Oct 14 21:44:40 nolliprivatecloud pvedaemon[2171]: <root@pam> end task UPID:nolliprivatecloud:00000A9A:000016DB:652B5217:qmstart:100:root@pam: Cannot>
Oct 14 21:46:34 nolliprivatecloud pvedaemon[3632]: starting termproxy UPID:nolliprivatecloud:00000E30:000043B0:652B528A:vncshell::root@pam:
Oct 14 21:46:34 nolliprivatecloud pvedaemon[2172]: <root@pam> starting task UPID:nolliprivatecloud:00000E30:000043B0:652B528A:vncshell::root@pam:

How did it got on the vfio_pci denylist and how to remove it?
 
Where could this vfio-pci device denylist, be? I removed the script to be sure it wasn't contributing to the problem.

The only place with "vfio_pci" is;
root@nolliprivatecloud:~# grep -rnw /etc/ -e "vfio-pci"
/etc/modprobe.d/pve-blacklist.conf:5: options vfio-pci ids=8086:0435, 1028:0738
root@nolliprivatecloud:~#

So, it seems that the option doesn't belong there?
 
Last edited:
GNU nano 7.2 /etc/modprobe.d/vfio-pci.conf
options vfio-pci disable_denylist=1

And now pfSense is happy and no script was necessary. This means PCI-passthrough document needs updating to meet modern kernel requirement.

Screen Shot 2023-10-15 at 10.04.01 AM.png
 
KrisFromFuture said:
We should be using "pve-blacklist.conf" or "blacklist.conf" in modprobe.d ?
Click to expand...

I would recommend to not use the default: pve-blacklist.conf one, because it could potentially be, respectively wants to be, overridden with a future update.
 
You must log in or register to reply here.
Top Bottom