Issue passing QAT VF with SR-IOV

D

DEHAAS

New Member
Proxmox Subscriber
May 3, 2022
8
1
3
Hi,

I am trying to run virtual machines on Proxmox using passthrough PCI with SR-IOV for Intel QAT. I have other machines with direct passthrough NICs which work just fine. I have installed the QAT driver (https://downloadmirror.intel.com/761891/QAT.L.4.20.0-00001.tar.gz) and enabled virtual functions as described in the Intel QAT documentation for KVM (https://01.org/sites/default/files/...t-technology/330689qatvirtappnoterev007us.pdf). The driver is configured for host mode with SR-IOV (./configure --enable-icp-sriov=host) The host processor is an Intel Xeon D-2100.

The qat service, and qat vf service seem to be fine.
1671556894928.png

1671556995182.png

1671556539958.png

I can list all of the virtual devices just fine on the host. However, they do not appear in the PCI selection list in the GUI. When added manually and trying to start the VM with a QAT VF device, I get the following error:

1671556604032.png

Any idea what the issue might be here, or how I can diagnose it further?
 
Sorry for resurrecting an old thread, struggling to passthrough the QAT adapter IQA89601G1P5 "8086:37c8"
I followed this document which describes how to passthrough QAT to KVM but in the end, I hit a problem that I couldn't resolve.
1) PCI device is not listed in the PCI devices dropdown in GUI (with enabled SR-IOV and installed drivers on the Proxmox host and with disabled SR-IOV without drivers)
if I add the device manually to vmid.conf, I see the device available within the VM but I get...

Code: 
[23.01-RELEASE][root@pfSense.home.lan]/root: dmesg | grep qat
qat0: <Intel c6xx QuickAssist> mem 0xf1800000-0xf183ffff,0xf1840000-0xf187ffff irq 21 at device 9.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1880000-0xf18bffff,0xf18c0000-0xf18fffff irq 25 at device 10.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1900000-0xf193ffff,0xf1940000-0xf197ffff irq 29 at device 11.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1800000-0xf183ffff,0xf1840000-0xf187ffff irq 21 at device 9.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1880000-0xf18bffff,0xf18c0000-0xf18fffff irq 25 at device 10.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1900000-0xf193ffff,0xf1940000-0xf197ffff irq 29 at device 11.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat_ocf0: <QAT engine>
qat_ocf0: no QAT IRQ instances available
device_attach: qat_ocf0 attach returned 6
qat0: <Intel c6xx QuickAssist> mem 0xf1800000-0xf183ffff,0xf1840000-0xf187ffff irq 21 at device 9.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1880000-0xf18bffff,0xf18c0000-0xf18fffff irq 25 at device 10.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1900000-0xf193ffff,0xf1940000-0xf197ffff irq 29 at device 11.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19

Solution for topic starter:
Code: 
On more recent kernels, it may be required to have the vfio-pci module inserted with
“disable_denylist=1”. Note that this can be done once with “sudo modprobe vfio-pci
disable_denylist=1” or persistently by adding the option “options vfio-pci
disable_denylist=1” to /etc/modprobe.d/vfio-pci.conf.
 
Last edited:
Impovich said:
Sorry for resurrecting an old thread, struggling to passthrough the QAT adapter IQA89601G1P5 "8086:37c8"
I followed this document which describes how to passthrough QAT to KVM but in the end, I hit a problem that I couldn't resolve.
1) PCI device is not listed in the PCI devices dropdown in GUI (with enabled SR-IOV and installed drivers on the Proxmox host and with disabled SR-IOV without drivers)
if I add the device manually to vmid.conf, I see the device available within the VM but I get...

Code: 
[23.01-RELEASE][root@pfSense.home.lan]/root: dmesg | grep qat
qat0: <Intel c6xx QuickAssist> mem 0xf1800000-0xf183ffff,0xf1840000-0xf187ffff irq 21 at device 9.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1880000-0xf18bffff,0xf18c0000-0xf18fffff irq 25 at device 10.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1900000-0xf193ffff,0xf1940000-0xf197ffff irq 29 at device 11.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1800000-0xf183ffff,0xf1840000-0xf187ffff irq 21 at device 9.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1880000-0xf18bffff,0xf18c0000-0xf18fffff irq 25 at device 10.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1900000-0xf193ffff,0xf1940000-0xf197ffff irq 29 at device 11.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat_ocf0: <QAT engine>
qat_ocf0: no QAT IRQ instances available
device_attach: qat_ocf0 attach returned 6
qat0: <Intel c6xx QuickAssist> mem 0xf1800000-0xf183ffff,0xf1840000-0xf187ffff irq 21 at device 9.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1880000-0xf18bffff,0xf18c0000-0xf18fffff irq 25 at device 10.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1900000-0xf193ffff,0xf1940000-0xf197ffff irq 29 at device 11.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19

Solution for topic starter:
Code: 
On more recent kernels, it may be required to have the vfio-pci module inserted with
“disable_denylist=1”. Note that this can be done once with “sudo modprobe vfio-pci
disable_denylist=1” or persistently by adding the option “options vfio-pci
disable_denylist=1” to /etc/modprobe.d/vfio-pci.conf.
Click to expand...
Hi Imovich,
Thanks a lot for replying here. I never got it to work. Today, I have tried again, following the same steps and applying the disable_denylist flag, but unfortunately I still get the same error. Here are my exact steps:

wget https://downloadmirror.intel.com/773821/QAT.L.4.21.0-00001.tar.gz
mkdir /tmp/qat4.21
tar -xf QAT.L.4.21.0-00001.tar.gz -C /tmp/qat4.21/
cd /tmp/qat4.21
apt install pve-headers-$(uname -r)
./configure --enable-icp-sriov=host
make install
modprobe vfio-pci disable_denylist=1
service qat_service restart
service qat_service_vfs stop
service qat_service_vfs start

I can see the VF with lspci -nd 8086:37c9, but they do not appear in the GUI. I have then tried to modify the vm.conf in /etc/pve/qemu-server by adding:

hostpci0: 0000:b7:01.5,pcie=1

Unfortunately, I still get the "TASK ERROR: Cannot bind 0000:b7:01.5 to vfio" error message when trying to boot the VM.
I must be doing something different than you if you can boot the VM with the device. Any change you can spot a difference in my steps?
 
HI DEHAAS,
I had the same issue as you before I added disable_denylist=1. BTW did you reboot the host after you disabled denylist?
Also, try to grep logs for denylist, that is where I found out what is going on.

I did another attempt and now I see something different than before

Code: 
[23.01-RELEASE][root@pfSense.home.lan]/root: dmesg | grep qat
qat_ocf0: <QAT engine>
qat_ocf0: no QAT IRQ instances available
device_attach: qat_ocf0 attach returned 6
qat_ocf0: <QAT engine>
qat_ocf0: no QAT IRQ instances available
device_attach: qat_ocf0 attach returned 6
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom