How To Delete Network Card and ADD Intel QAT adapter in Its Place?

N

Nollimox

Member
Mar 9, 2023
271
22
18
So, I bought an Intel QAT adapter card but, had to put it in the PCIe slot where the network was and move the network to another PCIe slot. I thought that Proxmox 8.1 was more intuitive but discovered its partially so, retaining the old and adding a new name (enp3s0f0, 1, 2, 3 - Intel i350-t4), when I go to delete/remove (enp1s0f0, 1, 2, 3) so I can rename the enp3s0f0 to enp1sofo, etc, I get the card cannot be found. Of course, because of that, the Intel QAT adapter cannot be found either.

So, my question is how I can do what I am wanting to do without having to open the device, uninstall the Intel QAT adapter, reinstall the network card just so I can remove the card with Proxmox?
 
Well, I had to do exactly, what I thought of not doing, of the not intuitive VM OS...open the device remove network, reboot, delete vmbr1 and vmbr2, delete enp1sofo, 1, 2, 3, then reboot and shutdown, install Intel QAT, and the network card, then reboot and reconfigured. The surprising thing is it just keep adding network interface now up to enp6s0f0,1,2,3...wow, enp1s0f0, 1, 2, 3 gone forever?

But, now, the PCIe isn't recognizing the Intel QAT...try adding but it's not in the list presented...how to resolve this? pfSense has the drivers...BUT why Proxmox not seeing the PCIe device? ALL I/O was passed-through. Strange, indeed. The motherboard is Intel and the device is also Intel...Linux should recognize it...what the hell?
 
Last edited:
It seems that Proxmox must have remove the Intel QAT driver(s) from the community edition and reserved it for the subscription enterprise environment. The drivers should have been included in Debian 11...can anyone confirm and how to put it back?
 
Found this: https://github.com/openzfs/zfs/discussions/12723
But, my Proxmox 8 is already on ZFS...I'll continue reading the article as there seems to be a light at the end od the tunnel. In the meantime, what command I could issue to release the network from the PCIe graphic slot in Proxmox...maybe, it will pass-through to pfSense.
 
What I am getting...

root@nollicomm:~# dmesg | grep -e DMAR -e IOMMU
[ 0.010539] ACPI: DMAR 0x0000000079C23C70 0000C8 (v01 INTEL EDK2 00000002 01000013)
[ 0.010581] ACPI: Reserving DMAR table memory at [mem 0x79c23c70-0x79c23d37]
[ 0.050083] DMAR: IOMMU enabled
[ 0.127116] DMAR: Host address width 39
[ 0.127117] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.127122] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[ 0.127124] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.127128] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[ 0.127129] DMAR: RMRR base: 0x00000079a87000 end: 0x00000079aa6fff
[ 0.127131] DMAR: RMRR base: 0x0000007d000000 end: 0x0000007f7fffff
[ 0.127132] DMAR: RMRR base: 0x00000079b39000 end: 0x00000079bb8fff
[ 0.127134] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.127135] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ 0.127136] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.130293] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 1.055829] DMAR: No ATSR found
[ 1.055830] DMAR: No SATC found
[ 1.055831] DMAR: IOMMU feature fl1gp_support inconsistent
[ 1.055832] DMAR: IOMMU feature pgsel_inv inconsistent
[ 1.055833] DMAR: IOMMU feature nwfs inconsistent
[ 1.055834] DMAR: IOMMU feature pasid inconsistent
[ 1.055835] DMAR: IOMMU feature eafs inconsistent
[ 1.055835] DMAR: IOMMU feature prs inconsistent
[ 1.055836] DMAR: IOMMU feature nest inconsistent
[ 1.055837] DMAR: IOMMU feature mts inconsistent
[ 1.055838] DMAR: IOMMU feature sc_support inconsistent
[ 1.055838] DMAR: IOMMU feature dev_iotlb_support inconsistent
[ 1.055839] DMAR: dmar0: Using Queued invalidation
[ 1.055842] DMAR: dmar1: Using Queued invalidation
[ 1.058299] DMAR: Intel(R) Virtualization Technology for Directed I/O

And this, why?

root@nollicomm:~# pvesh get /nodes/nollicomm/hardware/pci --pci-class-blacklist
Option pci-class-blacklist requires an argument
400 unable to parse option
pvesh get <api_path> [OPTIONS] [FORMAT_OPTIONS]
root@nollicomm:~#

Figured it out:

root@nollicomm:~# pvesh get /nodes/nollicomm/hardware/pci --pci-class-blacklist ""
a picture is more appropriate here;

Screen Shot 2023-07-30 at 3.22.54 AM.png

As you can see the second form the top one PCI device appears unknown...I am looking how enable as well check the Bios.
 
Last edited:
Well, I saw this thread: https://forum.proxmox.com/threads/e...y-on-host-but-vms-are-still-reachable.121803/ where @Neobin said;

"If you physically removed the GPU, the PCI-addresses of other devices will most likely change and therefore the PCI-address, that was before your GPU, may now be your NIC, for example.

So, with every (physical) hardware change, you should beforehand disable autostart for all VMs that have PCIe-passthrough set up and confirm/verify every PCI-address of the PCIe-passthroughed devices first, before you start those VMs again."

I am thinking that was my mistake...I should have turned off PCIe passthrough before I made the change removing the NIC from the graphic card slot as that's the only slot available that would take the Intel QAT adapter. So, it appears that it's not seeing the Intel QAT adapter because as far as Proxmox knows, what's in there is a NIC...it was and still is passed-through as a NIC.

My thinking was because the PCIe slot was already passed-through, it didn't matter, I could plug in the Intel QAT adapter and it would passthrough to pfSense who has the drivers.

So, now, If I removed all passthrough and then re-enabled it, would the Intel QAT adapter just passthrough without the need to install Linux drivers since it doesn't concern Proxmox?
 
Per here: https://www.kernel.org/doc/html/latest/PCI/sysfs-pci.html
/sys/devices/pci0000:17
|-- 0000:17:00.0
| |-- class
| |-- config
| |-- device
| |-- enable
| |-- irq
| |-- local_cpus
| |-- remove
| |-- resource
| |-- resource0
| |-- resource1
| |-- resource2
| |-- revision
| |-- rom
| |-- subsystem_device
| |-- subsystem_vendor
| `-- vendor
`-- ...
where PCI is the domain and bus number. In this case,the domain number is 0000 and the bus number is 17 (both values are in hex). This bus contains a single function device in slot 0. The domain and busnumbers are reproduced for convenience. So, perhaps I could run:
remove /sys/devices/pci0000:02 (see post #6)

But, then, how I add it back, enable?
 
Well, I tried...confused now...see below...it might be faster start over with a clean install of Proxmox and VM than waiting for help...at least, I know my hardware has Intel=pt...virtual performance.

root@nollicomm:~# /sys/devices/pci0000:02
-bash: /sys/devices/pci0000:02: No such file or directory
root@nollicomm:~# ls /sys/devices
breakpoint cstate_pkg isa msr pnp0 system uncore_cbox_0 uncore_cbox_3 uncore_imc
cpu i915 kprobe pci0000:00 power tracepoint uncore_cbox_1 uncore_cbox_4 uprobe
cstate_core intel_pt LNXSYSTM:00 platform software uncore_arb uncore_cbox_2 uncore_cbox_5 virtual
root@nollicomm:~#
 
Last edited:
Still at it:
root@nollicomm:~# lspci
00:00.0 Host bridge: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 07)
00:02.0 VGA compatible controller: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630]
00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model
00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
00:15.0 Serial bus controller: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #0 (rev 10)
00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
00:1b.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #17 (rev f0)
00:1b.4 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #21 (rev f0)
00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 (rev f0)
00:1d.2 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #11 (rev f0)
00:1f.0 ISA bridge: Intel Corporation Cannon Point-LP LPC Controller (rev 10)
00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS (rev 10)
00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
00:1f.5 Serial bus controller: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-LM (rev 10)
02:00.0 Non-Volatile memory controller: Kingston Technology Company, Inc. Device 5017 (rev 03)
03:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
03:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
03:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
05:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 04)
root@nollicomm:~# echo 1 >/sys/bus/pci/<05:00.0>/remove
-bash: /sys/bus/pci/: Is a directory
root@nollicomm:~# echo 1 >/sys/bus/pci/<pci05:00.0>/remove
-bash: /sys/bus/pci/: Is a directory
root@nollicomm:~#
 
Time to quit:

root@nollicomm:~# echo 1 >/sys/bus/pci/<05:00.0>/remove
-bash: /sys/bus/pci/: Is a directory
root@nollicomm:~# echo 1 >/sys/bus/pci/<pci05:00.0>/remove
-bash: /sys/bus/pci/: Is a directory
root@nollicomm:~# ls /sys/bus/pci
devices drivers drivers_autoprobe drivers_probe rescan resource_alignment slots uevent
root@nollicomm:~# echo 1 >/sys/bus/pci/slots/<pci05:00.0>/remove
-bash: /sys/bus/pci/slots/: Is a directory
root@nollicomm:~# ls /sys/bus/pci/slots/
1
root@nollicomm:~# echo 1 >/sys/bus/pci/slots/1/<pci05:00.0>/remove
-bash: /sys/bus/pci/slots/1/: Is a directory
root@nollicomm:~# ls /sys/bus/pci/slots/1/
adapter address attention cur_bus_speed latch max_bus_speed power
root@nollicomm:~# echo 1 >/sys/bus/pci/slots/1/adapter/<pci05:00.0>/remove
-bash: /sys/bus/pci/slots/1/adapter/: Is a directory
root@nollicomm:~# ls /sys/bus/pci/slots/1/adapter/
ls: cannot access '/sys/bus/pci/slots/1/adapter/': Not a directory
root@nollicomm:~#
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom