SOLUTION:
VM config:
- System -> Machine : q35
- System -> BIOS: OVMF (UEFi)
- System -> Pre-Enroll keys: UN-ticked (!)
PCIe Pass-through:
- VM -> Hardware -> PCI Device -> Raw Device -> Select one of the ID, then tick "All Functions" and tick "PCI-Express"
//OR
- Datacenter -> Resource Mappings -> Add -> Select ID with "Pass through all functions as one device" in Device column, then
- VM -> Hardware -> PCI Device -> Mapped Device -> Select the mapped device from previous step, also tick "PCI-Express"
Start the VM, host should release the kernel driver and map it to vfio-pci, the device should be fully available in the VM
ORIGINAL POST:
Hello Proxmox gurus" I am at dead-end with the following so hoping for someone to chime-in with a magical ideas/solution.
I have Dell R630 server with additional PCIe NIC Chelsio Dual Port T520-CR 10GbE 0J6VY6 (link) which I'd like to fully pass-through to one of my (Linux) VMs.
I am running PVE 9.0.10 on ZFS
(Details about IOMMU and other stuff at the bottom)
I see these under PCIe Device -> Raw Device
- If i pass-through any of the four red ones, VM boots fine but no NICs are available in the VM
- If i pass-through the fifth (green) one, VM boots fine and I get two NICs available in the VM - technically OK but not the desired state
- The blue are not NICs, so i ignore these
//EDIT: If I pass-through one of these the VM won't boot either. So this is causing the issue, but why? //EDIT
BUT ! What I'd like to do is to pass-through the whole PCIe device to the VM. The problem is that when i do so, the VM does not boot and gets stuck on this:
I tried both ways, result is the same, VM won't boot and i have to hard-stop it.
- VM -> Hardware -> PCI Device -> All Functions
- Datacenter -> Resource Mappings -> Pass through all functions as one device
Also when in this "limbo" state the node summary shows 1CPU and all memory it fuly utilized (could be misleading though)
Things i tried from various srouces, threads, guides, etc ... Note that none of this helped (I wouldn't be writting this lovestory otherwise)
- Set ROM-Bar = 0 (Un-ticked the option)
- Use q35 machine type instead of the default i440fx
- Use OVMF (UEFI) bios instead of SeaBIOS
- Use x86-64-v2-AES processor type (It is the default value these days but i found some old thread with similar case)
Details about the host machine:
- Dell Poweredge R630 with 2x Intel Xeon 2x E5-2640 v4
- BIOS updated to most-recent available version
- Following BIOS options are enabled
-- Virtual Technology
-- X2Apic Mode
-- SR-IOV Global Enable
- I have NOT added the grup entry intel_iommu=on since this is on by default in the recent kernels (as per the Proxmox docs)
- I see a 90+ IOMMU groups available when running following and together with the fact that i can pass-through the ports via the one ID i assume IOMMU works as expected. But for the sake of completeness...
I will not paste all 100 lines but here is a
This is the Chelsio Network driver. I am adding this since i found some old posts that there were some bugs in the earlier FW versions. I am on the 1.27.5 (most-recent) where the issues are resolved.
//EDIT2:
Adding output of
Any idea, clue or magic checkmark how to fix this?
Thank you in advance!
VM config:
- System -> Machine : q35
- System -> BIOS: OVMF (UEFi)
- System -> Pre-Enroll keys: UN-ticked (!)
PCIe Pass-through:
- VM -> Hardware -> PCI Device -> Raw Device -> Select one of the ID, then tick "All Functions" and tick "PCI-Express"
//OR
- Datacenter -> Resource Mappings -> Add -> Select ID with "Pass through all functions as one device" in Device column, then
- VM -> Hardware -> PCI Device -> Mapped Device -> Select the mapped device from previous step, also tick "PCI-Express"
Start the VM, host should release the kernel driver and map it to vfio-pci, the device should be fully available in the VM
ORIGINAL POST:
Hello Proxmox gurus" I am at dead-end with the following so hoping for someone to chime-in with a magical ideas/solution.
I have Dell R630 server with additional PCIe NIC Chelsio Dual Port T520-CR 10GbE 0J6VY6 (link) which I'd like to fully pass-through to one of my (Linux) VMs.
I am running PVE 9.0.10 on ZFS
(Details about IOMMU and other stuff at the bottom)
I see these under PCIe Device -> Raw Device
- If i pass-through any of the four red ones, VM boots fine but no NICs are available in the VM
- If i pass-through the fifth (green) one, VM boots fine and I get two NICs available in the VM - technically OK but not the desired state
- The blue are not NICs, so i ignore these
//EDIT: If I pass-through one of these the VM won't boot either. So this is causing the issue, but why? //EDIT
BUT ! What I'd like to do is to pass-through the whole PCIe device to the VM. The problem is that when i do so, the VM does not boot and gets stuck on this:
I tried both ways, result is the same, VM won't boot and i have to hard-stop it.
- VM -> Hardware -> PCI Device -> All Functions
- Datacenter -> Resource Mappings -> Pass through all functions as one device
Also when in this "limbo" state the node summary shows 1CPU and all memory it fuly utilized (could be misleading though)
Things i tried from various srouces, threads, guides, etc ... Note that none of this helped (I wouldn't be writting this lovestory otherwise)
- Set ROM-Bar = 0 (Un-ticked the option)
- Use q35 machine type instead of the default i440fx
- Use OVMF (UEFI) bios instead of SeaBIOS
- Use x86-64-v2-AES processor type (It is the default value these days but i found some old thread with similar case)
Details about the host machine:
- Dell Poweredge R630 with 2x Intel Xeon 2x E5-2640 v4
- BIOS updated to most-recent available version
- Following BIOS options are enabled
-- Virtual Technology
-- X2Apic Mode
-- SR-IOV Global Enable
- I have NOT added the grup entry intel_iommu=on since this is on by default in the recent kernels (as per the Proxmox docs)
- I see a 90+ IOMMU groups available when running following and together with the fact that i can pass-through the ports via the one ID i assume IOMMU works as expected. But for the sake of completeness...
for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; doneI will not paste all 100 lines but here is a
| grep Ethernet piece of itdmesg | grep -e DMAR -e IOMMUdmesg | grep remappingdmesg |grep cxgbThis is the Chelsio Network driver. I am adding this since i found some old posts that there were some bugs in the earlier FW versions. I am on the 1.27.5 (most-recent) where the issues are resolved.
//EDIT2:
Adding output of
lspci -v for the Chelsio Etherned and Storage controllerAny idea, clue or magic checkmark how to fix this?
Thank you in advance!
Attachments
Last edited:
