VF Assigned More Than Once

[spetrillo]

Hello all,

I am building an OPNsense firewall vm, using SR-IOV vfs on my Intel I350-T4 PCI card. When I go to assign the PCI ids to the vm config I am getting an error that says the VF is assigned more than once. In the screenshot you will see that the first two PCI IDs are indeed the same, but I selected individual vfs. Notice the last digit is not showing up, as it is in the last two PCI IDs. Is this a bug in Proxmox? Is there something I need to do to fix this?

Thanks,
Steve

 

[leesteken]

The last digit is the 'function' of the PCI(e) device. You probably enabled All Functions for both of the functions (02:10.0 and 02:10.1) of device 02:10. Don't do that or just select 02:10 (either .0 or .1) once with All Functions enabled (which will then not show the last digit).

 

[spetrillo]

So I guess my next question is what is the difference between selecting all functions or not to select it. In my use case I am building an OPNsense firewall. I used the physical functions of my Intel I350 card for Proxmox. I created a bond out of 3 of the 4 ports, and then assigned a VMBR to the bond, with the vlans I want passed across the bond. In the config of my OPNsense vm I then added a PCI device for each vlan I want passed across.

FYI...when I fired up this config it got the the point in the OPNsense install process to let me define the VLANs. Once it went to enumerate the VLANs it blew up the entire Proxmox server, and I literally had to reinstall Proxmox. A subsequent reboot sticks at something saying disabling IRQ 16.

 

[leesteken]

So I guess my next question is what is the difference between selecting all functions or not to select it.

If you select 02:10.0 without All Functions, then only 02:10.0 is passed through.If you select 02:10.1 without All Functions, then only 02:10.1 is passed through.
If you select either with All Functions, then 02:10.0 and 02:10.1 (and all other functions if there are more) are passed through. They are also presented inside the VM as functions of the same device (with All Functions) instead of all separate devices (with one function). Sometimes this matters.

 

[spetrillo]

So then yes I will need to remember to not select all functions bc I want the individual devices. Is a function just another term for virtual NIC, and that all the use of the NIC is part of that virtual function? I keep thinking a function is a limited set of uses.

 

[leesteken]

So then yes I will need to remember to not select all functions bc I want the individual devices.

Why does that matter? Individual functions can be used in the same way as individual devices in almost all cases. Just use All Functions, which will confuse the driver inside the VM the least. You can only passthrough a few devices and keeping the functions together allows you to pass more functions.

Is a function just another term for virtual NIC, and that all the use of the NIC is part of that virtual function? I keep thinking a function is a limited set of uses.

It's just what you call different functions of a PCIe device. It is not specific or limited to a NIC. GPU devices typically have a VGA function and a audio function. None of this has anything to do with networking.

EDIT: Maybe Wikipedia can explain it better: https://en.wikipedia.org/wiki/Peripheral_Component_Interconnect

 

[spetrillo]

OK lets use my 4 port card as the example. I want to use ports 1-3 for my vms, in addition to the OPNsense vm. Port 4 will only be used as the WAN port for OPNsense and nothing else. If I do what you say and check all functions does that mean that all 4 ports are assigned to the OPNsense VM?

 

[leesteken]

If I do what you say and check all functions does that mean that all 4 ports are assigned to the OPNsense VM?

If the hardware is one PCI(e) device with four functions (where each function is a network port) then yes, passthrough with All Functions will pass (one device with) all four ports (since each function is a port).

 

[leesteken]

If the hardware is one PCI(e) device with four functions (where each function is a network port) then yes, passthrough with All Functions will pass (one device with) all four ports (since each function is a port).

Note that it is also possible that one PCIe add-in card contains multiple devices. Those devices can have one or more functions. Some network add-in cards have four ports by using two devices with two functions. Then you need to pass two devices (with All Functions) to pass all four port.
As you can hopefully tell by now, card and devices and functions and ports are not the same. Devices and functions are part of the physical/logical PCI(e) layout. Add-in cards (the PCI(e) connector) and ports are part of the physical electrical (or possibly optical) interfaces.

 

[spetrillo]

Hmmm I think that confused me even more. I attached two screenshots.

First screenshot shows the 4 physical ports of the PCIe card. The second screenshot shows the virtual functions of the PCIe card. How would you assign it to a vm? I want to add a vf from each of the first 3 ports to my OPNsense vm. If I do all functions am I getting all the vfs?

 

[Spaneta]

Hello,

Here I _guess_ ... :
- 2:10.0 to 2:10.2 are the 3 "allocatable" VF for say port 1 of the i350-T4
- 2:10:4 to 2:10:6 are 3 VF ... port 2
- 2:11:0 to 2:11.2 are 3 VF ... port 3
- 2:11.4 to 2:11.6 are 3 VF ... port 4
- and that 1 VF of each port is automatically taken by the "Physical function" for PVE itself thus only 3 usable by VM's for each port

In _this_ hypothesis your first screenshot would imply you have pass-through the _same_ group of VF to 2 different NIC's in your VM thus leading to problems.

I would try to pass-through (first without "All functions") respectively as 4 NIC's inside the VM :
- 2:10.0
- 2:10.4
- 2:11.0
- 2:11.4

Hope it helps !

 

[spetrillo]

I didnt realize that you lose a VF per physical port. That is why the numbering was not making any sense to me!

 

[spetrillo]

Ok I have made soo many changes to my config today that I think I am going in circles. I am going to reload PVE and then I will be following this tutorial on how to define my IOMMU and then SR-IOV: https://www.reddit.com/r/Proxmox/comments/cm81tc/tutorial_enabling_sriov_for_intel_nic_x550t2_on/

The only difference from this is I use ZFS, and thus it will be SystemD rather than Grub. Once that piece is done I will follow it step for step. This is probably the best and most comprehensive tutorial I have seen so far, in one place.

 

[Spaneta]

This Reddit post looks promising.
Thanks for sharing !
Please come back to tell us if these directions helped in your scenario.

Have a nice day,

 

[spetrillo]

I found this one again. I actually responded here since I was able to reprogram my I350s to enable SR-IOV. It seems some of the PCI cards did not have this enabled: https://forums.servethehome.com/ind...iny-m920q-or-m720q-with-cx3-and-sr-iov.35664/

 

[spetrillo]

OK its been a fun day, with a couple reinstalls of PVE to get things right. I now have VFs defined for 3 of the 4 ports on my I350-T4. I also took the time to setup my Intel UHD 630 iGPU for mediated devices, so I can build a Plex VM and get hardware transcoding. I think I am going to stop here for a bit.

Both of these documents: https://www.reddit.com/r/Proxmox/comments/cm81tc/tutorial_enabling_sriov_for_intel_nic_x550t2_on/ and https://forums.servethehome.com/ind...iny-m920q-or-m720q-with-cx3-and-sr-iov.35664/ played a role in getting to where I am.

If I can give some advice...read the documents and then re-read them again. There are nuances everywhere and you might miss something. For example I thought I had to blacklist my NIC driver. Well I read the section wrong...it was the vf driver that needed to be blacklisted but then at the end the poster explained that it did not need to be blacklisted. You cannot just think its a straight copy paste. I would also focus on one topic, make sure its done to completion and working successfully, before moving on. I tried to do SR-IOV for my NIC and setup mediated device support for my iGPU. Turns out I bit off more than I could chew...cost me a couple hours and at least one reinstall.

Remember slow and steady wins the race! I will keep updating here as I move on with my journey. Next is to build an OPNsense VM and pass PFs to it. The VFs will be for the other VMs I need to build.

 

[spetrillo]

OK its been a fun day, with a couple reinstalls of PVE to get things right. I now have VFs defined for 3 of the 4 ports on my I350-T4. I also took the time to setup my Intel UHD 630 iGPU for mediated devices, so I can build a Plex VM and get hardware transcoding. I think I am going to stop here for a bit.

Both of these documents: https://www.reddit.com/r/Proxmox/comments/cm81tc/tutorial_enabling_sriov_for_intel_nic_x550t2_on/ and https://forums.servethehome.com/ind...iny-m920q-or-m720q-with-cx3-and-sr-iov.35664/ played a role in getting to where I am.

If I can give some advice...read the documents and then re-read them again. There are nuances everywhere and you might miss something. For example I thought I had to blacklist my NIC driver. Well I read the section wrong...it was the vf driver that needed to be blacklisted but then at the end the poster explained that it did not need to be blacklisted. You cannot just think its a straight copy paste. I would also focus on one topic, make sure its done to completion and working successfully, before moving on. I tried to do SR-IOV for my NIC and setup mediated device support for my iGPU. Turns out I bit off more than I could chew...cost me a couple hours and at least one reinstall.

Remember slow and steady wins the race! I will keep updating here as I move on with my journey. Next is to build an OPNsense VM and pass PFs to it. The VFs will be for the other VMs I need to build.

Ok I figured that I would ensure from now on...that my posts follow each other. This way ppl can follow along.

I did run into one potential problem, and that is my VFs are coming up before my PFs are enabled in the boot process. Does anyone know what I need to do in order to avoid this issue?

Here is some output from my boot up:

Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.2: PF still in reset state. Is the PF interface up?
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.2: Assigning random MAC address.
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.2: PF still resetting
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.2: Intel(R) I350 Virtual Function
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:10.6 enp1s0f2v1: renamed from eth0
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.2: Address: ea:61:21:81:f9:82
Jun 22 18:42:01 pve01 kernel: pci 0000:02:11.6: [8086:1520] type 00 class 0x020000 PCIe Endpoint
Jun 22 18:42:01 pve01 kernel: pci 0000:02:11.6: Adding to iommu group 24
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.6: enabling device (0000 -> 0002)
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.6: PF still in reset state. Is the PF interface up?
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.6: Assigning random MAC address.
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.6: PF still resetting
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.6: Intel(R) I350 Virtual Function
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.6: Address: 8e:c2:cf:62:be:9e
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.2 enp1s0f2v2: renamed from eth1
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.6 enp1s0f2v3: renamed from eth0
Jun 22 18:42:01 pve01 kernel: igb 0000:01:00.0: setting MAC aa:bb:cc:f0:00:00 on VF 0
Jun 22 18:42:01 pve01 kernel: igb 0000:01:00.0: Reload the VF driver to make this change effective.
Jun 22 18:42:01 pve01 kernel: igb 0000:01:00.0: The VF MAC address has been set, but the PF device is not up.
Jun 22 18:42:01 pve01 kernel: igb 0000:01:00.0: Bring the PF device up before attempting to use the VF device.
Jun 22 18:42:01 pve01 kernel: igb 0000:01:00.0: setting MAC aa:bb:cc:f0:00:01 on VF 1
Jun 22 18:42:01 pve01 kernel: igb 0000:01:00.0: Reload the VF driver to make this change effective.
Jun 22 18:42:01 pve01 kernel: igb 0000:01:00.0: The VF MAC address has been set, but the PF device is not up.
Jun 22 18:42:01 pve01 kernel: igb 0000:01:00.0: Bring the PF device up before attempting to use the VF device.

Does anyone know of a way to delay the vf creation, so that the physical ports come live in the boot process first? Am I making more out of this than I have to?

 

[spetrillo]

Ok I figured that I would ensure from now on...that my posts follow each other. This way ppl can follow along.

I did run into one potential problem, and that is my VFs are coming up before my PFs are enabled in the boot process. Does anyone know what I need to do in order to avoid this issue?

Here is some output from my boot up:

Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.2: PF still in reset state. Is the PF interface up?
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.2: Assigning random MAC address.
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.2: PF still resetting
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.2: Intel(R) I350 Virtual Function
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:10.6 enp1s0f2v1: renamed from eth0
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.2: Address: ea:61:21:81:f9:82
Jun 22 18:42:01 pve01 kernel: pci 0000:02:11.6: [8086:1520] type 00 class 0x020000 PCIe Endpoint
Jun 22 18:42:01 pve01 kernel: pci 0000:02:11.6: Adding to iommu group 24
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.6: enabling device (0000 -> 0002)
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.6: PF still in reset state. Is the PF interface up?
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.6: Assigning random MAC address.
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.6: PF still resetting
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.6: Intel(R) I350 Virtual Function
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.6: Address: 8e:c2:cf:62:be:9e
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.2 enp1s0f2v2: renamed from eth1
Jun 22 18:42:01 pve01 kernel: igbvf 0000:02:11.6 enp1s0f2v3: renamed from eth0
Jun 22 18:42:01 pve01 kernel: igb 0000:01:00.0: setting MAC aa:bb:cc:f0:00:00 on VF 0
Jun 22 18:42:01 pve01 kernel: igb 0000:01:00.0: Reload the VF driver to make this change effective.
Jun 22 18:42:01 pve01 kernel: igb 0000:01:00.0: The VF MAC address has been set, but the PF device is not up.
Jun 22 18:42:01 pve01 kernel: igb 0000:01:00.0: Bring the PF device up before attempting to use the VF device.
Jun 22 18:42:01 pve01 kernel: igb 0000:01:00.0: setting MAC aa:bb:cc:f0:00:01 on VF 1
Jun 22 18:42:01 pve01 kernel: igb 0000:01:00.0: Reload the VF driver to make this change effective.
Jun 22 18:42:01 pve01 kernel: igb 0000:01:00.0: The VF MAC address has been set, but the PF device is not up.
Jun 22 18:42:01 pve01 kernel: igb 0000:01:00.0: Bring the PF device up before attempting to use the VF device.

Does anyone know of a way to delay the vf creation, so that the physical ports come live in the boot process first? Am I making more out of this than I have to?

I am back at it and I have successfully built my OPNsense VM. It was a nerve wracking process bc I have done this many times before, only to have the vm die or PVE crash all together. Well not this time! For those of you wondering I took a screenshot of my config. I tried to use UEFI but it would not boot, so I went to SeaBIOS with Q35.

My OPNsense VM uses a Linux bridge to get to my mgmt vlan. It then uses PCIe passthrough(3 physical ports on my Intel I350) for all the other internal vlans. Last I use another Linux bridge(4th physical port on my Intel I350) for my WAN port, which goes up to my ISP router and picks up DHCP there. I still have alot of config work to do but this is a great start and very exciting!

 

[spetrillo]

I am back at it and I have successfully built my OPNsense VM. It was a nerve wracking process bc I have done this many times before, only to have the vm die or PVE crash all together. Well not this time! For those of you wondering I took a screenshot of my config. I tried to use UEFI but it would not boot, so I went to SeaBIOS with Q35.

My OPNsense VM uses a Linux bridge to get to my mgmt vlan. It then uses PCIe passthrough(3 physical ports on my Intel I350) for all the other internal vlans. Last I use another Linux bridge(4th physical port on my Intel I350) for my WAN port, which goes up to my ISP router and picks up DHCP there. I still have alot of config work to do but this is a great start and very exciting!

Ok all you folks running SR-IOV....I need some help!

My overall PVE network configuration is as follows:

1) Oboard NIC is set for mgmt vlan, using a Linux bridge. Its also where my PVE gets its mgmt IP via DHCP, as a Linux bridge.1
2) First 3 ports of my Intel I350 card are setup in a Linux bond, and then the Linux bridge calling the bond is setup to be vlan aware for vlans 10,12,20,25,30. Each physical port has 4 vfs created, for a total of 12 vfs.
3) 4th port of my Intel I350 card is setup to connect directly to my upstream ISP router, with a Linux bridge set to manual.

I have attached my network config for review.

In my OPNsense VM network config I have attached vmbr1 as the first network interface, within my mgmt vlan. I will be setting this as a static IP. The 2nd thru 4th network interfaces are the 1st thru 3rd physical PCIe passthrough ports on my I350 card. These should give me access to vlans 10,12,20,25,30. The 5th network interface is vmbr3, which goes to my upstream router directly. Within OPNsense I will set it as DHCP, so my ISPs router will handle its IP.

My switch is set as follows:

1) Port 1 is setup as tagged for vlan 1. My onboard NIC is connected to this.
2) Ports 2-4 is setup as a LACP LAGG, with the LAGG setup for tagged vlans 10,12,20,25,30.

In the OPNsense config, if I were using Linux bridges the bridge would be set for the vlan I want. How do I do this when I am passing through the individual physical port, or does this rely on the LACP LAGG config? I think I am getting screwed up with the LAGG vs the physical port. Is there a way to mimic the LAGG on the switch, as a physical LAGG to be passed through in PVE? I don't think so but I am no expert.

Am I better off just splitting up the LAGG and configuring the physical switch ports to handle the vlans? Then I just pass through each pf? I am getting confused, so I am hoping someone can provide me some visibility!

 

[spetrillo]

Ok all you folks running SR-IOV....I need some help!

My overall PVE network configuration is as follows:

1) Oboard NIC is set for mgmt vlan, using a Linux bridge. Its also where my PVE gets its mgmt IP via DHCP, as a Linux bridge.1
2) First 3 ports of my Intel I350 card are setup in a Linux bond, and then the Linux bridge calling the bond is setup to be vlan aware for vlans 10,12,20,25,30. Each physical port has 4 vfs created, for a total of 12 vfs.
3) 4th port of my Intel I350 card is setup to connect directly to my upstream ISP router, with a Linux bridge set to manual.

I have attached my network config for review.

In my OPNsense VM network config I have attached vmbr1 as the first network interface, within my mgmt vlan. I will be setting this as a static IP. The 2nd thru 4th network interfaces are the 1st thru 3rd physical PCIe passthrough ports on my I350 card. These should give me access to vlans 10,12,20,25,30. The 5th network interface is vmbr3, which goes to my upstream router directly. Within OPNsense I will set it as DHCP, so my ISPs router will handle its IP.

My switch is set as follows:

1) Port 1 is setup as tagged for vlan 1. My onboard NIC is connected to this.
2) Ports 2-4 is setup as a LACP LAGG, with the LAGG setup for tagged vlans 10,12,20,25,30.

In the OPNsense config, if I were using Linux bridges the bridge would be set for the vlan I want. How do I do this when I am passing through the individual physical port, or does this rely on the LACP LAGG config? I think I am getting screwed up with the LAGG vs the physical port. Is there a way to mimic the LAGG on the switch, as a physical LAGG to be passed through in PVE? I don't think so but I am no expert.

Am I better off just splitting up the LAGG and configuring the physical switch ports to handle the vlans? Then I just pass through each pf? I am getting confused, so I am hoping someone can provide me some visibility!

I am leaving the above post, even though I really could have deleted it. I firmly believe that you need to walk away from something when you cannot seem to figure it out. Trying to plow through can lead to more problems and you end up getting really frustrated. So after I posted above I walked away until now. I was just finishing dinner when it came to me. OPNsense can set LAGGs within its network config process....duhh! I am redoing my OPNsense install and indeed yes I now have 5 vlans that will run across one LAGG. It was staring me right in the face but I was too frustrated to see it.

Install is going well and I will get all my interfaces set. I am then going to stop. I will wake up early tomorrow, so I can shutdown my current physical firewall and activate the new virtual.

I do have one question I hope someone could respond to. I am at the point of beginning to use my vf interfaces for my other vms. As mentioned 3 of the 4 physical interfaces are in a LACP LAGG. Does that mean this config trickles down to the vfs or could I use the Linux bridge that I setup, that calls the Linux bond of the 3 interfaces. Not sure how to allocate the vfs. I am also starting to think I might not need the vfs, since I was able to pass through the physical interfaces and use them as PCIe devices in my OPNsense.