SR-IOV with Intel X520 82599ES Card and i3-6100TE

4

4920441

Member
Dec 7, 2021
62
7
13
55
Hi,

I just wanted to try out the SR-IOV functions, because I thought it would be a nice to have for internal networkin of the 10 GBIt Ports of my ex-Sophos Router which is equipped with a Flexport aka 4 Port X520 Intel Network Adapter.

In the Bios of the old Sophos there is nothing about SR-IOV but, all prerequisites are set, it seems, because I can see the dmesg output which confirms my settings


Code: 
dmesg | grep -i "SR-IOV"
[   11.113975] ixgbe 0000:05:00.0 enp5s0f0: SR-IOV enabled with 4 VFs
[   11.278829] ixgbe 0000:05:00.1 enp5s0f1: SR-IOV enabled with 4 VFs
[   11.447224] ixgbe 0000:06:00.0 enp6s0f0: SR-IOV enabled with 4 VFs
[   11.614671] ixgbe 0000:06:00.1 enp6s0f1: SR-IOV enabled with 4 VFs

And even the VIO Network cards are set up correctly:

Code: 
12: enp5s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master Flex1 state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:1c:7f:61:f1:fe brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off, query_rss off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off, query_rss off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off, query_rss off
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off, query_rss off
13: enp5s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master Flex2 state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:1c:7f:61:f1:ff brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off, query_rss off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off, query_rss off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off, query_rss off
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off, query_rss off
14: enp6s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master Flex3 state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:1c:7f:61:f2:00 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off, query_rss off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off, query_rss off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off, query_rss off
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off, query_rss off
15: enp6s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master Flex4 state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:1c:7f:61:f2:01 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off, query_rss off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off, query_rss off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off, query_rss off
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off, query_rss off


lspci also shows me for each real port 4 viritualized ones as well:

Code: 
05:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
05:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
05:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
06:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
06:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
06:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
06:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
06:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
06:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
06:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
06:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
06:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
06:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)


So far so good...

The funny thing is, when I try to set up the SR-IOV Feature in Proxmox (regardless if as a mapped group or as a single vio pci device) everythings seeeeems to goes smoothly....

1749579498537.png


But it doesn't:

When i try to start the container wit the mapped VIO Device it complains about a "no such host device" - wait? wasn't it there for a second?


Code: 
WARN: no efidisk configured! Using temporary efivars disk.
Use of uninitialized value $name in concatenation (.) or string at /usr/share/perl5/PVE/SysFSTools.pm line 315.
Use of uninitialized value $name in concatenation (.) or string at /usr/share/perl5/PVE/SysFSTools.pm line 315.
Use of uninitialized value $name in concatenation (.) or string at /usr/share/perl5/PVE/SysFSTools.pm line 315.
Use of uninitialized value $name in concatenation (.) or string at /usr/share/perl5/PVE/SysFSTools.pm line 315.
Use of uninitialized value $name in concatenation (.) or string at /usr/share/perl5/PVE/SysFSTools.pm line 315.
failed to reset PCI device '0000:05:10.2', but trying to continue as not all devices need a reset
kvm: -device vfio-pci,host=0000:05:10.2,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0: vfio /sys/bus/pci/devices/0000:05:10.2: no such host device: No such file or directory
no PCI device found for '0000:05:10.2'
TASK ERROR: start failed: QEMU exited with code 1



Anyway, he is right! All VIO Devices are gone now, either on NIC level,



Code: 
10: enp16s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 7c:5a:1c:51:82:88 brd ff:ff:ff:ff:ff:ff
11: enp17s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 7c:5a:1c:51:82:89 brd ff:ff:ff:ff:ff:ff
16: enp18s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master Port11 state DOWN mode DEFAULT group default qlen 1000
    link/ether 7c:5a:1c:51:82:8a brd ff:ff:ff:ff:ff:ff
17: enp18s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master Port12 state DOWN mode DEFAULT group default qlen 1000
    link/ether 7c:5a:1c:51:82:8b brd ff:ff:ff:ff:ff:ff
18: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether 7c:5a:1c:51:82:80 brd ff:ff:ff:ff:ff:ff
19: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 7c:5a:1c:51:82:80 brd ff:ff:ff:ff:ff:ff
20: Port11: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 7c:5a:1c:51:82:8a brd ff:ff:ff:ff:ff:ff
21: Port12: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 7c:5a:1c:51:82:8b brd ff:ff:ff:ff:ff:ff
22: Flex1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:1c:7f:61:f1:fe brd ff:ff:ff:ff:ff:ff
23: Flex2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 00:1c:7f:61:f1:ff brd ff:ff:ff:ff:ff:ff
24: Flex3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:1c:7f:61:f2:00 brd ff:ff:ff:ff:ff:ff
25: Flex4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:1c:7f:61:f2:01 brd ff:ff:ff:ff:ff:ff

Or on PCI Level:

Code: 
05:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
06:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
06:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

What is even more frustrating, despite the kvm Error in Proxmox I don't get *ANY* Log entry at allo, nor a dmesg error


Code: 
[  482.131682] ixgbe 0000:05:00.1: removed PHC on enp5s0f1
[  482.409759] Flex2: port 1(enp5s0f1) entered disabled state
[  482.410779] ixgbe 0000:05:00.1 enp5s0f1 (unregistering): left allmulticast mode
[  482.410785] ixgbe 0000:05:00.1 enp5s0f1 (unregistering): left promiscuous mode
[  482.410790] Flex2: port 1(enp5s0f1) entered disabled state
[  482.460360] ixgbe 0000:05:00.1: complete
[  483.603716] ixgbe 0000:06:00.0: removed PHC on enp6s0f0
[  483.881668] Flex3: port 1(enp6s0f0) entered disabled state
[  483.882484] ixgbe 0000:06:00.0 enp6s0f0 (unregistering): left allmulticast mode
[  483.882652] Flex3: port 1(enp6s0f0) entered disabled state
[  483.934430] ixgbe 0000:06:00.0: complete
[  485.075685] ixgbe 0000:05:00.0: removed PHC on enp5s0f0
[  485.353643] Flex1: port 1(enp5s0f0) entered disabled state
[  485.354464] ixgbe 0000:05:00.0 enp5s0f0 (unregistering): left allmulticast mode
[  485.354631] Flex1: port 1(enp5s0f0) entered disabled state
[  485.419449] ixgbe 0000:05:00.0: complete
[  486.547774] ixgbe 0000:06:00.1: removed PHC on enp6s0f1
[  486.825713] Flex4: port 1(enp6s0f1) entered disabled state
[  486.826461] ixgbe 0000:06:00.1 enp6s0f1 (unregistering): left allmulticast mode
[  486.826647] Flex4: port 1(enp6s0f1) entered disabled state
[  486.887491] ixgbe 0000:06:00.1: complete
[  487.781057] tap100i0: entered promiscuous mode
[  487.801122] vmbr0: port 3(tap100i0) entered blocking state
[  487.801130] vmbr0: port 3(tap100i0) entered disabled state
[  487.801150] tap100i0: entered allmulticast mode
[  487.801208] vmbr0: port 3(tap100i0) entered blocking state
[  487.801212] vmbr0: port 3(tap100i0) entered forwarding state
[  488.055392] tap100i0: left allmulticast mode
[  488.055413] vmbr0: port 3(tap100i0) entered disabled state




Only that 5.00 and 6.00 are unregistered and gone.... no hint why that happens??!


There a two possibilities:

1.) I am doing something considerebly wrong
2.) my hardware -despite it seems to be capable- is not capable......


Any help is appreciated!


Since it is an old Sophos Firewall, I think a firmware update for the UEFI/BIOS is out of reach.... if it does not work, it does not work. It is, what it is....;-)

Thanks a lot!

Cheers

4920441


P.S: I use PVE CE latest, all updates with PVE kernel 6.8.12-11-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-11 (2025-05-22T09:39Z) x86_64 GNU/Linux
P.P.S: my boot cmdline is - after lots of sensible editing "
GRUB_CMDLINE_LINUX_DEFAULT="mitigations=off intel_pstate=passive intel_iommu=on iommu=pt initcall_blacklist=sysfb_init usbcore.autosuspend=-1 split_lock_detect=off pci=realloc assign-busses"

But Nevertheless, it does not cahnge anything of the behavior at all.....
 
Last edited:
Hi,

i talked to some collegues which had the same task as a POC but with newer NICs from Mellanox (100G) and they said to me, it is not really worth the effort if you don't need severel 100 Gig througput on modern Epyc Hardware.

The Issues and potentical Problem - especially across a clustered installation - are somtimes really cumbersome, so it seems its better to keep it simple with ovs or linux-bridges to begin with.

In my case it does not seem to work at all, which is a bummer, but even if it does, what are the net benefits? Only lower CPU Load?

Cheers

4920441
 
You must log in or register to reply here.
Top Bottom