Using SR-IOV VF in CT with DPDK

[ninjabes]

Dear community,

I am trying to use some VFs of a NIC in a CT. What I did so far:

1. Enable SR-IOV in bios
2. Enable IOMMU with pass through
3. Load kernel modules on startup (vfio, vfio_iommu_type1, vfio_pci, vfio_virqfd)
4. Spawn two VF via

echo 2 | sudo tee /sys/class/net/enp1s0f0/device/sriov_numvfs

Regarding DPDK I did in the CT:

1. Build dpdk 19.11 from source
2. Set up hugepages like this:

 # adding the following to the grub config
hugepagesz=1G hugepages=16 default_hugepagesz=1G
mkdir /mnt/huge
mount -t hugetlbfs pagesize=1GB /mnt/huge

3. Binding the two VF to to vfio_pci like this:

sudo ~/workspace/dpdk-19.11/usertools/dpdk-devbind.py --bind=vfio-pci 0000:01:10.0
sudo ~/workspace/dpdk-19.11/usertools/dpdk-devbind.py --bind=vfio-pci 0000:01:10.2

Using lspci I can see the VFs in the CT:

$ lspci
[...]
01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
01:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
01:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
[...]

Binding the VFs to a DPDK compatible driver (vfio-pci) worked as well:

$ sudo ~/workspace/dpdk-19.11/usertools/dpdk-devbind.py --status-dev net

Network devices using DPDK-compatible driver
============================================
0000:01:10.0 '82599 Ethernet Controller Virtual Function 10ed' drv=vfio-pci unused=ixgbevf
0000:01:10.2 '82599 Ethernet Controller Virtual Function 10ed' drv=vfio-pci unused=ixgbevf

Network devices using kernel driver
===================================
0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=enp1s0f0 drv=ixgbe unused=vfio-pci 
0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if= drv=ixgbe unused=vfio-pci 
0000:08:00.0 'Device 8125' if= drv=r8169 unused=vfio-pci

I had to change the LXC config a little to make the binding possible. I looks like this:

arch: amd64
cores: 16
hostname: oran-sample-app-ct-raw
memory: 16384
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=96:52:47:8B:73:B4,ip=dhcp,type=veth
ostype: ubuntu
rootfs: rfci-lxc-zfs:subvol-204-disk-0,size=30G
swap: 0
lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:
lxc.mount.auto: proc:rw sys:rw
lxc.mount.entry: /lib/modules lib/modules none bind 0 0
lxc.net.1.type: phys
lxc.net.1.link: enp1s0f0

The interface is visible and ip link show shows me the respective VFs:

$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 96:52:47:8b:73:b4 brd ff:ff:ff:ff:ff:ff link-netnsid 0
3: enp1s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 80:61:5f:0d:df:aa brd ff:ff:ff:ff:ff:ff
    vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
    vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off

But when I run my app which is using DPDK its priting the following error:

EAL: Detected 16 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Auto-detected process type: PRIMARY
EAL: Multi-process socket /var/run/dpdk/wls_0/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL:   cannot open VFIO container, error 1 (Operation not permitted)
EAL: VFIO support could not be initialized
EAL: PCI device 0000:01:10.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:10ed net_ixgbe_vf
EAL: Driver cannot attach the device (0000:01:10.0)
EAL: Failed to attach device on primary process
sample-app: Network port doesn't exist

Any ideas what I am doing wrong? Is there a way to test DPDK indipendent of the program I am using? Any help is appreciated!! Thanks in advance

Cheers,
ninjab3s

 

[kjkent]

VFs / PFs using vfio-pci require the use of a vfio-token (a uuid) which can be set on the PF, and the VFs need to supply it in order to be used. Unfortunately it seems that this hasn't been implemented in qemu yet.