Dear community,
I am trying to use some VFs of a NIC in a CT. What I did so far:
1. Enable SR-IOV in bios
2. Enable IOMMU with pass through
3. Load kernel modules on startup (vfio, vfio_iommu_type1, vfio_pci, vfio_virqfd)
4. Spawn two VF via
Regarding DPDK I did in the CT:
1. Build dpdk 19.11 from source
2. Set up hugepages like this:
3. Binding the two VF to to vfio_pci like this:
Using lspci I can see the VFs in the CT:
Binding the VFs to a DPDK compatible driver (vfio-pci) worked as well:
I had to change the LXC config a little to make the binding possible. I looks like this:
The interface is visible and ip link show shows me the respective VFs:
But when I run my app which is using DPDK its priting the following error:
Any ideas what I am doing wrong? Is there a way to test DPDK indipendent of the program I am using? Any help is appreciated!! Thanks in advance
Cheers,
ninjab3s
I am trying to use some VFs of a NIC in a CT. What I did so far:
1. Enable SR-IOV in bios
2. Enable IOMMU with pass through
3. Load kernel modules on startup (vfio, vfio_iommu_type1, vfio_pci, vfio_virqfd)
4. Spawn two VF via
Code:
echo 2 | sudo tee /sys/class/net/enp1s0f0/device/sriov_numvfsRegarding DPDK I did in the CT:
1. Build dpdk 19.11 from source
2. Set up hugepages like this:
Code:
# adding the following to the grub config
hugepagesz=1G hugepages=16 default_hugepagesz=1G
mkdir /mnt/huge
mount -t hugetlbfs pagesize=1GB /mnt/huge
Code:
sudo ~/workspace/dpdk-19.11/usertools/dpdk-devbind.py --bind=vfio-pci 0000:01:10.0
sudo ~/workspace/dpdk-19.11/usertools/dpdk-devbind.py --bind=vfio-pci 0000:01:10.2Using lspci I can see the VFs in the CT:
Code:
$ lspci
[...]
01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
01:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
01:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
[...]Binding the VFs to a DPDK compatible driver (vfio-pci) worked as well:
Code:
$ sudo ~/workspace/dpdk-19.11/usertools/dpdk-devbind.py --status-dev net
Network devices using DPDK-compatible driver
============================================
0000:01:10.0 '82599 Ethernet Controller Virtual Function 10ed' drv=vfio-pci unused=ixgbevf
0000:01:10.2 '82599 Ethernet Controller Virtual Function 10ed' drv=vfio-pci unused=ixgbevf
Network devices using kernel driver
===================================
0000:01:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=enp1s0f0 drv=ixgbe unused=vfio-pci
0000:01:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if= drv=ixgbe unused=vfio-pci
0000:08:00.0 'Device 8125' if= drv=r8169 unused=vfio-pciI had to change the LXC config a little to make the binding possible. I looks like this:
Code:
arch: amd64
cores: 16
hostname: oran-sample-app-ct-raw
memory: 16384
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=96:52:47:8B:73:B4,ip=dhcp,type=veth
ostype: ubuntu
rootfs: rfci-lxc-zfs:subvol-204-disk-0,size=30G
swap: 0
lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:
lxc.mount.auto: proc:rw sys:rw
lxc.mount.entry: /lib/modules lib/modules none bind 0 0
lxc.net.1.type: phys
lxc.net.1.link: enp1s0f0The interface is visible and ip link show shows me the respective VFs:
Code:
$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 96:52:47:8b:73:b4 brd ff:ff:ff:ff:ff:ff link-netnsid 0
3: enp1s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 80:61:5f:0d:df:aa brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss off
vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off, query_rss offBut when I run my app which is using DPDK its priting the following error:
Code:
EAL: Detected 16 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Auto-detected process type: PRIMARY
EAL: Multi-process socket /var/run/dpdk/wls_0/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: cannot open VFIO container, error 1 (Operation not permitted)
EAL: VFIO support could not be initialized
EAL: PCI device 0000:01:10.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10ed net_ixgbe_vf
EAL: Driver cannot attach the device (0000:01:10.0)
EAL: Failed to attach device on primary process
sample-app: Network port doesn't existAny ideas what I am doing wrong? Is there a way to test DPDK indipendent of the program I am using? Any help is appreciated!! Thanks in advance
Cheers,
ninjab3s
