Can't reach gateway when using VLAN

[fenugurod]

Hey folks I need some help on setting SR-IOV to work with VLAN. I'm kinda loosing my mind at the moment over the days and days that I've been debugging this problem and I would appreciate some help.

My NIC is an Intel I350-T4 NIC. SR-IOV is configured and I have LAN and WAN access at my network. The freaking problem starts when I try to setup VLANs, I simply can't reach pfSense from the VLAN. The switch and AP looks like to be ok, I can reach other nodes at the VLAN, when I set an static ip because I can't get a IP from DHCP, but I simply can't reach the gateway.

These are some of the warnings that I've seen at my system. Could those 'IOMMU: feature inconsistent' be a problem?

> dmesg | grep -e DMAR -e IOMMU

[    0.010929] ACPI: DMAR 0x0000000078630000 000088 (v02 INTEL  EDK2     00000002      01000013)
[    0.010957] ACPI: Reserving DMAR table memory at [mem 0x78630000-0x78630087]
[    0.069067] DMAR: IOMMU enabled
[    0.158812] DMAR: Host address width 39
[    0.158813] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.158816] DMAR: dmar0: reg_base_addr fed90000 ver 4:0 cap 1c0000c40660462 ecap 29a00f0505e
[    0.158817] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.158821] DMAR: dmar1: reg_base_addr fed91000 ver 5:0 cap d2008c40660462 ecap f050da
[    0.158822] DMAR: RMRR base: 0x0000007e000000 end: 0x000000807fffff
[    0.158824] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.158825] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.158826] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.160320] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.333267] pci 0000:00:02.0: DMAR: Skip IOMMU disabling for graphics
[    0.391278] DMAR: No ATSR found
[    0.391279] DMAR: No SATC found
[    0.391280] DMAR: IOMMU feature fl1gp_support inconsistent
[    0.391280] DMAR: IOMMU feature pgsel_inv inconsistent
[    0.391281] DMAR: IOMMU feature nwfs inconsistent
[    0.391281] DMAR: IOMMU feature dit inconsistent
[    0.391282] DMAR: IOMMU feature sc_support inconsistent
[    0.391282] DMAR: IOMMU feature dev_iotlb_support inconsistent
[    0.391282] DMAR: dmar0: Using Queued invalidation
[    0.391284] DMAR: dmar1: Using Queued invalidation
[    0.391893] DMAR: Intel(R) Virtualization Technology for Directed I/O

This is the dmesg output https://www.coderstool.com/cs/RrYQB7 there are some warnings there but I don't know to which extend those could be a problem. Except for this one that looks suspect:

igb 0000:05:00.3 enp5s0f3: malformed Tx packet detected and dropped, LVMMC:0x34000000

This is the part that caught my attention because I'm using enp5s0f3v0 as the LAN interface, which is working ok, and I'm creating a VLAN in pfSense on top of that interface.

This is my /etc/network/interfaces config:

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

auto enp5s0f1
iface enp5s0f1 inet static
    address 10.0.10.2/24
    gateway 10.0.10.1
    dns-nameservers 1.1.1.1
    dns-search internal

auto enp3s0
iface enp3s0 inet manual

auto enp5s0f0
iface enp5s0f0 inet manual

auto enp5s0f2
iface enp5s0f2 inet manual

auto enp5s0f3
iface enp5s0f3 inet manual

And this is my systemd service that I use to configure SR-IOV during boot:

[Unit]
Description=Script to enable NIC SR-IOV on boot

[Service]
Type=oneshot
ExecStart=/usr/bin/bash -c '/usr/bin/echo 2 > /sys/class/net/enp5s0f0/device/sriov_numvfs'
ExecStart=/usr/bin/bash -c '/usr/bin/echo 2 > /sys/class/net/enp5s0f1/device/sriov_numvfs'
ExecStart=/usr/bin/bash -c '/usr/bin/echo 2 > /sys/class/net/enp5s0f2/device/sriov_numvfs'
ExecStart=/usr/bin/bash -c '/usr/bin/echo 2 > /sys/class/net/enp5s0f3/device/sriov_numvfs'

# enp5s0f0
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp5s0f0 vf 0 mac a0:36:9f:7d:35:00'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp5s0f0 vf 1 mac a0:36:9f:7d:35:01'

# enp5s0f1
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp5s0f1 vf 0 mac a0:36:9f:7d:35:02'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp5s0f1 vf 1 mac a0:36:9f:7d:35:03'

# enp5s0f2
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp5s0f2 vf 0 mac a0:36:9f:7d:35:04'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp5s0f2 vf 1 mac a0:36:9f:7d:35:05'

# enp5s0f3
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp5s0f3 vf 0 mac a0:36:9f:7d:35:06'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set enp5s0f3 vf 1 mac a0:36:9f:7d:35:07'

[Install]
WantedBy=multi-user.target

 

[fenugurod]

So I did a few more tests. I disabled SR-IOV completely and passed the 2 NIC ports to pfSense and then everything started working, including the VLANs. The problem is then SR-IOV. Where is the best place that I can go to see if I can fix this problem? I don't know if is some configuration problem or even hardware.

 

[eracerxrs]

I figured I would comment on this thread since it came up in my search to help find a solution to assigning a VLAN to one of my VF's and passing that through directly to the VM.

I have two Proxmox hosts:
Host1: the one with the VM with the VF-VLAN passed into it.
Host2: the one with pfSense VM which is using a VF passed to it for LAN routing.

My problems were solved by putting all ip link configurations into the /etc/systemd/system/sriov.service files for execution at boot time. CLI commands didn't work.

The key entries in each sriov.service files which got it working for me were:

Host1:

[Unit]
Description=Enable SR-IOV and detach guest VFs from host
Requires=network.target
After=network.target
Before=pve-firewall.service
[Service]
Type=oneshot
RemainAfterExit=yes

#Set promiscuous mode for the physical interface
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set $DEV promisc on'

#Create NIC VF
ExecStart=/usr/bin/bash -c 'echo 1 > /sys/class/net/$DEV/device/sriov_numvfs'

#Set Static MAC for VF
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set $DEV vf 0 mac $MAC'

#Assign VLAN to VF
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set $DEV vf 0 vlan $VLAN'

[Install]
WantedBy=multi-user.target

Host2:

[Unit]
Description=Enable SR-IOV and detach guest VFs from host
Requires=network.target
After=network.target
Before=pve-firewall.service
[Service]
Type=oneshot
RemainAfterExit=yes

#Set promiscuous mode for the physical interface
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set $DEV promisc on'

#Create NIC VFs
ExecStart=/usr/bin/bash -c 'echo 1 > /sys/class/net/$DEV/device/sriov_numvfs'

#Set static MAC for VF
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set $DEV vf 0 mac $MAC'

#Set trust to allow pfSense to route VLANs
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set dev $DEV vf 0 trust on'

[Install]
WantedBy=multi-user.target

- I didn't need to disable spoof checking or add trust to the VF on host1.
- Make sure promiscuous mode is enabled at boot for any physical interfaces that will have VF's bound to it.
- Reboot the hosts for good measure don't rely on systemctl restart