Unable to create Trunk port [Please Help :)]

jimibob

New Member
Feb 7, 2022
11
0
1
73
Hello,

Really grateful for some assistance, pulling my hair out here...

Setup: dell r730 with quad port broadcom nic (OEM) and a mellanox connectx-3, proxmox 7.2
VM: sophos xg with 3 vmbr (2 wan, 1 lan (trunk)).

Issue: Trunk port works fine when I use one of my Dell r730 integrated quad port nic. When I change the vmbr3 network (the trunk port) to a mellanox connect-x 3 card I cannot access the firewall. Assume this is a config issue with the mellanox card but not sure where to begin diagnosing.

Interfaces file looks like so:

Code:
auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

iface enp5s0 inet manual

auto vmbr0
iface vmbr0 inet static
    address 192.168.200.75/24
    gateway 192.168.200.1
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0
    up /bin/ip link set vmbr0 promisc on

auto vmbr1
iface vmbr1 inet manual
    bridge-ports eno2
    bridge-stp off
    bridge-fd 0
#Starlink - port 4

auto vmbr2
iface vmbr2 inet manual
    bridge-ports eno3
    bridge-stp off
    bridge-fd 0
#Shell - port 2

auto vmbr3
iface vmbr3 inet manual
    bridge-ports enp5s0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 1 2 3 4 5 6 7 8 9 10 99 100 200
    up /bin/ip link set vmbr3 promisc on
#SophosXG-Trunk - sfp+


Please let me know any commands that would be useful to run in diagnosing.

Code:
root@proxmox:~# dmesg | grep mlx
[    2.881140] mlx4_core: Mellanox ConnectX core driver v4.0-0
[    2.881176] mlx4_core: Initializing 0000:05:00.0
[    8.924784] mlx4_core 0000:05:00.0: DMFS high rate steer mode is: disabled performance optimized steering
[    8.925045] mlx4_core 0000:05:00.0: 63.008 Gb/s available PCIe bandwidth (8.0GT/s PCIe x8 link)
[    9.070079] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0
[    9.070197] mlx4_en 0000:05:00.0: Activating port:1
[    9.075587] mlx4_en: 0000:05:00.0: Port 1: Using 32 TX rings
[    9.075593] mlx4_en: 0000:05:00.0: Port 1: Using 16 RX rings
[    9.075794] mlx4_en: 0000:05:00.0: Port 1: Initializing port
[    9.076772] mlx4_en 0000:05:00.0: registered PHC clock
[    9.077562] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v4.0-0
[    9.078173] mlx4_core 0000:05:00.0 enp5s0: renamed from eth0
[    9.078528] <mlx4_ib> mlx4_ib_add: counter index 1 for port 1 allocated 1
[   13.999058] mlx4_en: enp5s0: Link Up
[   27.431861] mlx4_en: enp5s0: Steering Mode 1
[   27.452870] mlx4_en: enp5s0: Link Up
[  564.519984] mlx4_en: enp5s0: Link Down

(link is down as it's now unplugged)

Code:
root@proxmox:~# lshw -C network
  *-network
       description: Ethernet interface
       product: MT27500 Family [ConnectX-3]
       vendor: Mellanox Technologies
       physical id: 0
       bus info: pci@0000:05:00.0
       logical name: enp5s0
       version: 00
       serial: 00:02:c9:17:36:c0
       capacity: 56Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm vpd msix pciexpress bus_master cap_list rom ethernet physical fibre 10000bt-fd 40000bt-fd 56000bt-fd autonegotiation
       configuration: autonegotiation=off broadcast=yes driver=mlx4_en driverversion=4.0-0 firmware=2.42.5000 latency=0 link=no multicast=yes port=fibre
       resources: irq:90 memory:92400000-924fffff memory:91000000-917fffff
  *-network:0
       description: Ethernet interface
       product: NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
       vendor: Broadcom Inc. and subsidiaries
       physical id: 0
       bus info: pci@0000:02:00.0
       logical name: eno3
       version: 00
       serial: 80:18:44:f3:55:3a
       capacity: 1Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm vpd msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=tg3 driverversion=5.15.39-1-pve firmware=FFV20.8.4 bc 5720-v1.39 latency=0 link=no multicast=yes port=twisted pair
       resources: irq:94 memory:92230000-9223ffff memory:92240000-9224ffff memory:92250000-9225ffff memory:92800000-9283ffff
  *-network:1 DISABLED
       description: Ethernet interface
       product: NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
       vendor: Broadcom Inc. and subsidiaries
       physical id: 0.1
       bus info: pci@0000:02:00.1
       logical name: eno4
       version: 00
       serial: 80:18:44:f3:55:3b
       capacity: 1Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm vpd msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=tg3 driverversion=5.15.39-1-pve firmware=FFV20.8.4 bc 5720-v1.39 latency=0 link=no multicast=yes port=twisted pair
       resources: irq:96 memory:92200000-9220ffff memory:92210000-9221ffff memory:92220000-9222ffff memory:92840000-9287ffff
  *-network:0
       description: Ethernet interface
       product: NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
       vendor: Broadcom Inc. and subsidiaries
       physical id: 0
       bus info: pci@0000:01:00.0
       logical name: eno1
       version: 00
       serial: 80:18:44:f3:55:38
       size: 1Gbit/s
       capacity: 1Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm vpd msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=tg3 driverversion=5.15.39-1-pve duplex=full firmware=FFV20.8.4 bc 5720-v1.39 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
       resources: irq:89 memory:92330000-9233ffff memory:92340000-9234ffff memory:92350000-9235ffff memory:92900000-9293ffff
  *-network:1
       description: Ethernet interface
       product: NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
       vendor: Broadcom Inc. and subsidiaries
       physical id: 0.1
       bus info: pci@0000:01:00.1
       logical name: eno2
       version: 00
       serial: 80:18:44:f3:55:39
       capacity: 1Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm vpd msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=tg3 driverversion=5.15.39-1-pve firmware=FFV20.8.4 bc 5720-v1.39 latency=0 link=no multicast=yes port=twisted pair
       resources: irq:92 memory:92300000-9230ffff memory:92310000-9231ffff memory:92320000-9232ffff memory:92940000-9297ffff
  *-network:0
       description: Ethernet interface
       physical id: 3
       logical name: vmbr0
       serial: 80:18:44:f3:55:38
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=bridge driverversion=2.3 firmware=N/A ip=192.168.200.75 link=yes multicast=yes promiscuous=yes speed=10Gbit/s
  *-network:1
       description: Ethernet interface
       physical id: 4
       logical name: vmbr1
       serial: 80:18:44:f3:55:39
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=bridge driverversion=2.3 firmware=N/A link=yes multicast=yes speed=10Gbit/s
  *-network:2
       description: Ethernet interface
       physical id: 5
       logical name: vmbr2
       serial: 80:18:44:f3:55:3a
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=bridge driverversion=2.3 firmware=N/A link=yes multicast=yes speed=10Gbit/s
  *-network:3
       description: Ethernet interface
       physical id: 6
       logical name: vmbr3
       serial: 00:02:c9:17:36:c0
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=bridge driverversion=2.3 firmware=N/A link=yes multicast=yes promiscuous=yes speed=10Gbit/s
  *-network:4
       description: Ethernet interface
       physical id: 7
       logical name: tap101i0
       serial: d2:4b:06:f8:a3:54
       size: 10Mbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=tun driverversion=1.6 duplex=full link=yes multicast=yes port=twisted pair promiscuous=yes speed=10Mbit/s
  *-network:5
       description: Ethernet interface
       physical id: 8
       logical name: fwbr101i0
       serial: fe:08:37:84:c6:11
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=bridge driverversion=2.3 firmware=N/A link=yes multicast=yes speed=10Gbit/s
  *-network:6
       description: Ethernet interface
       physical id: 9
       logical name: fwpr101p0
       serial: f2:81:0a:5a:19:99
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=veth driverversion=1.0 duplex=full link=yes multicast=yes port=twisted pair speed=10Gbit/s
  *-network:7
       description: Ethernet interface
       physical id: a
       logical name: fwln101i0
       serial: fe:52:12:b7:88:3a
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=veth driverversion=1.0 duplex=full link=yes multicast=yes port=twisted pair speed=10Gbit/s
  *-network:8
       description: Ethernet interface
       physical id: b
       logical name: tap101i1
       serial: ba:02:51:e0:33:bf
       size: 10Mbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=tun driverversion=1.6 duplex=full link=yes multicast=yes port=twisted pair promiscuous=yes speed=10Mbit/s
  *-network:9
       description: Ethernet interface
       physical id: c
       logical name: fwbr101i1
       serial: fe:01:c6:0a:73:46
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=bridge driverversion=2.3 firmware=N/A link=yes multicast=yes speed=10Gbit/s
  *-network:10
       description: Ethernet interface
       physical id: d
       logical name: fwpr101p1
       serial: 7a:c0:e8:ed:1c:7d
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=veth driverversion=1.0 duplex=full link=yes multicast=yes port=twisted pair speed=10Gbit/s
  *-network:11
       description: Ethernet interface
       physical id: e
       logical name: fwln101i1
       serial: 6e:25:61:26:29:18
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=veth driverversion=1.0 duplex=full link=yes multicast=yes port=twisted pair speed=10Gbit/s
  *-network:12
       description: Ethernet interface
       physical id: f
       logical name: tap101i2
       serial: ca:75:51:2f:a7:c7
       size: 10Mbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=tun driverversion=1.6 duplex=full link=yes multicast=yes port=twisted pair promiscuous=yes speed=10Mbit/s
  *-network:13
       description: Ethernet interface
       physical id: 10
       logical name: fwbr101i2
       serial: 8e:1e:85:5e:3d:c3
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=bridge driverversion=2.3 firmware=N/A link=yes multicast=yes speed=10Gbit/s
  *-network:14
       description: Ethernet interface
       physical id: 11
       logical name: fwpr101p2
       serial: 9a:7a:58:12:4c:9c
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=veth driverversion=1.0 duplex=full link=yes multicast=yes port=twisted pair speed=10Gbit/s
  *-network:15
       description: Ethernet interface
       physical id: 12
       logical name: fwln101i2
       serial: 72:d2:b9:53:45:0d
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=veth driverversion=1.0 duplex=full link=yes multicast=yes port=twisted pair speed=10Gbit/s
  *-network:16
       description: Ethernet interface
       physical id: 13
       logical name: tap500i0
       serial: 8e:db:c7:e1:33:cd
       size: 10Mbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=tun driverversion=1.6 duplex=full link=yes multicast=yes port=twisted pair promiscuous=yes speed=10Mbit/s
  *-network:17
       description: Ethernet interface
       physical id: 14
       logical name: fwbr500i0
       serial: 86:d0:88:eb:21:e2
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=bridge driverversion=2.3 firmware=N/A link=yes multicast=yes speed=10Gbit/s
  *-network:18
       description: Ethernet interface
       physical id: 15
       logical name: fwpr500p0
       serial: 9a:ef:84:39:dd:4d
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=veth driverversion=1.0 duplex=full link=yes multicast=yes port=twisted pair speed=10Gbit/s
  *-network:19
       description: Ethernet interface
       physical id: 16
       logical name: fwln500i0
       serial: ca:3c:c7:27:48:2f
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=veth driverversion=1.0 duplex=full link=yes multicast=yes port=twisted pair speed=10Gbit/s
 
Last edited:

jimibob

New Member
Feb 7, 2022
11
0
1
73
Bump - really struggling to make this work and nothing obvious. Please, any advice?
 

bobmc

Renowned Member
May 17, 2018
644
106
63
65
Maybe I'm not reading this correctly but from what you've posted, the Mellanox only has a single port?
 

jimibob

New Member
Feb 7, 2022
11
0
1
73
Maybe I'm not reading this correctly but from what you've posted, the Mellanox only has a single port?
Correct, I wanted to give Sophos 2 1Gb NICs from the integrated for dual wan, and a 10Gb Mellanox for the trunk into the switch.

The dual wan works fine on integrated NIC, but the Mellanox doesn't work for the trunk. I've had to use all integrated NICs for the ports.

If you're coming here from Reddit I have a dual port Mellanox and a single port Mellanox. Both the same card just dual and single. Trying to get the single port working first but couldn't, so tried the double in case the single was broken. Neither are working, any scans or commands I can run to debug. As mentioned, this works fine on hyper-v 2019.
 

bobmc

Renowned Member
May 17, 2018
644
106
63
65
Are you trying to connect hosts directly (without an intermediate switch)?
 

jimibob

New Member
Feb 7, 2022
11
0
1
73
No, it will be plugged into the switch where the port is set to allow all vlans. The switch hasn't changed at all since migrating from hyper-v a few days ago. The sfp+ port has the same profile as the rj45 port (which is currently working). I think it's something to do with proxmox and Mellanox, or Mellanox card itself.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!