Hello,
Really grateful for some assistance, pulling my hair out here...
Setup: dell r730 with quad port broadcom nic (OEM) and a mellanox connectx-3, proxmox 7.2
VM: sophos xg with 3 vmbr (2 wan, 1 lan (trunk)).
Issue: Trunk port works fine when I use one of my Dell r730 integrated quad port nic. When I change the vmbr3 network (the trunk port) to a mellanox connect-x 3 card I cannot access the firewall. Assume this is a config issue with the mellanox card but not sure where to begin diagnosing.
Interfaces file looks like so:
Please let me know any commands that would be useful to run in diagnosing.
(link is down as it's now unplugged)
Really grateful for some assistance, pulling my hair out here...
Setup: dell r730 with quad port broadcom nic (OEM) and a mellanox connectx-3, proxmox 7.2
VM: sophos xg with 3 vmbr (2 wan, 1 lan (trunk)).
Issue: Trunk port works fine when I use one of my Dell r730 integrated quad port nic. When I change the vmbr3 network (the trunk port) to a mellanox connect-x 3 card I cannot access the firewall. Assume this is a config issue with the mellanox card but not sure where to begin diagnosing.
Interfaces file looks like so:
Code:
auto lo
iface lo inet loopback
iface eno1 inet manual
iface eno2 inet manual
iface eno3 inet manual
iface eno4 inet manual
iface enp5s0 inet manual
auto vmbr0
iface vmbr0 inet static
address 192.168.200.75/24
gateway 192.168.200.1
bridge-ports eno1
bridge-stp off
bridge-fd 0
up /bin/ip link set vmbr0 promisc on
auto vmbr1
iface vmbr1 inet manual
bridge-ports eno2
bridge-stp off
bridge-fd 0
#Starlink - port 4
auto vmbr2
iface vmbr2 inet manual
bridge-ports eno3
bridge-stp off
bridge-fd 0
#Shell - port 2
auto vmbr3
iface vmbr3 inet manual
bridge-ports enp5s0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 1 2 3 4 5 6 7 8 9 10 99 100 200
up /bin/ip link set vmbr3 promisc on
#SophosXG-Trunk - sfp+Please let me know any commands that would be useful to run in diagnosing.
Code:
root@proxmox:~# dmesg | grep mlx
[ 2.881140] mlx4_core: Mellanox ConnectX core driver v4.0-0
[ 2.881176] mlx4_core: Initializing 0000:05:00.0
[ 8.924784] mlx4_core 0000:05:00.0: DMFS high rate steer mode is: disabled performance optimized steering
[ 8.925045] mlx4_core 0000:05:00.0: 63.008 Gb/s available PCIe bandwidth (8.0GT/s PCIe x8 link)
[ 9.070079] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0
[ 9.070197] mlx4_en 0000:05:00.0: Activating port:1
[ 9.075587] mlx4_en: 0000:05:00.0: Port 1: Using 32 TX rings
[ 9.075593] mlx4_en: 0000:05:00.0: Port 1: Using 16 RX rings
[ 9.075794] mlx4_en: 0000:05:00.0: Port 1: Initializing port
[ 9.076772] mlx4_en 0000:05:00.0: registered PHC clock
[ 9.077562] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v4.0-0
[ 9.078173] mlx4_core 0000:05:00.0 enp5s0: renamed from eth0
[ 9.078528] <mlx4_ib> mlx4_ib_add: counter index 1 for port 1 allocated 1
[ 13.999058] mlx4_en: enp5s0: Link Up
[ 27.431861] mlx4_en: enp5s0: Steering Mode 1
[ 27.452870] mlx4_en: enp5s0: Link Up
[ 564.519984] mlx4_en: enp5s0: Link Down(link is down as it's now unplugged)
Code:
root@proxmox:~# lshw -C network
*-network
description: Ethernet interface
product: MT27500 Family [ConnectX-3]
vendor: Mellanox Technologies
physical id: 0
bus info: pci@0000:05:00.0
logical name: enp5s0
version: 00
serial: 00:02:c9:17:36:c0
capacity: 56Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm vpd msix pciexpress bus_master cap_list rom ethernet physical fibre 10000bt-fd 40000bt-fd 56000bt-fd autonegotiation
configuration: autonegotiation=off broadcast=yes driver=mlx4_en driverversion=4.0-0 firmware=2.42.5000 latency=0 link=no multicast=yes port=fibre
resources: irq:90 memory:92400000-924fffff memory:91000000-917fffff
*-network:0
description: Ethernet interface
product: NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
vendor: Broadcom Inc. and subsidiaries
physical id: 0
bus info: pci@0000:02:00.0
logical name: eno3
version: 00
serial: 80:18:44:f3:55:3a
capacity: 1Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm vpd msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=tg3 driverversion=5.15.39-1-pve firmware=FFV20.8.4 bc 5720-v1.39 latency=0 link=no multicast=yes port=twisted pair
resources: irq:94 memory:92230000-9223ffff memory:92240000-9224ffff memory:92250000-9225ffff memory:92800000-9283ffff
*-network:1 DISABLED
description: Ethernet interface
product: NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
vendor: Broadcom Inc. and subsidiaries
physical id: 0.1
bus info: pci@0000:02:00.1
logical name: eno4
version: 00
serial: 80:18:44:f3:55:3b
capacity: 1Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm vpd msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=tg3 driverversion=5.15.39-1-pve firmware=FFV20.8.4 bc 5720-v1.39 latency=0 link=no multicast=yes port=twisted pair
resources: irq:96 memory:92200000-9220ffff memory:92210000-9221ffff memory:92220000-9222ffff memory:92840000-9287ffff
*-network:0
description: Ethernet interface
product: NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
vendor: Broadcom Inc. and subsidiaries
physical id: 0
bus info: pci@0000:01:00.0
logical name: eno1
version: 00
serial: 80:18:44:f3:55:38
size: 1Gbit/s
capacity: 1Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm vpd msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=tg3 driverversion=5.15.39-1-pve duplex=full firmware=FFV20.8.4 bc 5720-v1.39 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
resources: irq:89 memory:92330000-9233ffff memory:92340000-9234ffff memory:92350000-9235ffff memory:92900000-9293ffff
*-network:1
description: Ethernet interface
product: NetXtreme BCM5720 2-port Gigabit Ethernet PCIe
vendor: Broadcom Inc. and subsidiaries
physical id: 0.1
bus info: pci@0000:01:00.1
logical name: eno2
version: 00
serial: 80:18:44:f3:55:39
capacity: 1Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm vpd msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=tg3 driverversion=5.15.39-1-pve firmware=FFV20.8.4 bc 5720-v1.39 latency=0 link=no multicast=yes port=twisted pair
resources: irq:92 memory:92300000-9230ffff memory:92310000-9231ffff memory:92320000-9232ffff memory:92940000-9297ffff
*-network:0
description: Ethernet interface
physical id: 3
logical name: vmbr0
serial: 80:18:44:f3:55:38
size: 10Gbit/s
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=bridge driverversion=2.3 firmware=N/A ip=192.168.200.75 link=yes multicast=yes promiscuous=yes speed=10Gbit/s
*-network:1
description: Ethernet interface
physical id: 4
logical name: vmbr1
serial: 80:18:44:f3:55:39
size: 10Gbit/s
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=bridge driverversion=2.3 firmware=N/A link=yes multicast=yes speed=10Gbit/s
*-network:2
description: Ethernet interface
physical id: 5
logical name: vmbr2
serial: 80:18:44:f3:55:3a
size: 10Gbit/s
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=bridge driverversion=2.3 firmware=N/A link=yes multicast=yes speed=10Gbit/s
*-network:3
description: Ethernet interface
physical id: 6
logical name: vmbr3
serial: 00:02:c9:17:36:c0
size: 10Gbit/s
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=bridge driverversion=2.3 firmware=N/A link=yes multicast=yes promiscuous=yes speed=10Gbit/s
*-network:4
description: Ethernet interface
physical id: 7
logical name: tap101i0
serial: d2:4b:06:f8:a3:54
size: 10Mbit/s
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=tun driverversion=1.6 duplex=full link=yes multicast=yes port=twisted pair promiscuous=yes speed=10Mbit/s
*-network:5
description: Ethernet interface
physical id: 8
logical name: fwbr101i0
serial: fe:08:37:84:c6:11
size: 10Gbit/s
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=bridge driverversion=2.3 firmware=N/A link=yes multicast=yes speed=10Gbit/s
*-network:6
description: Ethernet interface
physical id: 9
logical name: fwpr101p0
serial: f2:81:0a:5a:19:99
size: 10Gbit/s
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=veth driverversion=1.0 duplex=full link=yes multicast=yes port=twisted pair speed=10Gbit/s
*-network:7
description: Ethernet interface
physical id: a
logical name: fwln101i0
serial: fe:52:12:b7:88:3a
size: 10Gbit/s
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=veth driverversion=1.0 duplex=full link=yes multicast=yes port=twisted pair speed=10Gbit/s
*-network:8
description: Ethernet interface
physical id: b
logical name: tap101i1
serial: ba:02:51:e0:33:bf
size: 10Mbit/s
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=tun driverversion=1.6 duplex=full link=yes multicast=yes port=twisted pair promiscuous=yes speed=10Mbit/s
*-network:9
description: Ethernet interface
physical id: c
logical name: fwbr101i1
serial: fe:01:c6:0a:73:46
size: 10Gbit/s
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=bridge driverversion=2.3 firmware=N/A link=yes multicast=yes speed=10Gbit/s
*-network:10
description: Ethernet interface
physical id: d
logical name: fwpr101p1
serial: 7a:c0:e8:ed:1c:7d
size: 10Gbit/s
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=veth driverversion=1.0 duplex=full link=yes multicast=yes port=twisted pair speed=10Gbit/s
*-network:11
description: Ethernet interface
physical id: e
logical name: fwln101i1
serial: 6e:25:61:26:29:18
size: 10Gbit/s
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=veth driverversion=1.0 duplex=full link=yes multicast=yes port=twisted pair speed=10Gbit/s
*-network:12
description: Ethernet interface
physical id: f
logical name: tap101i2
serial: ca:75:51:2f:a7:c7
size: 10Mbit/s
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=tun driverversion=1.6 duplex=full link=yes multicast=yes port=twisted pair promiscuous=yes speed=10Mbit/s
*-network:13
description: Ethernet interface
physical id: 10
logical name: fwbr101i2
serial: 8e:1e:85:5e:3d:c3
size: 10Gbit/s
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=bridge driverversion=2.3 firmware=N/A link=yes multicast=yes speed=10Gbit/s
*-network:14
description: Ethernet interface
physical id: 11
logical name: fwpr101p2
serial: 9a:7a:58:12:4c:9c
size: 10Gbit/s
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=veth driverversion=1.0 duplex=full link=yes multicast=yes port=twisted pair speed=10Gbit/s
*-network:15
description: Ethernet interface
physical id: 12
logical name: fwln101i2
serial: 72:d2:b9:53:45:0d
size: 10Gbit/s
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=veth driverversion=1.0 duplex=full link=yes multicast=yes port=twisted pair speed=10Gbit/s
*-network:16
description: Ethernet interface
physical id: 13
logical name: tap500i0
serial: 8e:db:c7:e1:33:cd
size: 10Mbit/s
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=tun driverversion=1.6 duplex=full link=yes multicast=yes port=twisted pair promiscuous=yes speed=10Mbit/s
*-network:17
description: Ethernet interface
physical id: 14
logical name: fwbr500i0
serial: 86:d0:88:eb:21:e2
size: 10Gbit/s
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=bridge driverversion=2.3 firmware=N/A link=yes multicast=yes speed=10Gbit/s
*-network:18
description: Ethernet interface
physical id: 15
logical name: fwpr500p0
serial: 9a:ef:84:39:dd:4d
size: 10Gbit/s
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=veth driverversion=1.0 duplex=full link=yes multicast=yes port=twisted pair speed=10Gbit/s
*-network:19
description: Ethernet interface
physical id: 16
logical name: fwln500i0
serial: ca:3c:c7:27:48:2f
size: 10Gbit/s
capabilities: ethernet physical
configuration: autonegotiation=off broadcast=yes driver=veth driverversion=1.0 duplex=full link=yes multicast=yes port=twisted pair speed=10Gbit/s
Last edited: