Network and Bridge help

E

Exodes

New Member
Sep 9, 2024
2
0
1
I've spent a few days troubleshooting this and starting to go in circles, I don't know if I've missed something obvious, or I just don't understand Proxmox bridges after several tutorial watches.

Running Proxmox 8.2.4 - I had a GB NIC I was able to assign an IP too, then after watching tutorials I built a bridge, removing my 1GB NIC and assigned my 10GB NIC to it, and it seemed to be working. But when I was doing validations, I found when I disabled by 10GB NIC on the switch port, nothing happened, Proxmox stayed available. Shutting down the 1GB link takes down the server every time, even though VMBR0 (with the gateway) is now only assigned to the 10GB NIC. Im sure I had eth1 as the origional 1GB link, but trying to assign it to VMBR0 just results in connection log errors in Proxmox. It now looks like the 10GB NIC and the 1GB NIC share a MAC address, which is totally messing with me. I tried adding another 1GB bridge with a new IP, and it responds, but if I disable VMBR0, or shut the port, all connections are refused, I'm assuming because the gateway is on VMBR0.
Watching my interfaces on the Cisco switch, when I reboot Proxmox, the TenGB interfaces come up briefly then go down when Proxmox boots, I was wondering if the bridge with 2 interfaces causing a spanning tree issue back to the switch, but couldn't isolate the problem.

To bring a long story short my questions are:
How do you identify which physical NIC ports are in use by a bridge? - my current config does not list the 1GB nic, but if I drop that port all connections fail.
Do I need to separate the current NIC pairing so I can use the 10GB connection as the default? and if so, how?
My Cisco 3850 has bpduguard enabled, is it possibly sensing the bridge as a spanning tree issue?

Bridge Setup:


1725864867224.png

ip address output:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: eno8303: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
link/ether b4:45:06:fa:61:c6 brd ff:ff:ff:ff:ff:ff
altname enp4s0f0
3: ens2f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr1 state DOWN group default qlen 1000
link/ether b4:96:91:fe:6a:ac brd ff:ff:ff:ff:ff:ff
altname enp1s0f0
4: eno8403: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether b4:45:06:fa:61:c7 brd ff:ff:ff:ff:ff:ff
altname enp4s0f1
altname ens2f1
5: eth3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether b4:96:91:fe:6a:ad brd ff:ff:ff:ff:ff:ff
altname enp1s0f1
6: ens2f2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr1 state DOWN group default qlen 1000
link/ether b4:96:91:fe:6a:ae brd ff:ff:ff:ff:ff:ff
altname enp1s0f2
7: ens3f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether b4:96:91:08:db:b0 brd ff:ff:ff:ff:ff:ff
altname enp49s0f0
8: ens2f3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr1 state DOWN group default qlen 1000
link/ether b4:96:91:fe:6a:af brd ff:ff:ff:ff:ff:ff
altname enp1s0f3
9: ens3f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether b4:96:91:08:db:b2 brd ff:ff:ff:ff:ff:ff
altname enp49s0f1
10: idrac: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether b4:45:06:fa:61:c3 brd ff:ff:ff:ff:ff:ff
11: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether b4:45:06:fa:61:c6 brd ff:ff:ff:ff:ff:ff
inet 10.0.69.10/24 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::b645:6ff:fefa:61c6/64 scope link
valid_lft forever preferred_lft forever
12: vmbr1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether b4:96:91:fe:6a:ac brd ff:ff:ff:ff:ff:ff
inet 10.0.69.13/24 scope global vmbr1
valid_lft forever preferred_lft forever

/etc/network/interfaces file:

1725865625951.png


Show ARP:
Internet 10.0.69.10 27 b445.06fa.61c6 ARPA Vlan600
Internet 10.0.69.13 117 b445.06fa.61c6 ARPA Vlan600

mac address-table: *Gi 1/0/13 and TenGi 1/1/2 should show as well, but don't. 1/0/37 is the origional 1GB link
600 b445.06fa.61c6 DYNAMIC Gi1/0/37
 
First, I would recommend you to not configure the same subnet on two different interfaces if you don't absolutely need it. From my perspective, you would be much better off with an active-backup bonding configuration across your two bridges. I generally wouldn't recommend configuring two different IPs of the same subnet on two different Interfaces if you don't absolutely need it for a very specific setup. The reason being:

If you check your
INI: 
ip route
output, it'll probably look like this:

Code: 
...
10.0.69.0/24 dev vmbr0 proto kernel scope link src 10.0.69.10
10.0.69.0/24 dev vmbr1 proto kernel scope link src 10.0.69.13
...

So if you send a request to 10.0.69.13, it'll answer over vmbr0 since it's the first entry of the routing table and their priority is the same (which can't be changed since it's directly connected), only when vmbr0 is not answering you'll answer over vmbr1. If you configure 10.0.69.10 on an active-backup bonding interface, you'll only need one IP address and still achieve your backup. There are other ways to specifically route packages from a source address over a different interface, but unnecessary if you don't need it. You have to remove the VLAN configuration from the vmbridges as well and put it onto the bonding interface.
 
KevinS said:
First, I would recommend you to not configure the same subnet on two different interfaces if you don't absolutely need it. From my perspective, you would be much better off with an active-backup bonding configuration across your two bridges. I generally wouldn't recommend configuring two different IPs of the same subnet on two different Interfaces if you don't absolutely need it for a very specific setup. The reason being:

If you check your
INI: 
ip route
output, it'll probably look like thigs:

Code: 
...
10.0.69.0/24 dev vmbr0 proto kernel scope link src 10.0.69.10
10.0.69.0/24 dev vmbr1 proto kernel scope link src 10.0.69.13
...

So if you send a request to 10.0.69.13, it'll answer over vmbr0 since it's the first entry of the routing table and their priority is the same (which can't be changed since it's directly connected), only when vmbr0 is not answering you'll answer over vmbr1. If you configure 10.0.69.10 on an active-backup bonding interface, you'll only need one IP address and still achieve your backup. There are other ways to specifically route packages from a source address over a different interface, but unnecessary if you don't need it. You have to remove the VLAN configuration from the vmbridges as well and put it onto the bonding interface.
Click to expand...
ip route shows :

root@proxmoxve:~# ip route
default via 10.0.69.1 dev vmbr0 proto kernel onlink
10.0.69.0/24 dev vmbr0 proto kernel scope link src 10.0.69.10
10.0.69.0/24 dev vmbr1 proto kernel scope link src 10.0.69.13 linkdown
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back