Help with Nic Bond

[jwooton]

Hello,

Relatively new to this stuff, and I'm hoping to get some help with network settings. I have a server with 7 total nics, one of which is a dedicated iDRAC. I'm hoping to use one nic (eno8303) for a dedicated IP to reach ProxMox, and bond the other 4 (eno12399, eno12409, eno12419, and eno12429) to a single connection to pass to all the VMs (leaving one nic unused for the time). I can reach ProxMox just fine on our network, and the box can reach the internet for updates and such, but the VMs can't seem to get internet. I feel like it's something related to the gateway, but I'm not certain. I tried creating a new bridge and attaching the bond0 as the slave, but I can't input another gateway on vmbr1 so I feel like the second bridge can't reach the outside. When I set the VM to use vmbr0, it can reach internet, when it is using vmbr1, it cannot. The switch has been configured to accept aggregate links on these ports already, so that should not be the issue. Any ideas?

Thanks!

Config below:

auto lo
iface lo inet loopback

iface eno8303 inet manual

iface eno 8403 inet manual

auto eno12399
iface eno12399 inet manual

auto eno12409
iface eno12409 inet manual

auto eno12419
iface eno12419 inet manual

auto eno12429
iface eno12429 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves eno12399 eno12409 eno12419 eno12429
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3
        
auto vmbr0
iface vmbr0 inet static
        address 10.10.239.238/26
        gateway 10.10.239.194
        bridge-ports eno8303
        bridge-stp off
        bridge-fd 0
        
iface idrac inet manual

auto vmbr1
iface vmbr1 inet static
        address 10.10.239.239/26
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0

 

[tscret]

Hey

I think you're missunderstood something. As you only want to use your Bond to connect your VMs to the Outside you don't have to assign an IP to that Bridge. But as I understood the VMs are in that 10.10.239.193/26 Subnet?

Is your Switch LAG realy in LACP Mode there are multiple Methods of Bonding.

For me it was helpfull to see vmbr as an Switch.

To Debug:
- Try the Connection from vmbr1 to outside without bond just with one single Member Port (imo => do not configure an IP for Proxmox on vmbr1)
- then try a bond... maybe you need to go with active/standby

=> What Switch are you using?

 

[jwooton]

Hey, thanks for the reply!

Yes, the VMs and the ProxMox install are on the same subnet.

I just deleted the bond and added eno12399 to vmbr1 as the only member port and also left the IP info blank and the VM still does not have internet. Switching the VM to use vmbr0 does allow access to the internet.

I worked with our network team to ensure the switch is configured properly, so I really don't think that's the issue here.

 

[tscret]

Hey, thanks for the reply!

Yes, the VMs and the ProxMox install are on the same subnet.

I just deleted the bond and added eno12399 to vmbr1 as the only member port and also left the IP info blank and the VM still does not have internet. Switching the VM to use vmbr0 does allow access to the internet.

I worked with our network team to ensure the switch is configured properly, so I really don't think that's the issue here.

Could you ping the Gateway IP from the VM?
Did you changed the Bond/LAG/Port Channel Config on the switch either?

- when you add an IP to the vmbr1 can you ping that IP from your VM on the same bridge?

 

[tscret]

Could you post the output of

Ip a s | grep bond0

 

[jwooton]

Output from ip a s | grep bond0:



Also, as far as the switch is concerned:

 

[jwooton]

Could you ping the Gateway IP from the VM?
Did you changed the Bond/LAG/Port Channel Config on the switch either?

- when you add an IP to the vmbr1 can you ping that IP from your VM on the same bridge?

I am unable to ping the gateway from within the VM. Tried a few things but it always simply says "ping: connect: Network is unreachable". When I change to vmbr0 on the VM, it is able to ping the gateway.

 

[tscret]

I am unable to ping the gateway from within the VM. Tried a few things but it always simply says "ping: connect: Network is unreachable". When I change to vmbr0 on the VM, it is able to ping the gateway.

Bond seems Up and configured right with LACP

So there could be a few things:
- Mess in Networkingservice => systemctl restart networking
- If VLAN are setup on switch is the LAG configured with the right VLAN Untagged?
- Mess with ARP-Table (Switch or Proxmox) or PVSTP => simply wait after the vmbr change about 10 Minutes


Some other Tests: set again an IP on vmbr1 to proxmox
- Are you able to ping the Proxmox vmbr1 IP from outside the PVE. Yes: BOND is okay | No: probaply an Fault ond the bond (try another single switchport and only add these to vmbr1 as member to further test
- Is the vmbr1 IP reachable from a VM on it? Yes: Probably Linuxbridge is working correctly | No: Problem on that bridge... try to resolve it by recreating, check if vmbr1 is right specified on VM Config (NIC).

I've setups with Linuxbridge and OVS with and without Bond... sometimes after bigger changes there were stranfe behavior on Networking. Restarting Networking helped... sometimes only a reboot of pve did help.

Hope you could determine the rootcause

 

[jwooton]

Solved: not an issue with ProxMox.

The issue turned out to be that the management interface was set to VLAN60 on the switch, while the aggregate interface was set to VLAN30, and we did not have traffic permitted between them. For ease, I just move the ae5 pictured above to VLAN60 and it is working fine now.

Thanks for the help @tscret

 

[tscret]

Solved: not an issue with ProxMox.

The issue turned out to be that the management interface was set to VLAN60 on the switch, while the aggregate interface was set to VLAN30, and we did not have traffic permitted between them. For ease, I just move the ae5 pictured above to VLAN60 and it is working fine now.

Thanks for the help @tscret

Good News and you're welcome - please mark the thread as resolved