Help with multiple bridges and VMs

[Tiscan]

Just came over from UnRaid and I am having issues with my bridges.

I have two VLANs off of my swtich. One is a LAN and one is a DMZ.

The LAN is connected to eno3 (bridge is vmbr0) and the DMZ (bridge is vmbr1) is connected to eno4. I want the Proxmox server to be on the LAN and it's using vmbr0, no problems.

However the VMs I am creating and assigning to vmbr1 have no connection. They are fine if I swap them to vmbr0.

What am I doing wrong?

Here is my /etc/network/interfaces

auto lo
iface lo inet loopback

iface eno3 inet manual
#10G LAN

iface idrac inet manual

iface eno1 inet manual

iface eno2 inet manual

iface eno4 inet manual
#10G DMZ

auto vmbr0
iface vmbr0 inet static
        address 10.51.1.10/24
        gateway 10.51.1.1
        bridge-ports eno3
        bridge-stp off
        bridge-fd 0
#LAN

auto vmbr1
iface vmbr1 inet manual
        bridge-ports eno4
        bridge-stp off
        bridge-fd 0
#DMZ

 

[Dunuin]

However the VMs I am creating and assigning to vmbr1 have no connection

No connection to what?

Config looks fine. How do you setup your guests? DMZ is a dedicated subnet and you got some router somewhere that is routing between internet, LAN and DMZ?

 

[Tiscan]

Hey,

The DMZ is indeed a dedicated subnet (10.51.2.1/24) and there is a router in place between internet, LAN and DMZ. The router itself is working fine, I can validate that other devices on the DMZ have no connectivity issues.

The VM has a static IP of 10.51.2.12 with 10.51.2.1 as the gateway. It's not even able to ping the gateway. If set to DHCP it can't get an address (there is a DHCP server on the DMZ as well). I don't think this has anything to do with the VM config itself as if I assign the VM to vmbr0 the network comes up via the LAN DHCP server immediately.

I feel like I am missing something painfully obvious...

 

[Tiscan]

Ok more digging.. it seems Proxmox won't enable eno4...

I can plug the same cable into another interface and it comes up. Link is UP but is listed as NO-CARRIER. Which makes little sense as this same card and interfaces ran fine for both Unraid and TrueNAS Scale.

NIC is a Dell 0CD2VM Intel X550-T2 Dual-Port 10GB + Dual-Port 1GB Base-T RJ-45 Network Daughter Card

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    altname enp1s0f0
3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    altname enp1s0f1
4: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    altname enp2s0f0
5: eno4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master vmbr1 state DOWN group default qlen 1000
    altname enp2s0f1
6: idrac: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
7: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 10.51.1.10/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::266e:96ff:febe:2b0/64 scope link
       valid_lft forever preferred_lft forever
8: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet6 fe80::266e:96ff:febe:2b1/64 scope link
       valid_lft forever preferred_lft forever
9: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether ba:c8:f5:96:23:08 brd ff:ff:ff:ff:ff:ff

 

[XGKCode]

Hey NetMagi,
Im a newbie but I had something like this happen to me before. Have you tried putting the interface to up?

Type into the cli: ip link set <device> up

Replace <device> with the interface eno4?.

Hope this helps you make it work!

 

[Tiscan]

Thanks for the advice.

Unfortunately the interface is already UP. The issue is that eno4, while being UP is showing as NO-CARRIER. Meaning that the network jack detects no signal on the line. It's not a cable issue or a port issue on the switch as I can simply move the cable to another NIC port and it works, thereby eliminating those as potential failure points.

This means one of two things:
1) Somehow between running TrueNAS Scale and Proxmox this one port on the NIC died (kinda unlikely)
2) Proxmox driver for this NIC has some issues
3) ???

 

[Tiscan]

Solved: On a whim I powered down the server completely, even removed the power cord to ensure that even the iDRAC was powered off. Waited 5 min and started back up.. whamo, everything works!

I had tried reboots with no luck. So for anyone who finds this in the future it's a cold boot from a non powered state that seemed to fix it. No idea why.