Proxmox network not working after new hardware

[172pilot]

I've been searching, and cant find anything similar in any forum posts, so sorry if this is known... BUT..
I had a working cluster of 3 Intel NUC servers, each running ProxMox. I got a good deal on a newer NUC, so I bought it, and swapped the nvme and SATA drives from my slowest server into this new NUC, expecting it basically to just come back up and work as before.

The new NUC, as it turns out, has different NIC names, so networking didn't start up properly. I connected directly to the server, figured out what the NIC names were, and found some documentation online about editing the /etc/network/interfaces file, so I did. All I changed was the nic identifier (enp89s0 in this capture).

Networking now works, in so far as I can manage the server as a healthy member of the cluster, via it's IP of 192.168.1.253, and I can actually even migrate VMs to the server, but the VMs do not have access to the network, in either subnet.

What could cause networking to work from a host perspective, but not to work from the perspective of the VMs in this case? I have torn down and recreated the vmbr interfaces in the GUI, and I can't get it to do anything.

Just for a little more background, the reason I have this configured as a bond0 is because I used to have a USB-C NIC bonded with the built-in gig port, but I found this was more trouble than it was worth, but I didn't bother to remove all of the other bond related config. It is working on my other NUCs and was working on this one with the older hardware.

Any help is greatly appreciated! I've been pounding my head against the wall on this for a couple weeks on and off now, and my next attempt would just be a full reinstall, but I hate to do that and not learn something from my config error!!

auto lo
iface lo inet loopback

auto enp89s0
iface enp89s0 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves enp89s0
        bond-miimon 100
        bond-mode balance-rr

auto bond0.1
iface bond0.1 inet manual
#Workstations

auto bond0.10
iface bond0.10 inet manual

auto vmbr10
iface vmbr10 inet manual
        bridge-ports bond0.10
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 192.168.1.253/24
        gateway 192.168.1.1
        bridge-ports bond0.1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#Workstation vlan

 

[cazz]

Hi
Not sure if I can help but I do wonder a little about the code.
When I look at my proxmox I and I just use one ethernet port (eno2) I have bridge-ports that to vmbr0

I guess you use vmbr1 as bridge to your VM but vmbr1 go to bond0.1 and not enp89s0

Not sure if that is the right answer but it feel it does not connect right to your NIC?

My look like this

auto lo
iface lo inet loopback

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.2/24
        gateway 192.168.1.1
        bridge-ports eno2
        bridge-stp off
        bridge-fd 0

iface eno1 inet manual

iface eno3 inet manual

iface eno4 inet manual

I have four ethernet on this server but only use one (eno2)

 

[172pilot]

Thanks for the reply... Yeah - I dont think I was really clear in my original post, but I'm doing this because I have multiple VLANs setup over the one connection.. I have configured a VLAN 1, which is my default "normal" vlan, and a VLAN 10, which is what the .1 and .10 on the bond.x is supposed to mean. I dont have a huge amount of experience in recent Linux networking so that's where I'm falling short I think, but the strangest thing is that it seems to work just fine on my other 2 NUCs, and used to on this one as well, even on the same ethernet cable to the same port, which hasn't changed configs..

 

[cazz]

ok
Maybe is the settings in the VM host that need to change too?
Not sure what OS you are running in VM but maybe they too have another name then the original?

 

[172pilot]

I've spent some time this weekend trying to troubleshoot it and I found another clue. On my server that is working (proxmox2 in this picture) I have "fwbr*" and "fwln*" bridges setup, but in my non-working machine, I dont. When I google what those devices are, all I could find is that when you start a virtual, they're supposed to be created automatically for bridging purposes, and "dont worry about them".. But since they're missing, something isn't working. Another thing I discovered is that DHCP is actually assigning an appropriate IP to my VMs even though after getting the IP, they can't communicate at all, so I believe it to be something bridging or firewall related.

Also, looking at /sys/class/net, I see the interfaces in question are created on the working server, but not the non-working server. Since I didn't create these manually on either, I assume these are virtual interfaces that should be created at VM creation or startup time? I created a brand new VM directly on the broken server in question just to make sure it wasnt related to anything held from a previous virtual config, and had no more success..

I could reinstall from scratch, but I'd rather learn from this if possible, since it's just a lab environment..

Anyone know where/how these virtual interfaces get created, and how the bridge for the fwbr* device gets created?

 

[172pilot]

ok
Maybe is the settings in the VM host that need to change too?
Not sure what OS you are running in VM but maybe they too have another name then the original?

As far as the virtual NICs, they are still being virtualized - I have tried windows and linux hosts, and I can remove / add / reconfigure the virtual NICs as expected, and even change the type of the virtualized NIC and everything in the VM itself follows. I dont think it's a problem w/ the virtual talking to Proxmox, I think it's a problem with the bridging within proxmox.. (See other post w/ screenshots and results of more testing!!) Thanks for reading and helping!!

 

[cazz]

First I say this is more advanced for me but trying to help a little.
When I did google fwbr105i0 I did find some problem that did have problem with network and proxmox.
Some was hardware error in ethernet port but what I understand you can SSH your proxmox so that is not the problem.

I did find a replay that wolfgang have done i a post about networks devices
https://forum.proxmox.com/threads/unknown-devices-ifconfig.42407/#post-203988

fwbr<VMID>i<network interface X> is the firewall bridge where the filtering happens.
fwpr<VMID>i<network interface X> is the bridge in device.
fwln<VMID>i<network interface X> is the bridge out device.
tap<VMID>i<network interface X> is the connector port for the guest.

As you did find out the can be something about the firewall and fwpr* have to do something about the firewall
fwln is about bridge out the device so it does feel something is missing to make a correct bridge between host and VM.

 

[172pilot]

First I say this is more advanced for me but trying to help a little.
When I did google fwbr105i0 I did find some problem that did have problem with network and proxmox.
Some was hardware error in ethernet port but what I understand you can SSH your proxmox so that is not the problem.

I did find a replay that wolfgang have done i a post about networks devices
https://forum.proxmox.com/threads/unknown-devices-ifconfig.42407/#post-203988

fwbr<VMID>i<network interface X> is the firewall bridge where the filtering happens.
fwpr<VMID>i<network interface X> is the bridge in device.
fwln<VMID>i<network interface X> is the bridge out device.
tap<VMID>i<network interface X> is the connector port for the guest.

As you did find out the can be something about the firewall and fwpr* have to do something about the firewall
fwln is about bridge out the device so it does feel something is missing to make a correct bridge between host and VM.

Yeah - I think that's the same post I found.. It just says that the fw* interfaces are created automatically on startup, whereas on mine they are not, which I assume is related to the problem.. (Who knows if it's cause or symptom!!)

I am assuming something in the initial setup of proxmox, or maybe in the creation of the bond0 or vmbr interfaces detects the physical interface name and uses that name in a config to later create and destroy those bridges as the machines start and stop, but I'm not sure where that config may be stored.

I guess I could delete ALL of my network config and start again from scratch, but I'm not sure how to trigger the initial network configuration from the local console (I wont have web access at that point since my IP will be down 'till I can reconfigure it!)

If it's not a config file I can edit, then I guess I'm stuck reinstalling, or installing a separate NIC so I can maintain management while I destroy and rebuild the bond and vmbr interfaces. It wouldn't be a bad idea for me to have a dedicated management NIC in there anyway, but I had just cut down on wiring since I didn't seem to need it for performance.

 

[cazz]

What I understand it is same error when you create a new VM so I do wonder if that is any log to read when someone create a VM.
Maybe that say something what is missing.

It just feel that when you create a VM it try to make a bridge and that for the network but something misses but it continues to create a VM so maybe you can ready any log of that?

 

[172pilot]

What I understand it is same error when you create a new VM so I do wonder if that is any log to read when someone create a VM.
Maybe that say something what is missing.

It just feel that when you create a VM it try to make a bridge and that for the network but something misses but it continues to create a VM so maybe you can ready any log of that?

Yeah - I think that's a fair idea.. But I have no idea where proxmox may log that.. If nobody else chimes in I'll probably add a second NIC temporarily so that I can tear down the entire stack on this bond interface and start from scratch and see if it will start working. Next step would be a full reinstall I guess..

 

[Dunuin]

You could test the 4.15 kernel from the test repo in case your new hardware is to new or driver problems.

 

[spirit]

fwbr interfaces are created on vm startup, if you have the firewall checkbox enabled on the vm nic in the gui.

 

[172pilot]

fwbr interfaces are created on vm startup, if you have the firewall checkbox enabled on the vm nic in the gui.

Hmm.. so those are only created if I want to use the built-in firewall..?? I have been unchecking that to simplify my troubleshooting, so maybe that's why I'm not seeing that. I guess I'm going to resort to trying to rebuild networking from scratch, and if that doesn't work, a rebuild is in order..