[SOLVED] Save some from frustration if you cant access GUI or PVE has no connection at all

ieronymous

Active Member
Apr 1, 2019
251
18
38
44
Hi

Happened me before a year, happened again yesterday after an update which was correlated to pve update. At least the first time it happened I had a pop up message if I would like to update or keep the /etc/network/interfaces file. Now without warning or anything updated, I reboot the server (production environment) and voila. No access to gui / ssh . Afterwards when I logged in from the server itself, I noticed that it had no internet connection even though all the configuration was there and correct.

So with
ip route
default via 192.168.77.1 dev vmbr0 onlink linkdown (hmmmm)
192.168.77.0/24 dev vmbr0 proto kernel scope link src 192.168.77.200 linkdown (hmmmm)

with
cat /etc/resolv.conf
search your_named_domain.com
nameserver 192.168.77.1 ok here it sees the correct DNS (router or whatever you configured it to look at)

with
ip a
noticed that vmbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000

This leaded me to check bond0 upon which vmbr0 is based and it was down as well
bond0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
....and the same goes for the two ports the bond0 consists of.
enp25s7: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
enp7s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000

That was when I recalled having lived this again in the past. My next stop was interfaces file. So with

nano /etc/network/interfaces
I was trying to check each line and see what changed. For one more time it was the same thing.
auto enp7s0
iface enp7s0 inet manual

auto enp0s25
iface enp0s25 inet manual
.....the 2 ports that form the bond0 and bond 0 forms the vmbr0.

It added or uncommented the # I had and couldn t work. After I commented again with # like
#auto enp7s0
iface enp7s0 inet manual

#auto enp0s25
iface enp0s25 inet manual
... saved the file and tried with ifup / ifdown to restart the ports enp7s0 and enp7s0 (interfaces whatever they call them) didn t work. So I tried restarting the bond0 and last the vmbr0 but nothing (with this way).

Anyway, a reboot did the trick of restarting the ports/bonds/vmbrs and voila had net access again and could log in remotely to proxmox and start VMs. There are plenty of reasons of that might happen (ip conflict, router down (oh you didn t check) DNS resolvent ...etc) but I am giving you one more step to check on the process of troubleshooting.

PS I don t know if for some of you works with auto line enabled but for me it doesn t. I guess auto means autostart and not autoconfiguration?
 
Last edited:
Yes, it's auto-up, the configuration is in the next line iface there it states manual instead of dhcp.
Nice for the clarification of the abbreviation. So you are saying that it should be playing with the line auto enp7s0 (without deleting it or put a has #)
Then do you have an explanation of why it doesn t?

PS Each bond i have based on 2 ports is configured as LACP (802.3ad). Of course the same is true from the switches side. If it weren t it wouldnt have played at first place (and it plays for 2 years except the times after an update mess up with that interface file)
 
Got a similar problem when upgrading from PVE6.4 ontop of Debian 10 to PVE7.1 ontop of Debian 11. The network config that worked before wasn't working anymore and I needed to remove the 'auto ...' lines for my NICs to make the networking working again. As far I understand you just need the 'auto ...' for your vmbr because when the vmbr autostarts it will also start the bond and the bond will start the NICs. I thought back then maybe something changed between Debian 10 and 11 how interfaces are started. That first the NICs are upped and maybe then the bond or vmbr can't start if the NICs are already running and in use.
 
Last edited:
That first the NICs are upped and maybe then the bond or vmbr can't start if the NICs are already running and in use.
That might seem more reasonable explanation. Because it makes sense first the hardware to be initialized and afterwards the software services like vmbr (if you can call it a service protocol or whatever that might be).

As far I understand you just need the 'auto ...' for your vmbr because when the vmbr autostarts it will also start the bond and the bond will start the NICs.
i haven t tried with auto, only on vmbrs . I still have that line at bonds as well. Maybe it works the same with or without the auto at bonds level.

By the way, a little out of scope, but since we are talking about bonds /ports-nics I am going to give it a try.

Does proxmox needs a vmbr only to work, meaning to configure ip address / gateway ...etc. if yes what is the purpose of having the same tab options at ports-nics and bonds level as well. My issue is I have a 10g port which if I configure with a static ip only (no gateway) in order to have access to an iscsi share from a truenas server it states the status of port under Active column as stopped, If I remove that ip address from that port and make it part of vmbr (reboot the server of course - I don t know why ifupdown2 isn t pre installed on proxmox even though I can install it I don t want to - something breaks and can t remember what that is) then the Active column states the port with Yes.

Last but not least at which level (on proxmox side) you change the MTU (for jumbo frames) to 9000? Port-nic-interface, bond or vmbr?
Because all have that options under advanced.

Thank you.
 
Last but not least at which level (on proxmox side) you change the MTU (for jumbo frames) to 9000? Port-nic-interface, bond or vmbr?
All NICs shoud have it set, everything should just use it.
Does proxmox needs a vmbr only to work, meaning to configure ip address / gateway ...etc. if yes what is the purpose of having the same tab options at ports-nics and bonds level as well.
You normally don't. It is sufficient to have the options on the bridge, which consists of at least one outgoing interface, which is the bond in which at least one nic is enslaved.
 
All NICs shoud have it set, everything should just use it.
By that you mean -> set it on nics omly and bonds or vmbrs will use that value or set it on all of them?

You normally don't. It is sufficient to have the options on the bridge, which consists of at least one outgoing interface, which is the bond in which at least one nic is enslaved.
Sorry but your answer was very phrasal and lost the meaning. Normally you don t means I could be able to use-configure just the nic and don t use at all. Afterwards go to cli ping nas server and have an answer which I dont.

On the other hand with << It is sufficient to have the options on the bridge, which consists of at least one outgoing interface, which is the bond in which at least one nic is enslaved>> it is like you are saying yes it is mandatory to use vmbr but it doesn tmatter where you configure the static settings, nic or vmbr it self.
 
By that you mean -> set it on nics omly and bonds or vmbrs will use that value or set it on all of them?
Bonds and bridges inherit the properties from their lower level devices (NIC on the bottom, then bonding and there the bridge).

Sorry but your answer was very phrasal and lost the meaning. Normally you don t means I could be able to use-configure just the nic and don t use at all. Afterwards go to cli ping nas server and have an answer which I dont.

On the other hand with << It is sufficient to have the options on the bridge, which consists of at least one outgoing interface, which is the bond in which at least one nic is enslaved>> it is like you are saying yes it is mandatory to use vmbr but it doesn tmatter where you configure the static settings, nic or vmbr it self.
Sorry. The network is constructed in layers. Depends on the use case, but on a single server with two nics, you would probable set it up like this:

The bridge (vmbr0) is the only interface that gets an ip address. This bridge is used as an endpoint for all VMs and containers. In this bridge, you add a bond device (for the outgoing traffic) and this bond device enslaves the two NICs. As the nic is - as the name suggests - network interface card - the interface to the "real" network, all "real" network parameters like frame size, speeds etc. have to be set there. Inside the bridge, you only have virtualized network, so packages are just copied from one VM to the other on the same host, so all those layer 1+2 stuff does not really matter. The same goes for the "network speed" in a bridge. It is not bound to any known NIC speeds, it just copies memory, which is much, much faster and does not stick to "real world" network speeds.

I hope this was more clear.
 
  • Like
Reactions: ieronymous
Thank you for the time and answers

Bonds and bridges inherit the properties from their lower level devices (NIC on the bottom, then bonding and there the bridge).
Perfectly understandable. Probably so many times in conversations someone could just pop up and say this instead of it is tricky. Try and see...etc

The bridge (vmbr0) is the only interface that gets an ip address.
Even though on my system I have a setup like you said, meaning x3 vmbrs (each one based one a bond which in series based on 2 different nic-ports) and only the vmbr0 gets static ip and gateway.
So this setup adds to my initial question. How am I going to setup a nic-port-interface enp7s0 for instance that is a 10g port, with a static ip (different network segment than the VMs are using) in order to communicate via this 10g port with the nas server? (they are connected directly via DAC cable)
 
Last edited:
Even though on my system I have a setup like you said, meaning x3 vmbrs (each one based one a bond which in series based on 2 different nic-ports) and only the vmbr0 gets static ip and gateway.
So this setup adds to my initial question. How am I going to setup a nic-port-interface enp7s0 for instance that is a 10g port, with a static ip (different network segment than the VMs are using) in order to communicate via this 10g port with the nas server? (they are connected directly via DAC cable)
If its just a single 10Gbit NIC and you don't want any VM/LXC to use that NIC too, you can just set a static IP (without a gateway if you just want to use it to connect to your NAS) to that NIC. If you got two 10Gbit NICs give that bond the IP. If you also want VMs/LXCs to use that 10Gbit NICs (for example so that they can access the NAS too) I would create a bridge and give that bridge the IP.
 
If its just a single 10Gbit NIC and you don't want any VM/LXC to use that NIC too, you can just set a static IP (without a gateway if you just want to use it to connect to your NAS) to that NIC. If you got two 10Gbit NICs give that bond the IP. If you also want VMs/LXCs to use that 10Gbit NICs (for example so that they can access the NAS too) I would create a bridge and give that bridge the IP.
...which is exactly as I set it up, meaning even though the real 10gbe card has 2 ports, i need only the one configures so I have setup a static (of course without gateway). By the way I ve set up MTU 9000 on the port. From the NAS side I ve configured a static to the corresponding 10gb port and they are connected together via DAC sfp+ cable. Leds are blinking between on both sides. If I ping the address of the nas from the proxmox it is unavailable. That is what I am trying to figure out.
A clue might be the fact the specific 10gbe port at least on proxmox's side isn t active. I have checked autostart.
 
Did you verify that your DAC isn't vendor-locked by your NAS or NIC?
hm.....nope. But first I need to check what happens if I manage to make the port active fist.

@Dunuin .... well I did make it Active and nothing happened but I found that the problem is probably the 10gb card itself even it lights up when you plug a cable in. So many months I assumed all the other reasons (drivers, configuration, cable) except the hardware fault due to the fact the leds were on. Anyway .. I borrowed a card, install it and just by configuring (like before) the interface with a static IP it pinged at last the nas server.

Thank you for your thoughts...somewhere between checking for cable compatibility and different setups found the issue. It was a mellanox 3 card
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!