Strange Small issue need help with.

C

Calum.mck

New Member
Mar 2, 2025
7
0
1
Today I was having some issue with getting pveam update to work. I was getting errors:
download failed: 500 Can't connect to download.proxmox.com:80 (Temporary failure in name resolution)
2025-03-01 05:48:03 update failed - no signature file '/var/lib/pve-manager/apl-info/pveam-download.proxmox.com.tmp.107664.asc'

It was pretty clear the issue was some sort of connectivity issue.

Luckily, after messing around with some pings and such in the proxmox shell and an internal VM's shell. I was able to see that Proxmox was using an interface called "vmbr2." This was not the interface I had intended to use, "vmbr0". By changing some network configs and moving a couple wires around, I was able to update and move forward, technically.

However, the interface vmbr2 is currently stuck as the '"PROXMOX UPDATE'S INTERFACE'" for lack of better words. It is intended to be a separate bridge for my CISCO lab VMs. I have another "vmbr1" on the same card dedicated to a Truenas VM it's solid. I can in the short term, live with remembering to move things around and re-configure when I need to update proxmox, download container lists, or generally access the internet via proxmox's shell but that is a pain.

I have 3 interfaces configured on this rig, vmbr0, vmbr1, and vmbr2. Vmbr2 seems to be the one proxmox has chosen for it's self. Is there some way to select a different interface to do this over?
 
May be it is a gateway issue? The system will use the interface with default route on it configured. I assume, vmbr2 has a gateway configured. What about the other two vmbr*?
Would you like to post the content of /etc/network/interfaces file of the proxmox node?
 
The following is the interface file config. I have not changed vmbr2 back from the other day, I just haven't needed to as of yet just been on packet tracer lately. When I do physical labs, the NIC is normally configured to a different network and tends to change with labs, but the gateway(s) all stay the same—at least they have for a long time since I've been using this current router. I have labeled them the way I intend for them to work. The Truenas interface does work as intended.

auto lo
iface lo inet loopback

iface enp15s0 inet manual

iface enp5s0 inet manual

iface enp6s0 inet manual

iface enp16s0 inet manual

auto vmbr0
iface vmbr0 inet static
address 10.0.0.10/24
gateway 10.0.0.2
bridge-ports enp15s0
bridge-stp off
bridge-fd 0
#Onboard NIC for main interface

auto vmbr1
iface vmbr1 inet static
address 10.0.0.111/24
gateway 10.0.0.2
bridge-ports enp6s0
bridge-stp off
bridge-fd 0
#2.5Gbe Virtual Interface 1 for Freenas 'n such

auto vmbr2
iface vmbr2 inet manual
address 10.0.0.100/24
gateway 10.0.0.2
bridge-ports enp5s0
bridge-stp off
bridge-fd 0
#2.5Gbe Virtual Interface for Cisco Lab VMs

source /etc/network/interfaces.d/*



Please let me know if there is anything else that may help you help me here, it's just the damndest thing.
 
Based upon the logic here, it may be possible that changing "static" to "manual" may be the fix. However, I am a bit new to what some of the items inside these config files mean. It could just as easily break things as it could fix things, for all I know. If it would be possible to get a little bit of clarity before I made a commit with nano on this, that would be fantastic!
 
Only for a single interface the gateway must be defined. As this entry defines the default route for all traffic and is bound to a single network interface.
Removing the gateway entry for interfaces vmbr2 and vmbr1 should resolve the issue you have experienced.
If you have specific hosts or networks you want to reach from vmbr1 or vmbr2 only than you have to take a look at policy based routing.
 
Last edited:
  • Like
Reactions: news
I appreciate your response, but I am a little unsure of what you mean by this. The three separate interfaces have different jobs on the server and point to the same router gateway (10.0.0.2) otherwise, they would not have internet access. For example, if I pull the gateway from vmbr1 it would be the virtual equivalent of pulling an ethernet cable out of my true-nas/plex vm that would not help me; it would just create a new problem. I have moved it from different address spaces I know what falling off the network looks like. I plan to diversify this in the future, but for right now, this just works.

The intention is to have more high-speed connections for things I may want to prioritize and have more standard speeds for simple/smaller things. Proxmox has evidently decided to pick vmbr2 as its "interface of choice". I don't have better words; it's primary interface, and I would not like that to be the case vmbr0 the onboard NIC should be handling this. Besides this and the other interfaces have been working as they should. I don't know if there is a piece of minutia I am missing. Is there a mechanism by which Proxmox picks interfaces in situations like this where there are multiple?

Before I got the update fixed, prior to reconfiguring the gateway on vmbr2 it was set to 10.0.0.1. It didn't like that, and refused to make a connection and did not care that the other 2 interfaces (vmbr1 vmbr0) were online, it just needed to be vmbr2 I noticed this while pings out to google.com were failing. It kept using the same IP even when I pulled cables.
 
Calum.mck said:
The three separate interfaces have different jobs on the server and point to the same router gateway (10.0.0.2)
Click to expand...
This is a valid approach to get unstable connections. As a bonus trouble shooting is not really straight forward - sometimes "things" work, sometime they don't..., so just do not do that.

If you want to use multiple NICs either bond them (if your switch supports this) or use different (IP-) networks on them.

Example: if you go and just ping the gateway (ping 10.0.0.2) which NIC shall be used? This is not defined and will vary from time to time...

Just my 2€¢...
 
  • Like
Reactions: news
As UdoB pointed out, this will not work as intended and also this issue is not related to Proxmox but to how routing works.
Routing works by looking up a path to the peer. The lookup is done in a table. If multiple entries in this table lead to same peer, the more specific entry is chosen.
In a table like that
Code: 
default via 10.0.0.1 dev eth0
192.168.0.0/24 via 10.2.0.1 dev eth1
192.168.0.128/25 via 10.3.0.1 dev eth2
a lookup for 192.168.0.50 will lead to usage of eth1 as sending device
a lookup for 192.168.0.150 will lead to usage of eth2 as sending device
a lookup for 192.168.1.1 will lead to usage of eth0 as sending device

Your routing table looks roughly like this
Code: 
default via 10.0.0.2 dev vmbr0
default via 10.0.0.2 dev vmbr1
default via 10.0.0.2 dev vmbr2

Which one should the kernel choose to be the right one as they are all equal? Maybe it selected vmbr2, because it is the entry which was added last. This is therefore not the way to go.
If you need the other interfaces just for (very) simple failover, you can provide a separate metric to each entry. If one device is down, the next routing entry should be selected automatically.
If you need the interfaces in parallel for different purposes, than either provide some more specific routing entries to the selected targets, telling the system for a specific peer network to use vmbr1 only. Or you start using policy based routing based on the sending ip address.
 
Last edited:
  • Like
Reactions: UdoB
Hi I appreciate your responses and how fast replies have come in. I appreciate where you're coming from, and if possible, could you elaborate a little on the policy-based routing? Heh, not what it is, is it something that is done in proxmox, or is is done in my switch/router setup for this?

Would it maybe be a little easier to just slim down to a single 2.5gbe NIC I was having some stagger on a single 1gbe, which is why I'm currently split on 1x1gbe and 2x2.5gbe. I am studying for my CCNA currently, and until I'm a little more on the other side of things in the next couple of weeks, I don't think I should make any significant changes right now. But what a relevant project.

I'll appreciate any advice I can get to polish this up.
 
Thanks again!

Upon re-reading when UDoB said "bond," I did not understand they referred to a "linux bond" in the main node > System > Network, where I have configured "linux bridges". I'll have to look into my switch's specs as per UdoB's suggestion, I am using a Ubiquiti mini flex if that helps.

Flex switch goes to an Ubutuiti AP, which does most of the apartment. The flex I have feeding out to a stack out old cisco appliances I probably dont really need due to packet tracer but I'm a sucker for hands on I have a few old laptops and I have some Vms on Proxmox, making up a Win AD envinornment so it's worth it for me.1

I have to admit I did not see mention of mode-balance-alb; maybe it was implied. Would this be something configured in proxmox or at the switch level?

I apoloze for needing to have this spelled out for me.
 
Thank you again I for your help and patenence, it's funny in the fist article it's right there in the fist line that there can be only one gateway, and I don't know how you knew it or if it was a lucky guess but vmbr2 was the last interface that was configured so, that answers that question to a degree. I will spend some time on this when I can and get the config set up properly. It seems straightforward enough.
 
You must log in or register to reply here.
Top Bottom