Proxmox VE multiple Gateways

[dqq]

Hi,

I want to utilize IPs pointing to my server.
Some of them are failover IPs, that are set to my server and can be redirected to other machines in case of dedicated machine failure.


What I want to do is setting up IP address and gateway to my second Network Device, but I get default gateway error (as attached)

What I do have:
- Linux bridge vmbr0 for my main ip with gateway

What I want to add:
- Linux bridge vmbrX for my failover IP with different gateway


Thanks for help

 

[aaron]

There can be only one default gateway.

In most setups a failover is done on the router that is placed in front of Proxmox VE. I don't know a way of how to achieve failover in that way set up directly on Proxmox VE.

Maybe someone else has an idea or an actual working setup.

 

[showiproute]

Hi there,

I found this link: https://www.thomas-krenn.com/en/wiki/Two_Default_Gateways_on_One_System

Maybe it helps you?

Would be interessting to post any feedback from your side.

 

[Bent]

The advantages of a dual gateway setup on the OS level (Proxmox) for 'redundancy' is at best questionable. As @aaron said, usually you achieve failover / redundancy on the router level, which is way more suitable.

@dqq please provide more details about your setup here:

- Provider (//hosting company) (!)
- Do you want to use these additional IPs for a router / VM on Proxmox or Proxmox itself
- are all IPs GWs in the same subnet

 

[Proximate]

Another advantage might be security. You could have management on one network and vms/resources on others.

 

[rohitp]

I would like to create 2 separate networks as well, one for management (VM mgmt interface + clustering of proxmox servers) and other network for the VM's datapath.
Since I would be running heavy workloads on VM's I would like to isolate that network so that I do not starve out the management links (or lose access to proxmox GUI or RDP/SSH to VM's)
I am able to do this in vsphere with each VM's having 2 NIC's and placed in different port groups

My server has 2x1G and 2x100G interfaces and I would like to create a bond of 2x100G interface and point it to gateway 1 (for VM datapath) and bond of 2x1G interfaces and point it to gateway 2 (for VM management+accessing proxmox server)

Please let me know if this is possible with the current PVE 7.3

 

[fabian]

you can have different bridges, but there can only ever be one gateway on any single linux host (although you can have arbitrary routes for various parts of your network).

 

[lxiosjao]

Hi,

I have 2 network device + 1 Linux bridge :

Name	Type	Ports/slaves	CIDR	Gateway	Comments
eno1	Network Device		172.20.30.250/24		Management (1Gb interface)
eno2	Network Device				10Gb interface
vmbr0	Linux Bridge	eno2	10.0.0.250/24	10.0.0.254

2 networks :
172.20.30.0/24 : management network
10.0.0.0/24 : fast network for VM

From the management network I target eno1 to connect to the GUI for the maintenance.
But when I reach Internet (upgrade packages for example) without gw on eno1, PVE will use eno2 to reach Internet (there is the gw on eno2), right ?

In the my external firewall, I only have to allow the IP of eno2 (not eno1) if I need internet for the maintenance, is it correct ?

 

[fabian]

with the config above traffic to 172.20.30.250/24 should go over eno1, and everything else including the default route/uplink via vmbr0->eno2->10.0.0.254->... , except for guests attached to vmbr0 of course, which are directly reachable over vmbr0 on the PVE host itself

 

[lxiosjao]

with the config above traffic to 172.20.30.250/24 should go over eno1, and everything else including the default route/uplink via vmbr0->eno2->10.0.0.254->... , except for guests attached to vmbr0 of course, which are directly reachable over vmbr0 on the PVE host itself

Thanks for your answer.
Something I don't understand : why eno2 is used when I use for exemple "apt upgrade" from the host pve ?
I would like to manage PVE with eno1, it means :
- reaching webUI of pve with eno1 (it's working)
- upgrade PVE with eno1 (not working) : reaching internet

And I would like to use only eno2 for guests attached to vmbr0.

Is it possible ?

 

[fabian]

Something I don't understand : why eno2 is used when I use for exemple "apt upgrade" from the host pve ?

because it is your default route to the internet..

if you have two uplinks and want to use one as default, and the other for specific sources or subnet-originating traffic, then you need to route accordingly, but that is a bit more advanced of a setup..

 

[oendaps]

you can have different bridges, but there can only ever be one gateway on any single linux host (although you can have arbitrary routes for various parts of your network).

Hi, is there no option to get more than 1 gateway? because my problem is when its compared between physical server, the connection isn't stable, example for PLC machine

 

[fabian]

by the very definition of what a gateway is, you can only have one. I am not sure what you are trying to achieve, but it is very likely not solved by having "two gateways". please describe your actual problem in detail, then we might arrive at a solution

 

[oendaps]

Hi Fabian, thank you for feedback,

My config is :
vmbr0 - linux bridge - eno1
vmbr0.88 - linux vlan - CIDR 192.168.88.16/24 - Gateway 192.168.88.1
vmbr0.1 - linux vlan - CIDR 192.168.40.2/24

VM100 - windows server - IP 192.168.40.3/24 - Gateway 192.168.40.1

on PLC production to monitoring oil meter is has many device with ip 192.168.40.10,192.168.40.11,192.168.40.12 and so on

The problem is the connection from VM100 to PLC is not stable,sometimes its RTO randomly and reply again
On the other side, i've compared to another old server with the 192.168.40.1 as default gateway and its stable

I assume that communication only stable when its on the same network

So, is it possible on the proxmox when setting maybe another port to set another gateway?

Apologize if my explanation confused you, hope you get the point

 

[shanreich]

Do you really need another gateway in this case? The subnet in VM 100 is 192.168.40.0/24 - so everything in that subnet (according to you 192.168.40.10, 192.168.40.11, 192.168.40.12) should already be reached without a gateway. A gateway is for when you need to reach Hosts in a different subnet. In VM 100, you can add two network interfaces - one on vmbr0.88 for internet connectivity via 192.168.88.1 and one with the IP 192.168.40.3/24 and NO gateway for reaching the devices in 192.168.40.0/24. (You can also just add one port on vmbr0 of course and then configure the VLANs inside the VM)

Or do you want to funnel traffic for 192.168.40.0/24 through 192.168.40.1 because it is e.g. a firewall?

Another thing: Are you using VLAN 1? You should not use that VLAN in general, since it often is used as the management network (vendor-dependent, ...) and is also the default PVID for vlan-aware linux bridges. I'd recommend switching the VLAN Tag, it prevents lots of possible pitfalls / landmines.

 

[oendaps]

Hi Stefan, thank you for reply,

Yes, i can reach 192.168.40.10, 192.168.40.11, 192.168.40.12, etc from VM100
And i already add another network interface with 192.168.88.1 on VM100 and the result is still the same, not stable

the point is what a best solution if i have a VM100 with network 192.168.40.3 can stable communicate with 192.168.40.10, 192.168.40.11, 192.168.40.12,etc which is the PVE has a gateway 192.168.88.1?

you say on another thing and i get it and should i change the 192.168.40.3 to VLAN40 maybe to get more stable connection with minimum intruptions?

 

[shanreich]

the point is what a best solution if i have a VM100 with network 192.168.40.3 can stable communicate with 192.168.40.10, 192.168.40.11, 192.168.40.12,etc which is the PVE has a gateway 192.168.88.1?

adding another gateway won't fix an unstable connection. maybe there is some issue with the underlying network? Have you checked all cables/hardware/NICs involved?

you say on another thing and i get it and should i change the 192.168.40.3 to VLAN40 maybe to get more stable connection with minimum intruptions?

It might be the issue, but i cannot tell. You should change it either way since it is not recommended to use VLAN 1.

 

[oendaps]

Hi Stefan as i mentioned before "i've compared to another old server with the 192.168.40.1 as default gateway and its stable"
so i think the problem is sot from the cables/h-w/NICS

 

[shanreich]

the point is what a best solution if i have a VM100 with network 192.168.40.3 can stable communicate with 192.168.40.10, 192.168.40.11, 192.168.40.12,etc which is the PVE has a gateway 192.168.88.1?

If you have an IP address 192.168.40.3/24 configured in the VM, then traffic to 192.168.40.0/24 will not go via the gateway (because you will have a static route for 192.168.40.0/24). So, changing the gateway does not affect anything.

Changing the gateway on the PVE host does not affect how traffic from your VM gets routed, because your VM is not using PVE as a gateway.

So it is likely you have some misconfiguration / hardware issue along the way.

E:
Can you post your network configuration?

ip a
ip r

Can you post the VM configuration?

qm config <vmid>

Can you post the network configuration of the VM?