Mutiple NICS/Bridges to separate Routers/Internet

[KirstyA]

I have 3 NICS connecting to separate Routers and out to separate Internet connections.
I have set up the additional bridges vmbr1 and vmbr2, given them the required subnets and linked them to the physical NICS, see screenshot.
vmbr0 has the 'Default Gateway' on it, and I know I cannot put gateways on the other 2, but need to add static routes instead.
My second and third routers are at 192.168.44.15 and 192.168.22.15 respectively.



Do I now just need to put these two lines in my /etc/network/interfaces file to finish the job so that vm's on the second and third subnets can reach the routers?

post-up ip route add 192.168.44.0/24 via 192.168.44.15 dev enp4s0

post-up ip route add 192.168.22.0/24 via 192.168.22.15 dev enp3s0

Here is my current /etc/network/interfaces file with the additional bridges in.
The 2 lines I think I need to add, for the static routes, are highlighted in bold italics

Have I done this correctly?
Anything else I need to do??

network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp0s31f6 inet manual

iface enp3s0 inet manual

iface enp4s0 inet manual

auto vmbr0
iface vmbr0 inet static
address 192.168.4.221
netmask 255.255.255.0
gateway 192.168.4.15
bridge-ports enp0s31f6
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet static
address 192.168.44.221
netmask 255.255.255.0
bridge-ports enp4s0
bridge-stp off
bridge-fd 0
post-up ip route add 192.168.44.0/24 via 192.168.44.15 dev enp4s0

auto vmbr2
iface vmbr2 inet static
address 192.168.22.221
netmask 255.255.255.0
bridge-ports enp3s0
bridge-stp off
bridge-fd 0
post-up ip route add 192.168.22.0/24 via 192.168.22.15 dev enp3s0

 

[spirit]

No, this is wrong.

" ip route add 192.168.44.0/24 via 192.168.44.15 dev enp4s0".

basicly, it said: I want to reach network 192.168.44.0/24 behind router with ip 192.168.44.15.

what you want is, I want to reach internet (0.0.0.0/0) behind router with ip 192.168.44.15.


But you want use 2 default gw for 1 server. (or maybe with some tricks with different routing tables, and source ip routing, but it's complex)



What do you want to do with your 3 routers links ? do you want to do some kind of loadbalancing ?

or do you want to have some vms on vmbr0 using route1, some vms on vmbr1 using router2, some vms on vms on vmbr3 using router3 ?

 

[KirstyA]

What do you want to do with your 3 routers links ? do you want to do some kind of loadbalancing ?

No, no load balancing. I require complete isolation between the vm's on each bridge/router.

or do you want to have some vms on vmbr0 using route1, some vms on vmbr1 using router2, some vms on vms on vmbr3 using router3 ?

Yes I want the vm's on each bridge/route to go through the respective router only, with no chance of going anywhere else. I have services on each vm/bridge that require the same incoming ports open.

what you want is, I want to reach internet (0.0.0.0/0) behind router with ip 192.168.44.15

So you are saying if I put these 2 revised lines in, it will do the job?

post-up ip route add 0.0.0.0/0 via 192.168.44.15 dev enp4s0
post-up ip route add 0.0.0.0/0 via 192.168.22.15 dev enp3s0

Effectively then, all traffic from vm's on vmbr1 will be routed out of 192.168.44.15 regardless,
And all traffic from vm's on vmbr2 will be routed out of 192.168.22.15 regardless.
Correct?

With Gateways of 192.168.44.15 and 192.168.22.15 respectively, inside the vm's themselves, traffic will come in from the routers back to the respective vm's
Also Correct?

Or will I need to put something in the routers as well?

 

[Arvyr]

Why are you putting IPs onthese vmbrs anyway? That's the wrong spot!
What you wanna do is adding those vmbrs to your VMs and CTs and add the according IPs in there.

 

[KirstyA]

Why are you putting IPs onthese vmbrs anyway? That's the wrong spot!
What you wanna do is adding those vmbrs to your VMs and CTs and add the according IPs in there.

Then how will those vmbr's know how to reach the Internet without a route and no gateway?

 

[spirit]

Then how will those vmbr's know how to reach the Internet without a route and no gateway?

Until you use your proxmox as router/gateway for your vm, you don't need ip address on the bridges.
(Think like a you a big cable/layer2 between your router and your vm)

Simply set the gateway ip in your vms.

 

[KirstyA]

Until you use your proxmox as router/gateway for your vm, you don't need ip address on the bridges.
(Think like a you a big cable/layer2 between your router and your vm)

Simply set the gateway ip in your vms.

So you are saying just set the gateway of 192.168.44.15 inside my vm and it will reach the router?
Obviously the vm will have DNS also set to 192.168.44.15

what you want is, I want to reach internet (0.0.0.0/0) behind router with ip 192.168.44.15

So I don't need to do this now?

And also remove all IP addressing from vmbr1 and vmbr 2?

 

[KirstyA]

Until you use your proxmox as router/gateway for your vm, you don't need ip address on the bridges.
(Think like a you a big cable/layer2 between your router and your vm)

So thinking further the vmbr's should just be seen as unmanaged switches then.
All the routing/DNS is being done between the vm and the actual physical router.

Hypothetically then you could do the same for vmbr0 and the vm's would still reach the Internet, just the host Proxmox would not be able to.
Have I understood this correctly?

 

[spirit]

So you are saying just set the gateway of 192.168.44.15 inside my vm and it will reach the router?

yes

So I don't need to do this now?

yes

And also remove all IP addressing from vmbr1 and vmbr 2?

and yes

So thinking further the vmbr's should just be seen as unmanaged switches then.
All the routing/DNS is being done between the vm and the actual physical router.

yes, exactly

Hypothetically then you could do the same for vmbr0 and the vm's would still reach the Internet, just the host Proxmox would not be able to.
Have I understood this correctly?

yes, the ip address on vmbr0 is only for proxmox management.
When an interface (eth) is in a bridge, you need to put ip on the bridge.

(you can remove enp0s31f6 from vmbr0, and set ip address on enp0s31f6 directly for example)

 

[KirstyA]

yes, the ip address on vmbr0 is only for proxmox management.
When an interface (eth) is in a bridge, you need to put ip on the bridge.

Yes I understand that now. It's quite simple when you understand it properly.

(you can remove enp0s31f6 from vmbr0, and set ip address on enp0s31f6 directly for example)

In this mode if you added the network card to your vm it would be like 'passing through' the network card directly, so to speak.
But only that device would have access to it, hence the creation of Bridges in the first place.

This all makes sense now.
I'll make the changes and try it out.

I read all the forum posts about this and see so many people struggle with this, for use with things like pfSense firewall etc.
They assume additional complexity required by not understanding this key line from you which made it all 'click' for me.

'Think like a you a big cable/layer2 between your router and your vm' - spirit

And not understanding, 'the ip address on vmbr0 is only for proxmox management' - spirit

Thanks for your help spirit and also thank you to Arvyr as well.
When I have tried it out I'll come back and mark this thread as solved