2 nodes with 2 LAN interconnection

[mictian]

Good day everybody!

I've been using Proxmox for some time and configured 2 PVEs, join them into one cluster. Each one has a LAN for reaching VMs and containers from outside via the most common suggested way: network bridge+masquarade /etc/network/interfaces+iptables nat instructions. One network is 192.168.1.0/24, the other is 192.168.2.0/24

Despite the fact that both nodes "see" each other( I can migrate VMs and all that good stuff) the bridge network hosts can't (192.168.1.10 can't ping 192.168.2.10 for example). At this point, I'm asking for help so someone could suggest to me a decent way to connect them (Maybe I should just use nat on both nodes, maybe create a router, maybe vlan or corosync shenanigans).

Thanks in advance!

 

[Belegnor]

Hi mictian,

I don't know, how your network setup looks like, so I will try to give some generic advice ...

First, following the "most common suggested way" (network bridge+masquarade /etc/network/interfaces+iptables nat instructions) makes sense if you want to use the Proxmox node as a firewall. As you have at least two nodes, I wouldn't recommend that. Instead of this, I would recommend to use a dedicated firewall (e. g. PFSense VM or HW firewall). But this is just my opinion.

Second, you want to connect two different networks (192.168.1.0/24 and 192.168.2.0/24). Since you likely have VMs on both networks and you want to be able to mitgrate them from one node to the other, then both networks should be available on both nodes. This can be achieved setting the bridge to "VLAN aware". Alternatively, you can create subinterfaces (VLAN) and bridges on them, but the "VLAN aware" approach is easier to implement.

In order to use VLANs in your network, VLAN supporting Switches are required too. Otherwise, VMs in the same network won't be able to communicate with each other when they are runningon different nodes.

Then, if you want to connect two (or more) different networks to each other, a routing device is needed (dedicated firewall or router).

A basic network setup using Proxmox could look like this ...


This is just an example. You'll need to adapt it to your needs, in order to implement a similar setup. I hope. this helps ...

Best regards,
Belegnor

 

[mictian]

Hi mictian,

I don't know, how your network setup looks like, so I will try to give some generic advice ...

First, following the "most common suggested way" (network bridge+masquarade /etc/network/interfaces+iptables nat instructions) makes sense if you want to use the Proxmox node as a firewall. As you have at least two nodes, I wouldn't recommend that. Instead of this, I would recommend to use a dedicated firewall (e. g. PFSense VM or HW firewall). But this is just my opinion.

Second, you want to connect two different networks (192.168.1.0/24 and 192.168.2.0/24). Since you likely have VMs on both networks and you want to be able to mitgrate them from one node to the other, then both networks should be available on both nodes. This can be achieved setting the bridge to "VLAN aware". Alternatively, you can create subinterfaces (VLAN) and bridges on them, but the "VLAN aware" approach is easier to implement.

In order to use VLANs in your network, VLAN supporting Switches are required too. Otherwise, VMs in the same network won't be able to communicate with each other when they are runningon different nodes.

Then, if you want to connect two (or more) different networks to each other, a routing device is needed (dedicated firewall or router).

A basic network setup using Proxmox could look like this ...

View attachment 28800
This is just an example. You'll need to adapt it to your needs, in order to implement a similar setup. I hope. this helps ...

Best regards,
Belegnor

Thank you for the reply!


Well, that looks like something pretty hard for me to execute, so I need to make sure that it will be the right thing.

Both physical servers are located and maintained by a hosting provider. We started with one and we used the simplest solution for exposing local network. When we realized we need one more ASAP, I configured it the same way.

Since both nodes have been joined in a cluster - VM migration works fine. I was hoping for a common solution for making one network for both nodes since it seems like something obvious to do since your nodes are already in a cluster. While one firewall is a good suggestion, firstly I'd like to implement a joint network between the nodes. Or at least make them see each other. For example, I've been thinking about just a routing table entry for both nodes for each subnet. However, maybe such a direct approach is not ideal or even plain wrong in my case.

 

[Belegnor]

Both physical servers are located and maintained by a hosting provider.

... Oops ... ... I think I missed that point ...

Now is clear why you configured NAT and firewall on Proxmox ...

Well, the setup that I suggested is easy to implement for "on-premise" scenarios. I have no experience with hosting providers, so I cannot say, how easy or difficult is to implement it by a hoster. Perhaps you should ask your hoster about it.

There are a few routing related pain points I see in your case, but perhaps I've missed something else.

You told that every node has its own network (node A = 192.168.1.0/24 and node B = 192.168.2.0/24). In order to connect them, you'll need to add routes on every node pointing to the network on the other node (e. g ip route add 192.168.1.0 via <node A IP> ... ) or a routing device between both nodes.

An other issue that may occur is that VMs may get unreachable if they keep their IP when being migrated. And if the IP changes (DHCP configuration), then NAT rules may not work properly as they point to the wrong IP.

I don't know if it is possible to resolve this problem without VLANs. As said before, perhaps you should ask your hoster about this.

I'm sorry that I can't help in a better way. Perhaps someone else has a better idea

 

[mictian]

... Oops ... ... I think I missed that point ...

Now is clear why you configured NAT and firewall on Proxmox ...

Well, the setup that I suggested is easy to implement for "on-premise" scenarios. I have no experience with hosting providers, so I cannot say, how easy or difficult is to implement it by a hoster. Perhaps you should ask your hoster about it.

There are a few routing related pain points I see in your case, but perhaps I've missed something else.

You told that every node has its own network (node A = 192.168.1.0/24 and node B = 192.168.2.0/24). In order to connect them, you'll need to add routes on every node pointing to the network on the other node (e. g ip route add 192.168.1.0 via <node A IP> ... ) or a routing device between both nodes.

An other issue that may occur is that VMs may get unreachable if they keep their IP when being migrated. And if the IP changes (DHCP configuration), then NAT rules may not work properly as they point to the wrong IP.

I don't know if it is possible to resolve this problem without VLANs. As said before, perhaps you should ask your hoster about this.

I'm sorry that I can't help in a better way. Perhaps someone else has a better idea

Well, those are two dedicated servers that have been given, so since its our problem how we want to configure them. Both nodes are connected to the provider's routers, having different subnets to their external IP addresses though. However, when I tracerouted from one to another, it's clear that the packets don't go to the internet - they just go through a number of providers routers. Therefore nodes are part of one big network.

Anyway. Im not afraid that VMs will get unreachable after migration. When I was migrating a couple of these, I was well aware that I have to change the network config on them according to the new network I've migrated them into. I just need an optimal solution. I doubt Im the only person running proxmox not on-premise.

 

[Belegnor]

Well, those are two dedicated servers that have been given, so since its our problem how we want to configure them. Both nodes are connected to the provider's routers, having different subnets to their external IP addresses though. However, when I tracerouted from one to another, it's clear that the packets don't go to the internet - they just go through a number of providers routers. Therefore nodes are part of one big network.

Anyway. Im not afraid that VMs will get unreachable after migration. When I was migrating a couple of these, I was well aware that I have to change the network config on them according to the new network I've migrated them into. I just need an optimal solution. I doubt Im the only person running proxmox not on-premise.

Hi mictian

Were you able to resolve your connection problem? If not, then you could try to use the SDN features of Proxmox in order to connect the LANs on the different nodes with each other. For more information, please take a look at this:

https://pve.proxmox.com/pve-docs/chapter-pvesdn.html

I hope that this helps …

Best regards

 

[mictian]

Hi mictian

Were you able to resolve your connection problem? If not, then you could try to use the SDN features of Proxmox in order to connect the LANs on the different nodes with each other. For more information, please take a look at this:

https://pve.proxmox.com/pve-docs/chapter-pvesdn.html

I hope that this helps …

Best regards

No, I've been waiting for a clean solution. I'll check the link, thank you very much.

However, it confuses me a little, that both servers have no problem communicating through the corosync technology since Ive made a cluster out of them. I wonder why I can't somehow take advantage of that and just attach a LAN somehow.

 

[Belegnor]

No, I've been waiting for a clean solution. I'll check the link, thank you very much.

However, it confuses me a little, that both servers have no problem communicating through the corosync technology since Ive made a cluster out of them. I wonder why I can't somehow take advantage of that and just attach a LAN somehow.

Well, the corosnyc link is usually a dedicated connection for configuring HA or cluster setups and syncronizing config along all cluster members, so it only should be used for that. But even if you could use the corosync link, the problem is still the same.

As you need to connect the VMs on one node with the VMs on the other node, a clean solution would be in this case to configure VLANs on the interfaces of the nodes (e. g. VLAN aware bridge) and on the switches, so the same VLANs (and networks) are available and reachable on both nodes. The problem is, that the nodes seem to be attached to different networks, so "stretching" VLANs from one node to the other could be quite difficult, if changes cannot be done on the switching infrastructure too, so we could say your nodes are in "different locations".

I don't have any experience using SDN, as I never had to, but as far as I understand "the theory", you should be able "stretch" all the needed VLANs over an IP network. In some way, you could implement a setup like the one I suggested in my first response, but using the IP network as "transport medium" for your VLANs instead of the switch infrastructure itself ...