Configuration changes in existing cluster

[Yuri_AR]

Hi! I have a 4 node proxmox cluster, configured as follows:

* All nodes are connected to a switch, in the 172.16.0.0/24 IP range with one of their eth interfaces.
* One of the nodes has an additional ethernet interface acting as a VLAN trunk. This node also contains a VM with PFsense, managing the traffic on a bunch of VLANs.
* proxmox API/interface can be reached from any vlan in the .0 IP address, or in their 172.16.0.X address.

This architecture evolved over time in a rather chaotic way, and I'm now looking to improve and stabilize it. However, most if not all of the changes I want to make are not trivial to do and/or can't be done with the web interface, and I'm looking for advice on what the best way to proceed would be. I'm also looking for a sanity check, just in case any of my ideas are unsound.

1) I would like to have the node communication traffic in their own internal, physically separate network. This is in preparation for a 10Gb network upgrade further down the road, both for cluster traffic and shared storage. I've found a bunch of old manual procedures on the internet, but they all seem a bit sketchy and I'm not sure they are up to date anymore. I've looked on the web GUI and there's no option for this there either. Deleting/wiping nodes, re-creating the cluster, or starting from scratch is not viable. Would a managed switch be beneficial here or would using any of the remaining ethernet interfaces with a normal switch be enough?

This future upgrade will also include a physical, actual netgate appliance. Until then, WAN traffic would have to come in through the node which is now managing the VLANs. I'm fully aware of how shitty this arrangement is.

2) I want all the nodes to have access to the same VLANs, both for ease of deployment of new containers in the right networks across nodes, but also to allow us to use HA capabilities more easily. Would you recommend to have a shared VLAN trunk in one of the interfaces on each node, or would it be better to use the SDN feature? The SDN feature looks both what I need and also complete overkill. I don't really need separate zones for the different nodes or any of the advanced features, but I need, say, a machine in VLAN 99 to have HA capabilities across nodes.

3) I want to limit access to the web interface and API to a particular VLAN. This is one of the reasons to have an internal network. However, I've seen that the .0 IP address in each VLAN does have a proxmox interface/API available. Is there any way to specify in proxmox where this is served without breaking cluster communications, rather than filtering it externally?

4) Any other suggestions, things I may be missing, or best practices I may not be following for lack of experience are more than welcome.

 

[j.theisen]

Hi @Yuri_AR

thanks for posting on the forum!

Before diving into your actual topic, i emphasize the importance of backups for this operation. Something might go wrong and i don't want you to lose data.

That being said:

looking to improve and stabilize it

In this case i would recommend using the SDN stack from here on, as this makes configuring VMs and adding or removing f.ex. VLANs much more transparent, imho.

For reference i link the Corosync requirements [1], as they are the foundation for using HA in this scenario.

1) I would like to have the node communication traffic in their own internal, physically separate network. This is in preparation for a 10Gb network upgrade further down the road, both for cluster traffic and shared storage.

Please keep in mind to separate Corosync traffic from any form of Storage or VM traffic and use these interfaces only as redundant links or you might run into problems especially with HA enabled.
You mention "shared storage". Are you using Ceph or some other form of shared storage?

Changing the IP address of a cluster node is not recommended but still possible.
This short instruction [2] should suffice.

Would a managed switch be beneficial here or would using any of the remaining ethernet interfaces with a normal switch be enough?

A managed switch would f.ex. allow you to separate the cluster traffic into a separate VLAN without having to configure anything on the hosts network interface which is best practice but not required.

This future upgrade will also include a physical, actual netgate appliance. Until then, WAN traffic would have to come in through the node which is now managing the VLANs. I'm fully aware of how shitty this arrangement is.

Having a virtualised firewall is not unusual and such not "shitty" per se.
I'm a little confused on your current network setup since you only mention one node having the VLANs configured. Is there a reason why this is? Are all the VLAN interfaces configured on the pfsense or Proxmox side?

2) I want all the nodes to have access to the same VLANs, both for ease of deployment of new containers in the right networks across nodes, but also to allow us to use HA capabilities more easily. Would you recommend to have a shared VLAN trunk in one of the interfaces on each node, or would it be better to use the SDN feature? The SDN feature looks both what I need and also complete overkill. I don't really need separate zones for the different nodes or any of the advanced features, but I need, say, a machine in VLAN 99 to have HA capabilities across nodes.

In this matter "access" might mean different things, but from the rest of the question i figure you mean being able to give Containers and VMs access to the VLAN and not being able to access the Web GUI etc. from there, correct?
In this case i still recommend looking into the SDN [3]. It might seem intimidating at first, but will make your life easier in the long run imho. For your deployment you mostly have to look at the VLAN zone [4] and can leave all from Fabrics to IPAM to the side at first.
This will allow you to define VLANs on the datacenter level and having them available at all nodes at once.

3) I want to limit access to the web interface and API to a particular VLAN. This is one of the reasons to have an internal network. However, I've seen that the .0 IP address in each VLAN does have a proxmox interface/API available. Is there any way to specify in proxmox where this is served without breaking cluster communications, rather than filtering it externally?

The easiest way to do this in your new network setup is to only assign IP addresses to the nodes in the VLAN you want them to be available, commonly referred to as a management network.
If you still want to limit the access further you might want to look into using the integrated firewall.

Yours sincerely
Jonas

[1] https://pve.proxmox.com/pve-docs/chapter-pvecm.html#pvecm_cluster_requirements
[2] https://forum.proxmox.com/threads/change-cluster-nodes-ip-addresses.33406/#post-183483
[3] https://pve.proxmox.com/pve-docs/chapter-pvesdn.html
[4] https://pve.proxmox.com/pve-docs/chapter-pvesdn.html#pvesdn_setup_example_vlan

 

[Yuri_AR]

Hi! Thank you for your answer

Before diving into your actual topic, i emphasize the importance of backups for this operation. Something might go wrong and i don't want you to lose data.

Thank you! We do have periodic backups, and I plan to do an additional copy before I touch anything.

In this case i would recommend using the SDN stack from here on, as this makes configuring VMs and adding or removing f.ex. VLANs much more transparent, imho.

For reference i link the Corosync requirements [1], as they are the foundation for using HA in this scenario.

Please keep in mind to separate Corosync traffic from any form of Storage or VM traffic and use these interfaces only as redundant links or you might run into problems especially with HA enabled.

Yes, my plan is to have a separate ethernet link for corosync (Running on a small, unmanaged switch), then have a decent network for the shared storage that I'll upgrade in the future. One of my concerns is changing the current configuration to enable the new setup without Completely breaking the cluster or being unable to connect to the affected nodes to fix whatever happens.

"shared storage". Are you using Ceph or some other form of shared storage?

Not at the moment, but I'm hoping to add that capability soon-ish. Anything I should be taking into consideration is, again, more than welcome. I have used proxmox for a while but I've never had to deal with clustering, HA or shared storage before.

Changing the IP address of a cluster node is not recommended but still possible.
This short instruction [2] should suffice.

Thanks! I'll look into it and come back with any questions.

Having a virtualised firewall is not unusual and such not "shitty" per se.

Agreed, not shitty by itself, but certainly not optimal for this particular use case. It has given me headaches before and I'm looking forward to getting rid of it.

I'm a little confused on your current network setup since you only mention one node having the VLANs configured. Is there a reason why this is? Are all the VLAN interfaces configured on the pfsense or Proxmox side?

Yes, sorry, the cluster has been built over time with scraps, so everything is a bit chaotically patched in and it's a bit nonsensical even when explained correctly.

There are 4 nodes in the cluster. All of them are connected to an unmanaged switch, and that switch directly to the telco's router. The nodes have whatever IP was free on the router's DHCP block. One of the nodes, the one with the pfsense machine, has the VLAN trunk going to a managed switch where those vlans are used. The other nodes expose machines/services directly on the telco's router subnet.

So, right now, we have a cluster but none of the features that would justify having a cluster, because we can't have HA without shared storage, and all the vlan specific machines have to be spun in that one node that has the VLAN trunk. I need to figure out how to change the proxmox configuration so the setup makes sense without breaking anything.

In this matter "access" might mean different things, but from the rest of the question i figure you mean being able to give Containers and VMs access to the VLAN and not being able to access the Web GUI etc. from there, correct?
In this case i still recommend looking into the SDN [3]. It might seem intimidating at first, but will make your life easier in the long run imho. For your deployment you mostly have to look at the VLAN zone [4] and can leave all from Fabrics to IPAM to the side at first.
This will allow you to define VLANs on the datacenter level and having them available at all nodes at once.

Thank you, I'll look into SDN. A follow up question, would I still be able to use the pfsense router to manage/firewall the vlans? I will eventually add the physical/external appliance, but I'd like to keep the virtualized one working until then. Ideally, the less I have to change, the less chances I have of this blowing up in my face.

Once again, thanks! Let me know if there's any important information missing, I've been exploring the available options for a day or two and I'm still not very clear on a lot of things.

 

[Yuri_AR]

Hi again!

I've been reading about SDN and I have a few more specific questions.

* The difference between VLAN and VXLAN is that VXLANs make the SDN work wherever a cluster node may be as long as they have a connection, according to the documentation. In my case all the nodes are in the same physical rack. I'm assuming I don't have much use for VXLAN, but I wanted to ask if people use it as some sort of default option anyway, or if there's any additonal rule of thumb or best practice when it comes down to choosing between the two.

* If I understood the documentation correctly, I need a zone, in which I'll be configuring the VNets (I'm picturing VNets as equivalent to VLANs), and then on each VNet I need to create a subnet, which defines the IP range of the VNet. I'm having trouble visualizing what a zone would be equivalent to. Are they similar to normal VMBRs associated to an ethernet port?

* How come you have to create subnets in VLANs/VNets instead of defining an IP range in the net itself? I don't get why subnets are a thing.

* How does this play with my existing routing solution? Will I have a conflict if I define a VNet with a certain IP range and then connect it to a router that has a similar configuration? Is it OK if the configuration is the same and proxmox has the DHCP server disabled? I'm assuming it will work as long as the routing VM has access to the zone/bridge but I want to be sure, I'm still hazy on the details.

* Also, I wanted to ask how well would this SDN setup play with the IP change for the cluster nodes. My idea is to create a VLAN for cluster nodes using the SDN feature, and then change the IPs of the nodes to be in that management VLAN subnet. Would this break functionality or is the cluster going to work as long as they are all in the same subnet and have internet access?

* As to separating the corosync network from the rest of the cluster/machine/etc... Would it be recommended to add a corosync VLAN for this traffic? I'd like to do this and add a QoL rule to prevent heavy network traffic from degrading the cluster, but I'm not at all sure if this concern is reasonable or if having everything on the same management VLAN I mentioned earlier would be fine.

Thanks again! The help has been super useful so far

 

[j.theisen]

Sorry for the late reply!

* The difference between VLAN and VXLAN is that VXLANs make the SDN work wherever a cluster node may be as long as they have a connection, according to the documentation. In my case all the nodes are in the same physical rack. I'm assuming I don't have much use for VXLAN, but I wanted to ask if people use it as some sort of default option anyway, or if there's any additonal rule of thumb or best practice when it comes down to choosing between the two.

VXLAN is for a scenario where Layer 2 functionality is needed across sites where no direct Layer 2 link exists. So in your case with all nodes locally this would be a lot of overhead for very limited to no advantages apart from having seen the technology.
VLANs should be completely sufficient in your case.

* If I understood the documentation correctly, I need a zone, in which I'll be configuring the VNets (I'm picturing VNets as equivalent to VLANs), and then on each VNet I need to create a subnet, which defines the IP range of the VNet. I'm having trouble visualizing what a zone would be equivalent to. Are they similar to normal VMBRs associated to an ethernet port?

For a VLAN zone yes, every VNet is a separate VLAN with its own VLAN tag.
You CAN create a Subnet in the SDN to use the features outlined here [1] but you don't have to.
Zones on the other hand are the top container of the SDN stack on a per node level. Meaning a node can have multiple Zones i.e. a Simple Zone for Containers and VMs which should be isolated to that node and a VLAN zone for VMs which should be able to communicate with services outside the node.

* How come you have to create subnets in VLANs/VNets instead of defining an IP range in the net itself? I don't get why subnets are a thing.

This was done to allow multiple IP ranges on the same network.
I personally don't have a good example where you would want this, but it is still a feature.

How does this play with my existing routing solution? Will I have a conflict if I define a VNet with a certain IP range and then connect it to a router that has a similar configuration? Is it OK if the configuration is the same and proxmox has the DHCP server disabled? I'm assuming it will work as long as the routing VM has access to the zone/bridge but I want to be sure, I'm still hazy on the details.

No this shouldn't interfere since PVE itself doesn't use the defined Subnet per se but just holds it for management purposes (SNAT configurations excluded)
The DHCP plugin for Proxmox currently only supports simple Zones, so with VLAN zones this wouldn't work.
Exactly, as long as your pfsense has the appropriate VLAN interfaces defined in PVE and the appropriate IP config on those interfaces communication should work just fine between this and other nodes.

Also, I wanted to ask how well would this SDN setup play with the IP change for the cluster nodes. My idea is to create a VLAN for cluster nodes using the SDN feature, and then change the IPs of the nodes to be in that management VLAN subnet. Would this break functionality or is the cluster going to work as long as they are all in the same subnet and have internet access?

Changing IP addresses of cluster nodes is a little bit tricky and can in the worst case break the cluster. There are several threads discussing this topic [2] which should help.
Relying on the SDN to assign management IP addresses is also not the best option in my opinion. I have seen working examples of this but would personally much prefer a dedicated interface configured locally for this.

As to separating the corosync network from the rest of the cluster/machine/etc... Would it be recommended to add a corosync VLAN for this traffic? I'd like to do this and add a QoL rule to prevent heavy network traffic from degrading the cluster, but I'm not at all sure if this concern is reasonable or if having everything on the same management VLAN I mentioned earlier would be fine.

Separating the Corosync traffic is definitely advised for clusters which expect moderate to heavy load on the currently used NICs.
There is a whole guide on this here [3]

Best regards
Jonas

[1] https://pve.proxmox.com/pve-docs/chapter-pvesdn.html#pvesdn_config_subnet
[2] https://forum.proxmox.com/threads/change-cluster-nodes-ip-addresses.33406/
[3] https://pve.proxmox.com/wiki/Separate_Cluster_Network

 

[Yuri_AR]

Hi Jonas.

Don't worry, I'm not in a hurry. Thank you so much for the help, I'll keep looking into this and I'll come back if I have any more specific questions