Configuration changes in existing cluster

[Yuri_AR]

Hi! I have a 4 node proxmox cluster, configured as follows:

* All nodes are connected to a switch, in the 172.16.0.0/24 IP range with one of their eth interfaces.
* One of the nodes has an additional ethernet interface acting as a VLAN trunk. This node also contains a VM with PFsense, managing the traffic on a bunch of VLANs.
* proxmox API/interface can be reached from any vlan in the .0 IP address, or in their 172.16.0.X address.

This architecture evolved over time in a rather chaotic way, and I'm now looking to improve and stabilize it. However, most if not all of the changes I want to make are not trivial to do and/or can't be done with the web interface, and I'm looking for advice on what the best way to proceed would be. I'm also looking for a sanity check, just in case any of my ideas are unsound.

1) I would like to have the node communication traffic in their own internal, physically separate network. This is in preparation for a 10Gb network upgrade further down the road, both for cluster traffic and shared storage. I've found a bunch of old manual procedures on the internet, but they all seem a bit sketchy and I'm not sure they are up to date anymore. I've looked on the web GUI and there's no option for this there either. Deleting/wiping nodes, re-creating the cluster, or starting from scratch is not viable. Would a managed switch be beneficial here or would using any of the remaining ethernet interfaces with a normal switch be enough?

This future upgrade will also include a physical, actual netgate appliance. Until then, WAN traffic would have to come in through the node which is now managing the VLANs. I'm fully aware of how shitty this arrangement is.

2) I want all the nodes to have access to the same VLANs, both for ease of deployment of new containers in the right networks across nodes, but also to allow us to use HA capabilities more easily. Would you recommend to have a shared VLAN trunk in one of the interfaces on each node, or would it be better to use the SDN feature? The SDN feature looks both what I need and also complete overkill. I don't really need separate zones for the different nodes or any of the advanced features, but I need, say, a machine in VLAN 99 to have HA capabilities across nodes.

3) I want to limit access to the web interface and API to a particular VLAN. This is one of the reasons to have an internal network. However, I've seen that the .0 IP address in each VLAN does have a proxmox interface/API available. Is there any way to specify in proxmox where this is served without breaking cluster communications, rather than filtering it externally?

4) Any other suggestions, things I may be missing, or best practices I may not be following for lack of experience are more than welcome.

 

[j.theisen]

Hi @Yuri_AR

thanks for posting on the forum!

Before diving into your actual topic, i emphasize the importance of backups for this operation. Something might go wrong and i don't want you to lose data.

That being said:

looking to improve and stabilize it

In this case i would recommend using the SDN stack from here on, as this makes configuring VMs and adding or removing f.ex. VLANs much more transparent, imho.

For reference i link the Corosync requirements [1], as they are the foundation for using HA in this scenario.

1) I would like to have the node communication traffic in their own internal, physically separate network. This is in preparation for a 10Gb network upgrade further down the road, both for cluster traffic and shared storage.

Please keep in mind to separate Corosync traffic from any form of Storage or VM traffic and use these interfaces only as redundant links or you might run into problems especially with HA enabled.
You mention "shared storage". Are you using Ceph or some other form of shared storage?

Changing the IP address of a cluster node is not recommended but still possible.
This short instruction [2] should suffice.

Would a managed switch be beneficial here or would using any of the remaining ethernet interfaces with a normal switch be enough?

A managed switch would f.ex. allow you to separate the cluster traffic into a separate VLAN without having to configure anything on the hosts network interface which is best practice but not required.

This future upgrade will also include a physical, actual netgate appliance. Until then, WAN traffic would have to come in through the node which is now managing the VLANs. I'm fully aware of how shitty this arrangement is.

Having a virtualised firewall is not unusual and such not "shitty" per se.
I'm a little confused on your current network setup since you only mention one node having the VLANs configured. Is there a reason why this is? Are all the VLAN interfaces configured on the pfsense or Proxmox side?

2) I want all the nodes to have access to the same VLANs, both for ease of deployment of new containers in the right networks across nodes, but also to allow us to use HA capabilities more easily. Would you recommend to have a shared VLAN trunk in one of the interfaces on each node, or would it be better to use the SDN feature? The SDN feature looks both what I need and also complete overkill. I don't really need separate zones for the different nodes or any of the advanced features, but I need, say, a machine in VLAN 99 to have HA capabilities across nodes.

In this matter "access" might mean different things, but from the rest of the question i figure you mean being able to give Containers and VMs access to the VLAN and not being able to access the Web GUI etc. from there, correct?
In this case i still recommend looking into the SDN [3]. It might seem intimidating at first, but will make your life easier in the long run imho. For your deployment you mostly have to look at the VLAN zone [4] and can leave all from Fabrics to IPAM to the side at first.
This will allow you to define VLANs on the datacenter level and having them available at all nodes at once.

3) I want to limit access to the web interface and API to a particular VLAN. This is one of the reasons to have an internal network. However, I've seen that the .0 IP address in each VLAN does have a proxmox interface/API available. Is there any way to specify in proxmox where this is served without breaking cluster communications, rather than filtering it externally?

The easiest way to do this in your new network setup is to only assign IP addresses to the nodes in the VLAN you want them to be available, commonly referred to as a management network.
If you still want to limit the access further you might want to look into using the integrated firewall.

Yours sincerely
Jonas

[1] https://pve.proxmox.com/pve-docs/chapter-pvecm.html#pvecm_cluster_requirements
[2] https://forum.proxmox.com/threads/change-cluster-nodes-ip-addresses.33406/#post-183483
[3] https://pve.proxmox.com/pve-docs/chapter-pvesdn.html
[4] https://pve.proxmox.com/pve-docs/chapter-pvesdn.html#pvesdn_setup_example_vlan

 

[Yuri_AR]

Hi! Thank you for your answer

Before diving into your actual topic, i emphasize the importance of backups for this operation. Something might go wrong and i don't want you to lose data.

Thank you! We do have periodic backups, and I plan to do an additional copy before I touch anything.

In this case i would recommend using the SDN stack from here on, as this makes configuring VMs and adding or removing f.ex. VLANs much more transparent, imho.

For reference i link the Corosync requirements [1], as they are the foundation for using HA in this scenario.

Please keep in mind to separate Corosync traffic from any form of Storage or VM traffic and use these interfaces only as redundant links or you might run into problems especially with HA enabled.

Yes, my plan is to have a separate ethernet link for corosync (Running on a small, unmanaged switch), then have a decent network for the shared storage that I'll upgrade in the future. One of my concerns is changing the current configuration to enable the new setup without Completely breaking the cluster or being unable to connect to the affected nodes to fix whatever happens.

"shared storage". Are you using Ceph or some other form of shared storage?

Not at the moment, but I'm hoping to add that capability soon-ish. Anything I should be taking into consideration is, again, more than welcome. I have used proxmox for a while but I've never had to deal with clustering, HA or shared storage before.

Changing the IP address of a cluster node is not recommended but still possible.
This short instruction [2] should suffice.

Thanks! I'll look into it and come back with any questions.

Having a virtualised firewall is not unusual and such not "shitty" per se.

Agreed, not shitty by itself, but certainly not optimal for this particular use case. It has given me headaches before and I'm looking forward to getting rid of it.

I'm a little confused on your current network setup since you only mention one node having the VLANs configured. Is there a reason why this is? Are all the VLAN interfaces configured on the pfsense or Proxmox side?

Yes, sorry, the cluster has been built over time with scraps, so everything is a bit chaotically patched in and it's a bit nonsensical even when explained correctly.

There are 4 nodes in the cluster. All of them are connected to an unmanaged switch, and that switch directly to the telco's router. The nodes have whatever IP was free on the router's DHCP block. One of the nodes, the one with the pfsense machine, has the VLAN trunk going to a managed switch where those vlans are used. The other nodes expose machines/services directly on the telco's router subnet.

So, right now, we have a cluster but none of the features that would justify having a cluster, because we can't have HA without shared storage, and all the vlan specific machines have to be spun in that one node that has the VLAN trunk. I need to figure out how to change the proxmox configuration so the setup makes sense without breaking anything.

In this matter "access" might mean different things, but from the rest of the question i figure you mean being able to give Containers and VMs access to the VLAN and not being able to access the Web GUI etc. from there, correct?
In this case i still recommend looking into the SDN [3]. It might seem intimidating at first, but will make your life easier in the long run imho. For your deployment you mostly have to look at the VLAN zone [4] and can leave all from Fabrics to IPAM to the side at first.
This will allow you to define VLANs on the datacenter level and having them available at all nodes at once.

Thank you, I'll look into SDN. A follow up question, would I still be able to use the pfsense router to manage/firewall the vlans? I will eventually add the physical/external appliance, but I'd like to keep the virtualized one working until then. Ideally, the less I have to change, the less chances I have of this blowing up in my face.

Once again, thanks! Let me know if there's any important information missing, I've been exploring the available options for a day or two and I'm still not very clear on a lot of things.