Cluster, Network, HA, Configuration

[pille99]

hello all

i have already some open questions, i am pretty new to proxmox. i love to play around but this time i dont have time (in 2 days my esx will be decom and i need to get everything up and running).

i have a 3 node cluster, ceph with 10 gb cluster and 1 gb public network, 1 nic for income traffic with 2 IPs, and currently 1 NIC is not in use (if somebody knows how to use it, please tell me - its a internal network, without access to internet)
the most videos or description i have found are more for home networks. only 1 server but never saw a complex environment. here are my questions

1. how the nodes talk to each other ?
2. how the nodes know where to flow the traffic (income IP on Node 1, but the VM is on Node 2) ?
3. am i correct to setup on all 3 Servers the same subnets ? otherwise on a failover the network would be unknown ?
4. advice needed for firewall. my concept is opnsense (best firewall ever), the public NIC will be configured to it, and the Private LANs connected to it too, everything goes to the firewall and they manage the in/out-going traffic. pretty simple. maybe better to have on all Nodes a firewall ?
4.a, if an HA event occures the vm will be started on the other nodes and use that config/firewall (keep in mind - all 3 Nodes have complete different subnets and gateways)
5. does it make sense to configure the firewall on the Node itself ? only traffic from all Nodes allowed (maybe dns too) ?

as you understand - i need to fully understand what happens in case of failure, what does the network do and if everything will continuous with the services. thx for your input. much appreciated

 

[shrdlicka]

Hi,

1.) are you talking here about the pve cluster network [1], Ceph [2] or both?
2.) The basic config would be linux bridges. A VM is running on a node, if it gets switched to some other node, the underlying infrastructure (switches, routers ...) have to take care to send data to another port. The nodes itself are not doing anything here as far as I know.
3.) Not exactly sure what you mean here. Most people have their nodes on the same subnet.
4.) Depends on what you are planing to do on your hosts. There is a firewall [3] but it has some limitations.
4.a) yes it will use that firewall, and bring its own rules with it as well
5.) hard to tell from what you are writing, some people have a firewall between their nodes and the internet and do not use the PVE firwall


Regards



[1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pvecm
[2] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pveceph
[3] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pve_firewall