PVE cluster firewall levels (cluster/node/vm-lxc)

[Urbaman]

Hi,

I do have a local three-node cluster, all the nodes on same LAN, internal traffic managed with bonds/vmbr/vlans to a single managed switch.
All of the nodes have the very same network settings (NICS-interface-bond(s)-vmbr(s)-vlan(s)).
I have many services running between the nodes, on different vlans (ceph, a three-vm gluster cluster, a kubernetes cluster, ....)

Now, I am at the moment managing the internal cluster network with a pfsense vm, getting the external traffic and managing all of the internal vlans.
At the moment, the PVE firewall is disabled on all levels (cluster, host/nodes, vm/lxc).

My next step would be to enable the pve firewall, to harden the security of the cluster even before getting to the pfSense vm, but I need some insight on how it actually works, before making disasters.
If I enable it only on cluster level (not host or vm/lxc level, as they would be managed by the pfSense vm), would the inter-nodes traffic be closed (ceph for example)?
Or do I need to actually copy all of the pfSense "internal" rules on the cluster level pve-firewall (like enabling ceph traffic for example)?

Thanks.

 

[Chris]

Hi,
there are default firewall rules which assure that traffic necessary for the cluster to work are allowed, see https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pve_firewall_default_rules including rules for Ceph clusters managed by Proxmox VE.

If you enable the firewall on the cluster level, but not on the node and/or VM level (for VMs you will also have to check that is enabled/disabled on the virtual NIC), the firewall will not filter any traffic, as the rules are not applied to neither.

Note however that you will have to adapt your rules for e.g. your kubernetes cluster and all the other customized setups.

 

[Urbaman]

So, for further checking if I understood.

1) Keeping the firewall only on cluster level is no use: I have to apply the rules at node and/or vm level.
2) As soon as I apply it at node level, and say I have a dedicated vlan for glusterfs storage network (say vlan90, network 10.0.90.x), I would need to open that traffic (and keep it dedicated) to permit the comunication between the different glusterfs vms across the three nodes (that is, replicating what pfSense is already doing).

 

[Chris]

So, for further checking if I understood.

1) Keeping the firewall only on cluster level is no use: I have to apply the rules at node and/or vm level.
2) As soon as I apply it at node level, and say I have a dedicated vlan for glusterfs storage network (say vlan90, network 10.0.90.x), I would need to open that traffic (and keep it dedicated) to permit the comunication between the different glusterfs vms across the three nodes (that is, replicating what pfSense is already doing).

Enabling/disabling the firewall on cluster level acts as a global on/off switch for the firewall in the cluster. In order to have effect, it must be enabled on cluster level. If this is the case, you can individually switch it on/off on node and/or VM level. For traffic in between VMs, you can define rules on the datacenter level, specifing a security groups for these VMs. See also https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pve_firewall_security_groups

Best is to set up a test environment and experiment with these firewall features and their effect if you don't want to risk your currently running system.