Ceph mons fail with cluster firewall rules

[ben90818532]

Hey folks,

I'm seeing an issue when the cluster firewall is enabled with the Ceph Macro (both in and out).

1) Enable firewall

2) Immediately ceph fills with mons going down and keeps going until the firewall is disabled.

I've even tried setting the default input and output policies to ACCEPT (and disabled all other rules) and it still fails, only when I disable the firewall entirely does it start working again.

[global]
     auth client required = cephx
     auth cluster required = cephx
     auth service required = cephx
     cluster network = 192.168.7.0/24
     fsid = 174a37a3-2363-41ee-904c-57735daee0fd
     keyring = /etc/pve/priv/$cluster.$name.keyring
     mon allow pool delete = true
     osd journal size = 5120
     osd pool default min size = 2
     osd pool default size = 3
     public network = 192.168.7.0/24

[mds]
     keyring = /var/lib/ceph/mds/ceph-$id/keyring

[osd]
     keyring = /var/lib/ceph/osd/ceph-$id/keyring

[mon.server1]
     host = server1
     mon addr = 192.168.7.43:6789

[mon.server2]
     host = server2
     mon addr = 192.168.7.40:6789

[mon.server3]
     host = server3
     mon addr = 192.168.7.41:6789

 

[sb-jw]

Did you enabled and configure the Firewall in Datacenter and Node level? If yes, please show us the FW config.

 

[ben90818532]

You gave me a tip (thank you!), firewalls were enabled on all nodes except for one.

I disabled and re-enabled the firewall on each node, now for some reason it works with the firewall enabled using the Ceph macro.

 

[ben90818532]

With zero changes for the last week, I just logged in today to find Ceph broken (couldn't communicate with other nodes), I disabled the firewall and it all started working again.

What is best practice RE Proxmox hosts and ceph firewall?

I even tried complete allow all access via firewall and it wouldn't work, It only worked with a completely disabled firewall.

 

[ben90818532]

Bingo, I think I've solved it when I saw this:

https://forum.proxmox.com/threads/host-unreachable-with-firewall.49273/#post-230218

"If you remove the source address it should be any (0.0.0.0/0). If you specify one it is assumed to be /32."

In my ceph config:
cluster network = 192.168.7.0/24
public network = 192.168.7.0/24

In the Ceph firewall rule I had to set the range as 192.168.7.40/24-192.168.7.45/24

From testing I think that's solved the problem.