Blocking 22 on firewall breaks ceph networking

[thesquiffy]

Hi, I have current configuration running on 3x ovh servers' cluster:

vrack vlans:

10.240.10.x -> vlan for servers communications and VMs
10.240.99.x -> vlan for ceph

I want to close ssh ports on al nodes from the outside, keeping them to be able to talk each other and keep everything in sync.

To do so, I have created a IPset with all the addresses either from 10.240.10.x / 10.240.99.x vlans
Then I have added a rule to allow SSH for IPSet as destination.
Then a deny for SSH in general.

When I start the firewall, ceph starts desyncing and I need to turn the firewall off.

I don't really unterstand if firewall rules are processed in a specific order; I see I cannot change any priority and any new rule it's always added at first place

Any help it's really appreciated.
Thank you!

 

[ph0x]

You can drag and drop the rules around and yes, they are in a first-come-first-serve order.
ssh shouldn't break Ceph, but did you maybe block the Ceph ports as well? You can easily add a firewall rule with a Ceph macro for your respective VLAN. Keep in mind that if you enable the firewall with DROP policy, everything that you don't have an allowing rule for gets blocked.

 

[thesquiffy]

You can drag and drop the rules around and yes, they are in a first-come-first-serve order.
ssh shouldn't break Ceph, but did you maybe block the Ceph ports as well? You can easily add a firewall rule with a Ceph macro for your respective VLAN. Keep in mind that if you enable the firewall with DROP policy, everything that you don't have an allowing rule for gets blocked.

Hi, thank you, I didn't know they were draggable

Now I'm just using default DROP on INPUT with two rules:

1- allow 8006 (to keep managing)
2- allow for CEPH macro and source/dest set with the ipset containing all the nodes

Anyways, when I'm going to activate the firewall ceph section still begins desyncing, high latencies and so.

Can't really figure out how to let it keep working.

 

[ph0x]

Is the ceph macro enabled? Let me think about for a minute, I had the same issue when first enabling the firewall. Maybe I can recall what the problem was.

 

[ph0x]

I had ssh disabled via firewall now for several minutes and that had no impact on Ceph at all.
Two possibilities come to my mind:
- The rule with the macro is not enabled
- The IPSet with the cluster members does not contain the adresses of the Ceph network

 

[thesquiffy]

I had ssh disabled via firewall now for several minutes and that had no impact on Ceph at all.
Two possibilities come to my mind:
- The rule with the macro is not enabled
- The IPSet with the cluster members does not contain the adresses of the Ceph network

I honestly don't even know how to enable/disable macros....
Yes, IPSet includes 3x cluster network ips and 3x ceph network ips; double checked it.

I have though 2x bridge interfaces: vmbr0 connected to the public and vmbr1 bridged to 2nd nic.

 

[ph0x]

There is a small checkbox with the header "On" in every rule to enable it. Is that box checked?

 

[thesquiffy]

Hi, yes, it was turned on.

I think I have partially solved my problem adding ALLOW both IN and OUT directions for ceph macro.
Now ceph stays up-green.
Anyways, I'm unable to move data between hosts, for instance if I try to migrate a VM from node to node, task begin but stays there forever:

task started by HA resource agent
2021-04-07 09:27:20 starting migration of VM 100 to node 'node2' (xxx.xxx.xxx.xxx)


Maybe there is any other ports I need to manage to allow data transfer to occur?

Thank you!

 

[ph0x]

Migration is done on ports 60000:60050 on the network that you define under Datacenter->Options->Migration

 

[thesquiffy]

Trying to limit the issue to one thing at once, I'm currently working on port 22 only (macro ssh). If I activate it (in, accept, source my pve cluster ipset), no ssh connections can be made between hosts (attempted through ssh command directly)... If I ssh to their public endpoints... it works.

This is going to drive me crazy....

I'm going to test opening 60000:60050 for data migration though.

 

[thesquiffy]

I finally spot the problem... Quick note for anyone else in the future can find this thread usefull: it was the wrong interface...

TL;DR
I run small lab cluster and each node is hosted as bare metal in ovh servers; they are equipped with two network interfaces: one is connected to public, the other to private vlans (via vrack)

I linked the private interface to two vlan: 10.240.10.x for VM's communication and 10.240.99.x for ceph service

So, to be able to allow the VMs to also talk to other physical servers I own in same vrack, I did a second bridge (other than the default vmbr0) which is vmbr1 (bridge-ports eno2.ID and bridge-vlan-aware yes).

I was attempting to close everything from public, working to ALLOW a few ports on vmbr0 instead of vmbr1 which is DROP by default policy if you just turn on the firewall: my mistake.

So, my problem wasn't really related to ceph, it was an attempt to tight security on public side.

My final solution:

- install and configure nignx to reverse from higher, non standard port (>1024) to local 127.0.0.1:8006 also reusing my custom certificate.
- allow sshd to listen to a second non standard port (>1024) and keep 22 indeed.
- lock everything on vmbr0, allowing only for nginx and ssh secondary ports; thus finally closing 22 and 8006
- allow everything on vmbr1 for protocols tcp and icmp (who don't like pings here? )

I would also like to leave here my nginx config; it can be done with whatever you like, I'm comfortable with this reverse proxy:

server {
        server_name _;

        # put something non standard port here
        # you have to use your own custom certificate
        listen XXXXX ssl http2;
        listen [::]:XXXXX ssl http2;
        ssl        on;
        ssl_certificate         /etc/pve/local/pveproxy-ssl.pem;
        ssl_certificate_key     /etc/pve/local/pveproxy-ssl.key;

        location / {
                proxy_pass         https://127.0.0.1:8006/;
                proxy_redirect     off;
                proxy_set_header   Host             $host;
                proxy_set_header   X-Real-IP        $remote_addr;
                proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
                proxy_http_version 1.1;
                proxy_set_header   Upgrade $http_upgrade;
                proxy_set_header   Connection "Upgrade";
        }
}

Thank you.