Editing crushmap

[waynevaghi]

Hi have a 3 node cluster.
One node had a physical failure. We removed it from the cluster but it seems some entries remained behind in the crushmap:

Is it safe for me to manually edit the crushmap and remove all entries pointing to the host NZ-AUk-Host1?

# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable chooseleaf_stable 1
tunable straw_calc_version 1
tunable allowed_bucket_algs 54

# devices
device 0 osd.0 class ssd
device 1 osd.1 class ssd
device 3 osd.3 class ssd
device 4 osd.4 class ssd
device 5 osd.5 class ssd
device 6 osd.6 class ssd
device 7 osd.7 class ssd
device 8 osd.8 class ssd
device 12 osd.12 class ssd

# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 zone
type 10 region
type 11 root

# buckets
host NZ-AUK-Host2 {
id -3 # do not change unnecessarily
id -4 class ssd # do not change unnecessarily
# weight 1.746
alg straw2
hash 0 # rjenkins1
item osd.0 weight 0.437
item osd.7 weight 0.437
item osd.8 weight 0.437
item osd.12 weight 0.437
}
host NZ-AUK-Host3 {
id -5 # do not change unnecessarily
id -6 class ssd # do not change unnecessarily
# weight 2.183
alg straw2
hash 0 # rjenkins1
item osd.1 weight 0.437
item osd.3 weight 0.437
item osd.4 weight 0.437
item osd.5 weight 0.437
item osd.6 weight 0.437
}
host NZ-AUK-Host1 {
id -7 # do not change unnecessarily
id -8 class ssd # do not change unnecessarily
# weight 0.000
alg straw2
hash 0 # rjenkins1
}
root default {
id -1 # do not change unnecessarily
id -2 class ssd # do not change unnecessarily
# weight 3.929
alg straw2
hash 0 # rjenkins1
item NZ-AUK-Host2 weight 1.746
item NZ-AUK-Host3 weight 2.183
item NZ-AUK-Host1 weight 0.000
}

# rules
rule replicated_rule {
id 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit
}

# end crush map

 

[shrdlicka]

Have you removed the monitors, osd, managers, ... as well?

 

[waynevaghi]

Yes, everything has been removed. It is only in the crush map I still see some entries.

 

[shrdlicka]

If you would like to clean it up you would need to remove this section:

host NZ-AUK-Host1 {
   id -7        # do not change unnecessarily
   id -8 class ssd        # do not change unnecessarily
   # weight 0.000
   alg straw2
   hash 0    # rjenkins1
}

and from the root default section:

item NZ-AUK-Host1 weight 0.000

 

[waynevaghi]

Thank you, and this won't affect the 2 nodes remaining? We have a 3rd server arriving to replace the dead node. This changes won't affect adding a new node with the same IP's and name into the cluster?

 

[shrdlicka]

To answer quickly: the node already has a weight of 0.000, so it isn't used currently. If you did the cleanup [1] then it should be no problem adding the same node with the same name&IP again.


[1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_remove_a_cluster_node

 

[waynevaghi]

So I edited the crush map and then did a fresh install of of the PVE on a new node. I joined it to the cluster and installed CEPH but I cannot add any monitors. I get the error "Could not connect to ceph cluster despite configured monitors (500)"

All nodes are setup exactly the same following an internal document. How can I get around this error now?

 

[shrdlicka]

Did you check if there is still the config of the old mon in /etc/pve/ceph.conf?

What does pveceph status? Especially on the newly added node.

 

[waynevaghi]

Yes, /etc/pve/ceph.conf was clear of any config of the old node.
If I run pveceph status it just times out.
I also can't ping the public IP I configured for ceph on the new node but can ping the other IP's from the new node.

 

[shrdlicka]

ip a shows the ip adsress is configured? Is there a firewall active? Can you reach ports 3300 or 6789 (ceph mon ports) on the other nodes (i.e. does telnet [host] 3300 connect)?

 

[mbstein]

This looks so familiar, like THE ISSSUES ON my mini"CLUSTER"

IP Configuration seems to be ok, but TCP ports 3300 and 6789 are not open on the newly added node,
and any attempt to create monitors fails with a timeout.

what process might need to be repaired or resarted to have the ports for monitor communication opened?