[SOLVED] Add new host to exist pve-ceph-cluster

[adamli]

Hi guys for some reason, i only want to add an new host to a exist ceph-pve-cluster. Only add it to ceph but not pve-cluster.
I already get a ceph-pve-cluster and it work perfectly with three nodes(pve1, pve2, pve3), and pve4 is new to be add.

So I follow the step:
①install ceph using pve-gui version is as same as others'
②scp the `/etc/ceph/ceph.client.admin.keyring` and `/etc/ceph/ceph.conf`
Then I got following files in pve4

Spoiler: files

root@pve4:~# ls -l /etc/ceph/
total 12
-rw------- 1 ceph ceph 151 Feb 12 11:21 ceph.client.admin.keyring
-rw-r--r-- 1 ceph ceph 635 Feb 12 11:39 ceph.conf
-rw-r--r-- 1 root root 92 Oct 21 2021 rbdmap

③restart the mon and ceph service and using ceph -s to check but I got some error:

root@pve4:~# ceph -s
[errno 13] RADOS permission denied (error connecting to the cluster)
root@pve4:~# ceph -s
2024-02-12T11:54:36.593+0800 7fb6a7fff700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
[errno 13] RADOS permission denied (error connecting to the cluster)
root@pve4:~# ceph -s
  cluster:
    id:     ab52a496-d3b5-4881-a0d6-c9431f69c305
    health: HEALTH_WARN
            OSD count 0 < osd_pool_default_size 3
 
  services:
    mon: 1 daemons, quorum pve4 (age 13m)
    mgr: no daemons active (since 25m)
    osd: 0 osds: 0 up, 0 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:

I try to use chmod to give 600 to keyring file and 644 to the ceph.conf, but still not work.
Also I try to add the pve4 information to the ceph.conf by editor

Spoiler: ceph.conf

[global]
auth_client_required = cephx
auth_cluster_required = cephx
auth_service_required = cephx
cluster_network = 192.168.2.1/24
fsid = 699290a7-4a8e-4b7a-9760-8dad9bb64e1e
mon_allow_pool_delete = true
mon_host = 192.168.2.1 192.168.2.2 192.168.2.3 192.168.2.4
ms_bind_ipv4 = true
ms_bind_ipv6 = false
osd_pool_default_min_size = 2
osd_pool_default_size = 3
public_network = 192.168.2.1/24

[client]
keyring = /etc/pve/priv/$cluster.$name.keyring

[mon.pve1]
public_addr = 192.168.2.1

[mon.pve2]
public_addr = 192.168.2.2

[mon.pve3]
public_addr = 192.168.2.3

[mon.pve4]
public_addr = 192.168.2.4

Is that I using the wrong way to join the ceph-cluster? Need you guys to give some advice on it! Appreciate a lot!

 

[sb-jw]

If you don't join the cluster, you won't be able to access the same config status as the others. I would therefore strongly advise you to simply include it as a node. You are not obliged to use it, it doesn't have to be identical if it only does storage anyway. But basically I always recommend designing all nodes identically, that's the easiest thing.

By the way, you have no reason to make more than 3 mons. On the one hand, they always have to be odd and on the other hand, 5 Mons is only necessary for several hundred OSDs. You can go to 7 if you have a topology that requires it. The 7 Mons are an absolute exception.

 

[adamli]

If you don't join the cluster, you won't be able to access the same config status as the others. I would therefore strongly advise you to simply include it as a node. You are not obliged to use it, it doesn't have to be identical if it only does storage anyway. But basically I always recommend designing all nodes identically, that's the easiest thing.

By the way, you have no reason to make more than 3 mons. On the one hand, they always have to be odd and on the other hand, 5 Mons is only necessary for several hundred OSDs. You can go to 7 if you have a topology that requires it. The 7 Mons are an absolute exception.

Thanks for your advice! The new pve has it own VMs, so it cannot join the cluster, unless I backup and delete it first. For not familiar with how to quickly backup and restore all the things about vm I may not afford that risk to do it.

And I solve the problem by copy kering file to right place...

[client]
keyring = /etc/pve/priv/$cluster.$name.keyring

Anyway Thank you for your valuable advice, I'll remember it and try next time when i need to add another new host!

 

[sb-jw]

I don't know exactly what you did, but you definitely didn't do it right. This thing will definitely fall on your feet sooner or later.

If you want to use the CEPH on the new one, then your steps were definitely the wrong ones. If you wanted to put pve4 with its disk in the CEPH cluster, then your steps were also incorrect.

What you've apparently done here is two separate CEPH clusters, one with 3 mons and one with one mon. I don't have a plan as to how you're going to have them connected, but it won't end well.

 

[adamli]

I don't know exactly what you did, but you definitely didn't do it right. This thing will definitely fall on your feet sooner or later.

If you want to use the CEPH on the new one, then your steps were definitely the wrong ones. If you wanted to put pve4 with its disk in the CEPH cluster, then your steps were also incorrect.

What you've apparently done here is two separate CEPH clusters, one with 3 mons and one with one mon. I don't have a plan as to how you're going to have them connected, but it won't end well.

Because the VMs is already in pve4, I cannot just easily make it join the pve cluster, and i just need it to join ceph.
And I just want it to be the ceph client to consume the rdb and cephfs provided by the cluster made by pve1-3.
So I actually just use the pve-gui to install ceph, and copy the ceph.conf and kerying to the right place(Before I change the ceph.conf, it actually like you say got two separate CEPH clusters).

Right now I can successfully see the ceph cluster info(pve1-3) in pve4 command line ceph -s and successfully mount the cephfs provided by pve1-3.
Which part do you think is unreasonable? Please feel free to let me know.

 

[b101]

Because the VMs is already in pve4, I cannot just easily make it join the pve cluster, and i just need it to join ceph.
And I just want it to be the ceph client to consume the rdb and cephfs provided by the cluster made by pve1-3.
So I actually just use the pve-gui to install ceph, and copy the ceph.conf and kerying to the right place(Before I change the ceph.conf, it actually like you say got two separate CEPH clusters).

Right now I can successfully see the ceph cluster info(pve1-3) in pve4 command line ceph -s and successfully mount the cephfs provided by pve1-3.
Which part do you think is unreasonable? Please feel free to let me know.

and why not backup the vms on pve4 to any network share and restore to node 1-2-3 cluster and do things the correct way? once pve4's VMs are backed/restored, nuke pve4 and add it correctly to cluster.