'migrate' a MDS to another server

[stefws]

I want to remove a server from both my Ceph cluster and PVE cluster in order to reinstall it.

I've removed all mon+osds, but it hold one of my two mrs instances.

How do I add another server as a mds and remove the old one?

Can not find anything on [re]moving MDS expects for ceps-deploy, only now it says:

[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.21): /usr/bin/ceph-deploy mds create node2
[ceph_deploy][ERROR ] ConfigError: Cannot load config: [Errno 2] No such file or directory: 'ceph.conf'; has `ceph-deploy new` been run in this directory?

thought I knew which server was used as admin when installing cep, only they all act the same and has the same ¨/.cephdeploy.conf content...

 

[tom]

ceph-deploy is not used on Proxmox VE Ceph server, instead we have "pveceph" - so I assume you talk here about a separate ceph cluster and not about "Proxmox VE Ceph Server"?

 

[stefws]

Right, I properly did install my Ceph cluster through pveceph initial

One easily gets confused when browsing various docs including Ceph.com.

Still I believe I used: "ceph-deploy mds create" to create my two MDS with, so .. how else to create MDSs?

Anyway anyone knows how to remove one of two and create another MDS?

 

[tom]

just to note, pveceph does not support MDS (yet), we use/install only MON and OSD.

 

[stefws]

I know, but I have created successfully two MDS earlier, though I right now are a bit confused as to how...

root@node4:~# ceph -s
    cluster 16fe2dcf-2629-422f-a649-871deba78bcd
     health HEALTH_OK
     monmap e19: 3 mons at {0=10.0.3.4:6789/0,1=10.0.3.2:6789/0,2=10.0.3.1:6789/0}, election epoch 1282, quorum 0,1,2 2,1,0
     [B]mdsmap[/B] e97: 1/1/1 up {0=1=up:active}, 1 up:standby
     osdmap e1757: 18 osds: 18 up, 18 in
      pgmap v116143: 1280 pgs, 5 pools, 79177 MB data, 38550 objects
            230 GB used, 2138 GB / 2369 GB avail
                1280 active+clean

So question is how to modify/move one MDS to another host...

 

[stefws]

I properly did something along these lines as I visited this blog before...

 

[stefws]

Yes this way I managed to create another mds and thus could retire one

ceph.conf content:

[mds]
         keyring = /var/lib/ceph/mds/mds.$id/keyring


[mds.0]
         host = node3


[mds.1]
         host = node4


[mds.2]
         host = node2



Added a third MDS mds.2:

root@node2:~# mkdir /var/lib/ceph/mds/mds.2
root@node2:~# ceph auth get-or-create mds.2 mds 'allow ' osd 'allow *' mon 'allow rwx' > /var/lib/ceph/mds/mds.2/keyring
root@node2:~# service ceph start mds.2
=== mds.2 === 
Starting Ceph mds.2 on node2...
starting mds.2 at :/0
root@node2:~# 



root@node2:~# ceph -s
    cluster 16fe2dcf-2629-422f-a649-871deba78bcd
     health HEALTH_OK
     monmap e29: 3 mons at {0=10.0.3.4:6789/0,1=10.0.3.2:6789/0,2=10.0.3.1:6789/0}, election epoch 1314, quorum 0,1,2 2,1,0
     [B]mdsmap[/B] e98: 1/1/1 up {0=1=up:active}, [B]2[/B] up:standby
     osdmap e1757: 18 osds: 18 up, 18 in
      pgmap v126494: 1280 pgs, 5 pools, 79180 MB data, 38553 objects
            230 GB used, 2138 GB / 2369 GB avail
                1280 active+clean
  client io 10176 B/s wr, 4 op/s


Retired/Stopped mds.0:

root@node3:~# service ceph stop mds.0
=== mds.0 === 
Stopping Ceph mds.0 on node3...kill 3623...done


root@node3:~# ceph -s
    cluster 16fe2dcf-2629-422f-a649-871deba78bcd
     health HEALTH_OK
     monmap e29: 3 mons at {0=10.0.3.4:6789/0,1=10.0.3.2:6789/0,2=10.0.3.1:6789/0}, election epoch 1314, quorum 0,1,2 2,1,0
     [B]mdsmap[/B] e99: 1/1/1 up {0=1=up:active}, [B]1[/B] up:standby
     osdmap e1757: 18 osds: 18 up, 18 in
      pgmap v126494: 1280 pgs, 5 pools, 79180 MB data, 38553 objects
            230 GB used, 2138 GB / 2369 GB avail
                1280 active+clean