[SOLVED] How to remove old mds from ceph? (actually slow mds message)

[lifeboy]

I had a failed node, which I replaced, but the MDS (for cephfs) that was on that node is still reported in the GUI as slow. How can I remove that? It's not in ceph.conf or storage.conf

MDS_SLOW_METADATA_IO 1 MDSs report slow metadata IOs
mdssm1(mds.0): 6 slow metadata IOs are blocked > 30 secs, oldest blocked for 543 secs
MDS_SLOW_REQUEST 1 MDSs report slow requests
mdssm1(mds.0): 7 slow requests are blocked > 30 sec

mds.0 does not exist anymore.

# pveceph mds destroy 0
disabling service 'ceph-mds@0.service'
stopping service 'ceph-mds@0.service'
cannot cleanup MDS 0 directory, '/var/lib/ceph/mds/ceph-0' not found
removing ceph auth for 'mds.0'

How do I remove the old mds.0 from ceph?

 

[lifeboy]

<bump>. Someone must have run into this issue before? Maybe it's just an annoyance and doesn't affect the cluster, but then again, maybe it does. I'd really like remove that.

 

[aaron]

The old node on which the MDS was running is gone and not part of the cluster anymore? Did you also remove it from the Proxmox VE cluster?

And you removed any mentions of the MDS in the /etc/pve/ceph.conf? They usually have a section similar to this:

[mds.cephtest1]
     host = cephtest1
     mds_standby_for_name = pve

Can you post the output of ceph -s please?

 

[lifeboy]

The old node on which the MDS was running is gone and not part of the cluster anymore? Did you also remove it from the Proxmox VE cluster?

Yes, the node first removed and then rebuilt. The node was completely removed before I added the rebuilt one.

And you removed any mentions of the MDS in the /etc/pve/ceph.conf? They usually have a section similar to this:

[mds.cephtest1]
     host = cephtest1
     mds_standby_for_name = pve

Can you post the output of ceph -s please?

# ceph -s
  cluster:
    id:     a6092407-216f-41ff-bccb-9bed78587ac3
    health: HEALTH_ERR
            1 MDSs report slow metadata IOs
            1 MDSs report slow requests
            noout flag(s) set
            Reduced data availability: 1 pg inactive, 1 pg incomplete
            392 stuck requests are blocked > 4096 sec. Implicated osds 4
 
  services:
    mon: 5 daemons, quorum hp1,2,s4,s5,s6
    mgr: s1(active), standbys: s5
    mds: cephfs-1/1/1 up  {0=sm1=up:active}, 3 up:standby
    osd: 23 osds: 23 up, 23 in
         flags noout


 
  data:
    pools:   4 pools, 1312 pgs
    objects: 525.15k objects, 1.63TiB
    usage:   8.07TiB used, 14.0TiB / 22.1TiB avail
    pgs:     0.076% pgs not active
             1311 active+clean
             1    incomplete
 
  io:
    client:   2.47KiB/s wr, 0op/s rd, 0op/s wr

Here's my /etc/pve/ceph.conf

# cat /etc/pve/ceph.conf
[global]
     auth client required = cephx
     auth cluster required = cephx
     auth service required = cephx
     cluster network = 192.168.121.0/24
     filestore xattr use omap = true
     fsid = a6092407-216f-41ff-bccb-9be...
     mon allow pool delete = true
     osd journal size = 5120
     osd op queue = wpq
     osd op queue cut off = high
     osd pool default min size = 1
     public network = 192.168.121.0/24

[client]
     keyring = /etc/pve/priv/$cluster.$name.keyring

[mds]
     keyring = /var/lib/ceph/mds/ceph-$id/keyring

[osd]
     bluestore_cache_autotune = true
     keyring = /var/lib/ceph/osd/ceph-$id/keyring

[mds.s1]
     host = s1
     mds standby for name = pve

[mds.s5]
     host = s5
     mds standby for name = pve

[mds.s4]
     host = s4
     mds standby for name = pve

[mds.sm1]
     host = sm1
     mds standby for name = pve

[mon.2]
     host = s1
     mon addr = 192.168.121.33:6789

[mon.s4]
     host = s4
     mon addr = 192.168.121.35:6789

[mon.hp1]
     host = hp1
     mon addr = 192.168.121.30:6789

[mon.s6]
     host = s6
     mon addr = 192.168.121.37:6789

[mon.s5]
     host = s5
     mon addr = 192.168.121.36:6789

root@sm1:~#

 

[alexskysilk]

delete all your mds daemons except one.

if the problem persists, go to the node that the phantom mds is appearing, and delete anything in /var/lib/ceph/mds/

Once you return to a healthy state, add ONE ADDITIONAL STANDBY MDS DAEMON. you dont need (nor benefit) to run so many daemons.

 

[lifeboy]

delete all your mds daemons except one.

if the problem persists, go to the node that the phantom mds is appearing, and delete anything in /var/lib/ceph/mds/

Once you return to a healthy state, add ONE ADDITIONAL STANDBY MDS DAEMON. you dont need (nor benefit) to run so many daemons.

I will do this.

However, I have created many mds daemons because these machines are old. Any of them could go down at some point and if the two that host the mds daemons go down at the same time I'd be screwed.

Is there a downside to having many mds daemons?

 

[alexskysilk]

mds daemons have to keep a record of all metadata synchronized between them. While only one metadata controller is ACTUALLY doing the serving (well, 2 if you choose active/active) the rest have to do all the work without any actual benefit.

As you can imagine, ALL MDS daemon have to continuously monitor the file system, and EACH OTHER. all of that causes traffic, and cpu load. I think you can see where this is going.

However, I have created many mds daemons because these machines are old. Any of them could go down at some point

old doesnt mean unstable. if you did mean unstable, remove them from the cluster. that's no way to run production.

 

[lifeboy]

mds daemons have to keep a record of all metadata synchronized between them. While only one metadata controller is ACTUALLY doing the serving (well, 2 if you choose active/active) the rest have to do all the work without any actual benefit.

As you can imagine, ALL MDS daemon have to continuously monitor the file system, and EACH OTHER. all of that causes traffic, and cpu load. I think you can see where this is going.

old doesnt mean unstable. if you did mean unstable, remove them from the cluster. that's no way to run production.

This cluster is primarily used as a backup. We run Proxmox Backup Server on it, replicate some databases to it and use it for testing, so it's not primary production. We have had old drives fail a couple of time though, but I hear what you're saying about too many mds's.

 

[lifeboy]

It has now become clearer to me what happens. I removed all the mds's and then the message changed. It seems that the active mds generates this message, although I have trouble finding the message from the console. The messager pertains to the active mds. Previously it was mdssm1, but now it's mdss1.

# ceph mds stat
cephfs-1/1/1 up  {0=s1=up:active}, 1 up:standby

Now the message in die GUI is:

1 MDSs report slow metadata IOs
mdss1(mds.0): 6 slow metadata IOs are blocked > 30 secs, oldest blocked for 6154 secs

and

1 MDSs report slow requests
mdss1(mds.0): 1 slow requests are blocked > 30 sec

It seems this is related to the error I still have in ceph.

# ceph -s
  cluster:
    id:     a6092407-216f-41ff-bccb-9bed78587ac3
    health: HEALTH_ERR
            1 MDSs report slow metadata IOs
            1 MDSs report slow requests
            noout flag(s) set
            Reduced data availability: 1 pg inactive, 1 pg incomplete
            196 stuck requests are blocked > 4096 sec. Implicated osds 4
 
  services:
    mon: 5 daemons, quorum hp1,2,s4,s5,s6
    mgr: s1(active), standbys: s5
    mds: cephfs-1/1/1 up  {0=s1=up:active}, 1 up:standby
    osd: 23 osds: 23 up, 23 in
         flags noout
 
  data:
    pools:   4 pools, 1312 pgs
    objects: 525.18k objects, 1.63TiB
    usage:   8.18TiB used, 13.9TiB / 22.1TiB avail
    pgs:     0.076% pgs not active
             1310 active+clean
             1    active+clean+scrubbing+deep
             1    incomplete
 
  io:
    client:   8.58KiB/s wr, 0op/s rd, 2op/s wr

I'm going to ignore this for now and focus on fixing the inactive and incomplete pg's.

 

[lifeboy]

Good news. I fixed the "pg incomplete" error in ceph with the help of this post and now that the cluster in healthy, the slow mds message has gone away too!