ceph filesystem stuck in read only

[userone]

Hi,
i'm looking for some help/ideas/advices in order to solve the problem that occurs on my metadata
server after the server reboot.
"Ceph status" warns about my MDS being "read only" but the fileystem and the data seem healthy.
It is still possible to access the content of my cephfs volumes since it's read only but i don't know how
to make my filesystem writable again.

Logs keeps showing the same error when i restart the MDS server :

2022-11-04T11:50:14.506+0100 7fbbf83c2700  1 mds.0.6872 handle_mds_map state change up:reconnect --> up:rejoin
2022-11-04T11:50:14.510+0100 7fbbf83c2700  1 mds.0.6872 rejoin_start
2022-11-04T11:50:14.510+0100 7fbbf83c2700  1 mds.0.6872 rejoin_joint_start
2022-11-04T11:50:14.702+0100 7fbbf83c2700  1 mds.0.6872 rejoin_done
2022-11-04T11:50:15.546+0100 7fbbf83c2700  1 mds.node3-5 Updating MDS map to version 6881 from mon.3
2022-11-04T11:50:15.546+0100 7fbbf83c2700  1 mds.0.6872 handle_mds_map i am now mds.0.6872
2022-11-04T11:50:15.546+0100 7fbbf83c2700  1 mds.0.6872 handle_mds_map state change up:rejoin --> up:active
2022-11-04T11:50:15.546+0100 7fbbf83c2700  1 mds.0.6872 recovery_done -- successful recovery!
2022-11-04T11:50:15.550+0100 7fbbf83c2700  1 mds.0.6872 active_start
2022-11-04T11:50:15.558+0100 7fbbf83c2700  1 mds.0.6872 cluster recovered.
2022-11-04T11:50:18.190+0100 7fbbf5bbd700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request.
2022-11-04T11:50:18.190+0100 7fbbf5bbd700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request.
2022-11-04T11:50:18.554+0100 7fbbf23b6700  1 mds.0.cache.dir(0x1000006cf14) commit error -22 v 1933183
2022-11-04T11:50:18.554+0100 7fbbf23b6700 -1 log_channel(cluster) log [ERR] : failed to commit dir 0x1000006cf14 object, errno -22
2022-11-04T11:50:18.554+0100 7fbbf23b6700 -1 mds.0.6872 unhandled write error (22) Invalid argument, force readonly...
2022-11-04T11:50:18.554+0100 7fbbf23b6700  1 mds.0.cache force file system read-only
2022-11-04T11:50:18.554+0100 7fbbf23b6700  0 log_channel(cluster) log [WRN] : force file system read-only

More info:

  cluster:
    id:     f36b996f-221d-4bcb-834b-19fc20bcad6b
    health: HEALTH_WARN
            1 MDSs are read only
            1 MDSs behind on trimming

  services:
   mon: 5 daemons, quorum node2-4,node2-5,node3-4,node3-5,node1-1 (age 22h)
   mgr: node2-4(active, since 28h), standbys: node2-5, node3-4, node3-5, node1-1
    mds: 3/3 daemons up, 3 standby
    osd: 112 osds: 112 up (since 22h), 112 in (since 2w)

  data:
    volumes: 2/2 healthy
    pools:   12 pools, 529 pgs
    objects: 8.54M objects, 1.9 TiB
    usage:   7.8 TiB used, 38 TiB / 46 TiB avail
    pgs:     491 active+clean
             29  active+clean+snaptrim
             9   active+clean+snaptrim_wait

All MDSs, MONs and OSDs are in version 16.2.9 and running pve 7.2-5

 

[alexskysilk]

you're having networking issues on the ceph private interface. Check the interface network statistics, and make sure you're not sharing it for any other purpose.

 

[André Dierker]

Hello,

I have the same issue after the last updates und a reboot.

Jan 11 12:48:56 mimas systemd[1]: Started Ceph metadata server daemon.
Jan 11 12:48:56 mimas ceph-mds[2607691]: starting mds.mimas at
Jan 11 12:49:26 mimas ceph-mds[2607691]: 2023-01-11T12:49:26.447+0100 7fcdc0f9c700 -1 log_channel(cluster) log [ERR] : failed to commit dir 0x1 object, errno -22
Jan 11 12:49:26 mimas ceph-mds[2607691]: 2023-01-11T12:49:26.447+0100 7fcdc0f9c700 -1 mds.0.83715 unhandled write error (22) Invalid argument, force readonly...
Jan 11 12:49:27 mimas ceph-mds[2607691]: 2023-01-11T12:49:27.159+0100 7fcdc47a3700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request.

➜  ~ ceph fs status
cephISO - 3 clients
=======
RANK  STATE    MDS      ACTIVITY     DNS    INOS   DIRS   CAPS
 0    active  titan  Reqs:    0 /s    50     23     17     12  
      POOL          TYPE     USED  AVAIL
cephISO_metadata  metadata  2540k  9999G
  cephISO_data      data    3365M  9999G
kube - 25 clients
====
RANK  STATE    MDS      ACTIVITY     DNS    INOS   DIRS   CAPS
 0    active  mimas  Reqs:    0 /s  9338   9185   1533   2937  
     POOL        TYPE     USED  AVAIL
kube_metadata  metadata  36.4G  6666G
  kube_data      data    76.2G  6666G
STANDBY MDS
  hyperion  
MDS version: ceph version 17.2.5 (e04241aa9b639588fa6c864845287d2824cb6b55) quincy (stable)

➜  ~ ceph fs dump
e83728
enable_multiple, ever_enabled_multiple: 1,1
default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1
 
Filesystem 'cephISO' (1)
fs_name cephISO
epoch   83622
flags   12 joinable allow_snaps allow_multimds_snaps
created 2021-08-02T09:12:59.456137+0200
modified        2023-01-11T11:12:56.047906+0100
tableserver     0
root    0
session_timeout 60
session_autoclose       300
max_file_size   1099511627776
required_client_features        {}
last_failure    0
last_failure_osd_epoch  136308
compat  compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds 1
in      0
up      {0=160887170}
failed
damaged
stopped
data_pools      [2]
metadata_pool   3
inline_data     disabled
balancer
standby_count_wanted    1
[mds.titan{0:160887170} state up:active seq 35 addr [v2:172.16.0.3:6800/2893881428,v1:172.16.0.3:6801/2893881428] compat {c=[1],r=[1],i=[7ff]}]
 
 
Filesystem 'kube' (2)
fs_name kube
epoch   83727
flags   12 joinable allow_snaps allow_multimds_snaps
created 2021-08-19T09:03:41.362567+0200
modified        2023-01-11T12:50:55.369823+0100
tableserver     0
root    0
session_timeout 60
session_autoclose       300
max_file_size   1099511627776
required_client_features        {}
last_failure    0
last_failure_osd_epoch  136810
compat  compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds 1
in      0
up      {0=160921813}
failed
damaged
stopped
data_pools      [9]
metadata_pool   10
inline_data     disabled
balancer
standby_count_wanted    1
[mds.mimas{0:160921813} state up:active seq 33 addr [v2:172.16.0.2:6800/3424665214,v1:172.16.0.2:6801/3424665214] compat {c=[1],r=[1],i=[7ff]}]
 
 
Standby daemons:
 
[mds.hyperion{-1:160923252} state up:standby seq 1 addr [v2:172.16.0.1:6804/2664048575,v1:172.16.0.1:6805/2664048575] compat {c=[1],r=[1],i=[7ff]}]
dumped fsmap epoch 83728

Proxmox: 7.3-4
Kernel: Linux 5.15.83-1-pve #1 SMP PVE 5.15.83-1 (2022-12-15T00:00Z) x86_64 GNU/Linux

It's a three node cluster, running in the exact same configuration since over a year and a healthy ceph. When I disable the manager on Mimas the error moves to another host with the same error message.
I can assure that there is no network problem and all disks are healthy too.

Regards
André

 

[André Dierker]

Hello,

I was able to fix this with setting mds_dir_max_commit_size to 80.

ceph config set mds mds_dir_max_commit_size 80
ceph fs fail <fs_name>
ceph fs set <fs_name> joinable true

I found help in the ceph issue tracker https://tracker.ceph.com/issues/58082