Ceph Pacific constant repeating "MDSs are read only"

[jasonsansone]

Testing 7 Beta with Ceph Pacific 16.2.4. The cluster keeps randomly entering "1 MDSs are read only". This issue never occurred previously under Nautilus or Octopus. The cluster is otherwise healthy. The MDS must be restarted to temporarily clear the error until it reappears hours later.

Spoiler: Error Log

2021-07-01T14:39:45.075487-0500 mds.maverick (mds.0) 53 : cluster [ERR] failed to store backtrace on ino 0x100002d3851 object, pool 2, errno -2
2021-07-01T14:39:45.075610-0500 mds.maverick (mds.0) 54 : cluster [WRN] force file system read-only
2021-07-01T14:39:46.419669-0500 mds.maverick (mds.0) 55 : cluster [ERR] failed to store backtrace on ino 0x100003791cc object, pool 2, errno -2
2021-07-01T14:39:46.419695-0500 mds.maverick (mds.0) 56 : cluster [ERR] failed to store backtrace on ino 0x100003791cd object, pool 2, errno -2
2021-07-01T14:39:46.419701-0500 mds.maverick (mds.0) 57 : cluster [ERR] failed to store backtrace on ino 0x100003791ce object, pool 2, errno -2
2021-07-01T14:39:46.419704-0500 mds.maverick (mds.0) 58 : cluster [ERR] failed to store backtrace on ino 0x100003791cf object, pool 2, errno -2
2021-07-01T14:39:46.419709-0500 mds.maverick (mds.0) 59 : cluster [ERR] failed to store backtrace on ino 0x100003791d1 object, pool 2, errno -2
2021-07-01T14:39:52.750724-0500 mon.viper (mon.0) 66964 : cluster [WRN] Health check failed: 1 MDSs are read only (MDS_READ_ONLY)
2021-07-01T14:39:52.805860-0500 mon.viper (mon.0) 66965 : cluster [DBG] mds.? [v2:192.168.2.201:6800/2091993529,v1:192.168.2.201:6801/2091993529] up:active
2021-07-01T14:39:52.805925-0500 mon.viper (mon.0) 66966 : cluster [DBG] fsmap CephFS:1 {0=maverick=up:active} 2 up:standby
2021-07-01T14:40:00.000178-0500 mon.viper (mon.0) 66967 : cluster [WRN] Health detail: HEALTH_WARN 1 MDSs are read only
2021-07-01T14:40:00.000217-0500 mon.viper (mon.0) 66968 : cluster [WRN] [WRN] MDS_READ_ONLY: 1 MDSs are read only
2021-07-01T14:40:00.000242-0500 mon.viper (mon.0) 66969 : cluster [WRN] mds.maverick(mds.0): MDS in read-only mode

Pool 2 is the CephFS_data pool.

I ran a full forward scrub using "ceph tell mds.CephFS:0 scrub start / force recursive repair". It completes without error.

 

[jasonsansone]

It did it again.

2021-07-01T21:52:00.975669-0500 mds.viper (mds.0) 78 : cluster [WRN] force file system read-only
2021-07-01T21:52:09.270904-0500 mon.viper (mon.0) 70522 : cluster [WRN] Health check failed: 1 MDSs are read only (MDS_READ_ONLY)
2021-07-01T21:52:09.332355-0500 mon.viper (mon.0) 70523 : cluster [DBG] mds.? [v2:192.168.2.205:6800/3039898877,v1:192.168.2.205:6801/3039898877] up:active
2021-07-01T21:52:09.332444-0500 mon.viper (mon.0) 70524 : cluster [DBG] fsmap CephFS:1 {0=viper=up:active} 2 up:standby