Ceph osd server no respond to df

[Allen Chen]

Hi ,

Just found a problem in my ceph environment. Ceph osd server no respond to df
ceph health status is ok, totally 3 osd's failed due to xfs errors.

[root@ceph-ric-sata-02 ~]# ceph -s
cluster 0245417b-c40b-435a-a09e-7cc20de6d6b6
health HEALTH_OK
monmap e3: 3 mons at {}
election epoch 1756, quorum 0,1,2 ceph-ric-fec-01,ceph-ric-fec-02,ceph-ric-sata-01
osdmap e1315: 40 osds: 37 up, 37 in
flags sortbitwise,require_jewel_osds
pgmap v49403086: 1024 pgs, 1 pools, 14849 GB data, 3712 kobjects
29692 GB used, 39205 GB / 68897 GB avail
1021 active+clean
3 active+clean+scrubbing
client io 7182 kB/s wr, 0 op/s rd, 174 op/s wr

[root@ceph-ric-sata-02 ~]# ceph -v
ceph version 10.2.11

[root@ceph-ric-sata-02 ~]# uname -a
Linux 4.4.1-1.el7.elrepo.x86_64 #1 SMP Sun Jan 31 16:49:23 EST 2016 x86_64 x86_64 x86_64 GNU/Linux

[root@ceph-ric-sata-02 ~]# df

It just hang in there when I issue df... ctl+c not working, have to close session.


2 osd's failed on sata-03 and 1 osd failed on sata-02, so many xfs error logs in dmesg
[55967858.365372] XFS (sdc1): xfs_log_force: error -5 returned.
[55967888.364857] XFS (sdc1): xfs_log_force: error -5 returned.
[55967918.364342] XFS (sdc1): xfs_log_force: error -5 returned.
[55967948.363833] XFS (sdc1): xfs_log_force: error -5 returned.
[55967978.363313] XFS (sdc1): xfs_log_force: error -5 returned.
I have got an advise to remove osd's and re-add them.

Please comments why df has no respond.
Thanks so much.

 

[Alwin]

2 osd's failed on sata-03 and 1 osd failed on sata-02, so many xfs error logs in dmesg
[55967858.365372] XFS (sdc1): xfs_log_force: error -5 returned.
[55967888.364857] XFS (sdc1): xfs_log_force: error -5 returned.
[55967918.364342] XFS (sdc1): xfs_log_force: error -5 returned.
[55967948.363833] XFS (sdc1): xfs_log_force: error -5 returned.
[55967978.363313] XFS (sdc1): xfs_log_force: error -5 returned.

That's most likely the reason, as 'df' tries to read the filesystem allocation of all mounted filesystems.

 

[Allen Chen]

That's most likely the reason, as 'df' tries to read the filesystem allocation of all mounted filesystems.

Thanks mate.