CEPH problem

solaris249 · Jan 1, 2017

I have ceph cluster with 5 node in proxmox and this is my ceph status:
ceph health detail
HEALTH_WARN 2 pgs incomplete; 2 pgs stuck inactive; 2 pgs stuck unclean; 3 requests are blocked > 32 sec; 1 osds have slow requests
pg 1.ce is stuck inactive since forever, current state incomplete, last acting [7,2,1]
pg 1.1b4 is stuck inactive since forever, current state incomplete, last acting [4,5,7]
pg 1.ce is stuck unclean since forever, current state incomplete, last acting [7,2,1]
pg 1.1b4 is stuck unclean since forever, current state incomplete, last acting [4,5,7]
pg 1.ce is incomplete, acting [7,2,1]
pg 1.1b4 is incomplete, acting [4,5,7]
1 ops are blocked > 16777.2 sec
1 ops are blocked > 1048.58 sec
1 ops are blocked > 262.144 sec
1 ops are blocked > 16777.2 sec on osd.7
1 ops are blocked > 1048.58 sec on osd.7
1 ops are blocked > 262.144 sec on osd.7
1 osds have slow requests

my proxmox vm's doesn't work anymore.
help me please.
how can I fix my ceph cluster?

Ashley · Jan 1, 2017

Looks like you have maybe some OSD's down?

What does Proxmox show on Ceph -> OSD's

solaris249 · Jan 1, 2017

It's my osd tree.I haven't any problem with OSD's.all of OSDs are up and in
may I have some another problems with OSD's?
how can I check OSD's for some another problem?

ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 25.47997 root default
-2 5.45999 host pve16
0 3.64000 osd.0 up 1.00000 1.00000
3 1.81999 osd.3 up 1.00000 1.00000
-3 5.45999 host pve18
5 3.64000 osd.5 up 1.00000 1.00000
1 1.81999 osd.1 up 1.00000 1.00000
-4 5.45999 host pve19
4 3.64000 osd.4 up 1.00000 1.00000
2 1.81999 osd.2 up 1.00000 1.00000
-5 3.64000 host pve17
7 3.64000 osd.7 up 1.00000 1.00000
-6 5.45999 host pve15
9 3.64000 osd.9 up 1.00000 1.00000
8 1.81999 osd.8 up 1.00000 1.00000

spirit · Jan 1, 2017

solaris249 said:
It's my osd tree.I haven't any problem with OSD's.all of OSDs are up and in
may I have some another problems with OSD's?
how can I check OSD's for some another problem?

ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 25.47997 root default
-2 5.45999 host pve16
0 3.64000 osd.0 up 1.00000 1.00000
3 1.81999 osd.3 up 1.00000 1.00000
-3 5.45999 host pve18
5 3.64000 osd.5 up 1.00000 1.00000
1 1.81999 osd.1 up 1.00000 1.00000
-4 5.45999 host pve19
4 3.64000 osd.4 up 1.00000 1.00000
2 1.81999 osd.2 up 1.00000 1.00000
-5 3.64000 host pve17
7 3.64000 osd.7 up 1.00000 1.00000
-6 5.45999 host pve15
9 3.64000 osd.9 up 1.00000 1.00000
8 1.81999 osd.8 up 1.00000 1.00000

isn't osd.6 missing ?

what is your pool replication configuration ? (size?)

udo · Jan 1, 2017

solaris249 said:
It's my osd tree.I haven't any problem with OSD's.all of OSDs are up and in
may I have some another problems with OSD's?
how can I check OSD's for some another problem?

ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 25.47997 root default
-2 5.45999 host pve16
0 3.64000 osd.0 up 1.00000 1.00000
3 1.81999 osd.3 up 1.00000 1.00000
-3 5.45999 host pve18
5 3.64000 osd.5 up 1.00000 1.00000
1 1.81999 osd.1 up 1.00000 1.00000
-4 5.45999 host pve19
4 3.64000 osd.4 up 1.00000 1.00000
2 1.81999 osd.2 up 1.00000 1.00000
-5 3.64000 host pve17
7 3.64000 osd.7 up 1.00000 1.00000
-6 5.45999 host pve15
9 3.64000 osd.9 up 1.00000 1.00000
8 1.81999 osd.8 up 1.00000 1.00000

Hi,
is there enough free space on the OSDs - esp. osd.7 ?

Udo

solaris249 · Jan 1, 2017

I removed osd6 yesterday.and recovery finished.replication size of pool is 3

solaris249 · Jan 1, 2017

udo said:
Hi,
is there enough free space on the OSDs - esp. osd.7 ?

Udo

yes,it's 34.04% used

udo · Jan 1, 2017

solaris249 said:
yes,it's 34.04% used

Hi,
have you tried to restart osd.7?

Do you use jumbo-frames?

What is the output of

Code:

ceph -s

ceph -w | head -40

Udo

solaris249 · Jan 2, 2017

udo said:
Hi,
have you tried to restart osd.7?

Do you use jumbo-frames?

What is the output of

Code:

ceph -s ceph -w | head -40

Udo

ceph -s
cluster e778e423-b7db-4c9a-868c-2fafc8cae285
health HEALTH_WARN
2 pgs incomplete
2 pgs stuck inactive
2 pgs stuck unclean
14 requests are blocked > 32 sec
monmap e7: 5 mons at {0=192.168.200.216:6789/0,1=192.168.200.219:6789/0,2=192.168.200.218:6789/0,3=192.168.200.217:6789/0,4=192.168.200.215:6789/0}
election epoch 890, quorum 0,1,2,3,4 4,0,3,2,1
osdmap e12230: 9 osds: 9 up, 9 in
pgmap v3942635: 512 pgs, 1 pools, 2825 GB data, 714 kobjects
8469 GB used, 17599 GB / 26069 GB avail
509 active+clean
2 incomplete
1 active+clean+scrubbing
client io 5391 B/s wr, 1 op/s
=============================================
ceph -w | head -40
cluster e778e423-b7db-4c9a-868c-2fafc8cae285
health HEALTH_WARN
2 pgs incomplete
2 pgs stuck inactive
2 pgs stuck unclean
14 requests are blocked > 32 sec
monmap e7: 5 mons at {0=192.168.200.216:6789/0,1=192.168.200.219:6789/0,2=192.168.200.218:6789/0,3=192.168.200.217:6789/0,4=192.168.200.215:6789/0}
election epoch 890, quorum 0,1,2,3,4 4,0,3,2,1
osdmap e12230: 9 osds: 9 up, 9 in
pgmap v3942643: 512 pgs, 1 pools, 2825 GB data, 714 kobjects
8469 GB used, 17599 GB / 26069 GB avail
510 active+clean
2 incomplete
client io 17404 B/s wr, 3 op/s

2017-01-02 08:25:15.885199 mon.0 [INF] pgmap v3942643: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 17404 B/s wr, 3 op/s
================================================
how can I restart osd.7 ?

solaris249 · Jan 2, 2017

It's my new "ceph -w |head -40"
ceph -w | head -40
cluster e778e423-b7db-4c9a-868c-2fafc8cae285
health HEALTH_WARN
2 pgs incomplete
2 pgs stuck inactive
2 pgs stuck unclean
14 requests are blocked > 32 sec
monmap e7: 5 mons at {0=192.168.200.216:6789/0,1=192.168.200.219:6789/0,2=192.168.200.218:6789/0,3=192.168.200.217:6789/0,4=192.168.200.215:6789/0}
election epoch 890, quorum 0,1,2,3,4 4,0,3,2,1
osdmap e12230: 9 osds: 9 up, 9 in
pgmap v3942643: 512 pgs, 1 pools, 2825 GB data, 714 kobjects
8469 GB used, 17599 GB / 26069 GB avail
510 active+clean
2 incomplete
client io 17404 B/s wr, 3 op/s

2017-01-02 08:25:15.885199 mon.0 [INF] pgmap v3942643: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 17404 B/s wr, 3 op/s
2017-01-02 08:25:36.391086 mon.0 [INF] pgmap v3942644: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 2178 B/s wr, 0 op/s
2017-01-02 08:26:08.014058 mon.0 [INF] pgmap v3942645: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 129 B/s wr, 0 op/s
2017-01-02 08:26:34.945883 mon.0 [INF] pgmap v3942646: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 152 B/s wr, 0 op/s
2017-01-02 08:26:37.554544 mon.0 [INF] pgmap v3942647: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 415 B/s wr, 0 op/s
2017-01-02 08:26:42.218942 mon.0 [INF] pgmap v3942648: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 1689 B/s wr, 0 op/s
2017-01-02 08:26:43.238113 mon.0 [INF] pgmap v3942649: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 721 B/s wr, 0 op/s
2017-01-02 08:26:47.465053 mon.0 [INF] pgmap v3942650: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:26:41.223676 osd.1 [INF] 1.145 deep-scrub starts
2017-01-02 08:26:52.465582 mon.0 [INF] pgmap v3942651: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:26:57.801876 mon.0 [INF] pgmap v3942652: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:27:03.259419 mon.0 [INF] pgmap v3942653: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:27:08.026275 mon.0 [INF] pgmap v3942654: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 400 B/s wr, 0 op/s
2017-01-02 08:27:09.045531 mon.0 [INF] pgmap v3942655: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 708 B/s wr, 0 op/s
2017-01-02 08:27:11.837116 mon.0 [INF] pgmap v3942656: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:27:12.846647 mon.0 [INF] pgmap v3942657: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:27:14.875809 mon.0 [INF] pgmap v3942658: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:27:15.884957 mon.0 [INF] pgmap v3942659: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:27:19.906112 mon.0 [INF] pgmap v3942660: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:27:24.907778 mon.0 [INF] pgmap v3942661: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:27:29.906498 mon.0 [INF] pgmap v3942662: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:27:34.957006 mon.0 [INF] pgmap v3942663: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 407 B/s wr, 0 op/s
2017-01-02 08:27:35.975677 mon.0 [INF] pgmap v3942664: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 1351 B/s wr, 0 op/s
2017-01-02 08:27:36.984991 mon.0 [INF] pgmap v3942665: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 2018 B/s wr, 0 op/s
2017-01-02 08:27:38.012789 mon.0 [INF] pgmap v3942666: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 4021 B/s wr, 0 op/s
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe

solaris249 · Jan 2, 2017

osd.7 restarted with this command:systemctl restart -l ceph-osd.7.1483256510.301600030.service
It's my new status:

ceph -s
cluster e778e423-b7db-4c9a-868c-2fafc8cae285
health HEALTH_WARN
2 pgs incomplete
2 pgs stuck inactive
2 pgs stuck unclean
3 requests are blocked > 32 sec
monmap e7: 5 mons at {0=192.168.200.216:6789/0,1=192.168.200.219:6789/0,2=192.168.200.218:6789/0,3=192.168.200.217:6789/0,4=192.168.200.215:6789/0}
election epoch 890, quorum 0,1,2,3,4 4,0,3,2,1
osdmap e12234: 9 osds: 9 up, 9 in
pgmap v3942849: 512 pgs, 1 pools, 2825 GB data, 714 kobjects
8469 GB used, 17599 GB / 26069 GB avail
510 active+clean
2 incomplete
client io 888 B/s wr, 0 op/s
==============================================================
ceph -w |head -40
cluster e778e423-b7db-4c9a-868c-2fafc8cae285
health HEALTH_WARN
2 pgs incomplete
2 pgs stuck inactive
2 pgs stuck unclean
3 requests are blocked > 32 sec
monmap e7: 5 mons at {0=192.168.200.216:6789/0,1=192.168.200.219:6789/0,2=192.168.200.218:6789/0,3=192.168.200.217:6789/0,4=192.168.200.215:6789/0}
election epoch 890, quorum 0,1,2,3,4 4,0,3,2,1
osdmap e12234: 9 osds: 9 up, 9 in
pgmap v3942855: 512 pgs, 1 pools, 2825 GB data, 714 kobjects
8469 GB used, 17599 GB / 26069 GB avail
510 active+clean
2 incomplete
client io 1471 B/s wr, 0 op/s

2017-01-02 08:46:44.183838 mon.0 [INF] pgmap v3942855: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 1471 B/s wr, 0 op/s
2017-01-02 08:46:58.619612 osd.9 [INF] 1.1b8 scrub starts
2017-01-02 08:47:03.747503 mon.0 [INF] pgmap v3942856: 512 pgs: 509 active+clean, 1 active+clean+scrubbing, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:47:08.624507 mon.0 [INF] pgmap v3942857: 512 pgs: 509 active+clean, 1 active+clean+scrubbing, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 167 B/s wr, 0 op/s
2017-01-02 08:47:09.633434 mon.0 [INF] pgmap v3942858: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 695 B/s wr, 0 op/s
2017-01-02 08:47:06.629222 osd.9 [INF] 1.1b8 scrub ok
2017-01-02 08:47:09.028697 osd.8 [INF] 1.64 scrub starts
2017-01-02 08:47:14.184252 mon.0 [INF] pgmap v3942859: 512 pgs: 509 active+clean, 1 active+clean+scrubbing, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:47:19.090304 mon.0 [INF] pgmap v3942860: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:47:14.947353 osd.8 [INF] 1.64 scrub ok
2017-01-02 08:47:35.103376 mon.0 [INF] pgmap v3942861: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 195 B/s wr, 0 op/s
2017-01-02 08:47:38.628308 mon.0 [INF] pgmap v3942862: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 419 B/s wr, 0 op/s
2017-01-02 08:47:39.637842 mon.0 [INF] pgmap v3942863: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 2709 B/s wr, 1 op/s
2017-01-02 08:47:44.205501 mon.0 [INF] pgmap v3942864: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 1468 B/s wr, 0 op/s
2017-01-02 08:47:43.563875 osd.7 [WRN] 1 slow requests, 1 included below; oldest blocked for > 240.280506 secs
2017-01-02 08:47:43.563883 osd.7 [WRN] slow request 240.280506 seconds old, received at 2017-01-02 08:43:43.283326: osd_op(client.5324163.0:29283 rbd_data.c79d7238e1f29.000000000001c29b [set-alloc-hint object_size 4194304 write_size 419
4304,write 0~4194304] 1.9e7462ce RETRY=3 ack+ondisk+retry+write+known_if_redirected e12233) currently reached_pg
2017-01-02 08:48:08.737794 mon.0 [INF] pgmap v3942865: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 166 B/s wr, 0 op/s
2017-01-02 08:48:24.136285 mon.0 [INF] pgmap v3942866: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 102 B/s wr, 0 op/s
2017-01-02 08:48:35.122317 mon.0 [INF] pgmap v3942867: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 310 B/s wr, 0 op/s
2017-01-02 08:48:38.950515 mon.0 [INF] pgmap v3942868: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 829 B/s wr, 0 op/s
2017-01-02 08:48:39.959111 mon.0 [INF] pgmap v3942869: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 1693 B/s wr, 0 op/s
2017-01-02 08:48:41.001937 mon.0 [INF] pgmap v3942870: 512 pgs: 509 active+clean, 1 active+clean+scrubbing, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 2017 B/s wr, 0 op/s
2017-01-02 08:48:43.639561 mon.0 [INF] pgmap v3942871: 512 pgs: 509 active+clean, 1 active+clean+scrubbing, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 1112 B/s wr, 0 op/s
2017-01-02 08:48:39.056643 osd.4 [INF] 1.f4 scrub starts
2017-01-02 08:48:44.263500 osd.4 [INF] 1.f4 scrub ok
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe

udo · Jan 2, 2017

Hi,
IO-Error sounds not so good.
Any SMART-info which show an problem on a OSD?

The question again - do you use Jumbo-frames?

Are you sure, that the cluster was healthy after removing osd.6?

Udo

solaris249 · Jan 2, 2017

udo said:
Hi,
IO-Error sounds not so good.
Any SMART-info which show an problem on a OSD?

The question again - do you use Jumbo-frames?

Are you sure, that the cluster was healthy after removing osd.6?

Udo

what's jumbo frame?
I haven't used jumbo frame?how can I know that jubo frame is using or not?
I added osd6 again

udo · Jan 2, 2017

solaris249 said:
what's jumbo frame?
I haven't used jumbo frame?how can I know that jubo frame is using or not?
I added osd6 again

Hi,
Jumbo frames are "bigger" network packets (MTU >1500).
How looks your network config?

Code:

grep -i mtu /etc/network/interfaces

or post the content of /etc/network/interfaces (perhaps anonymous your IPs)

Udo

udo · Jan 2, 2017

solaris249 said:
I added osd6 again

Hi,
is osd.6 in and up?

Code:

ceph osd tree

Udo

solaris249 · Jan 2, 2017

No,I didn't use Jumbo Frame
all of my OSD'd is in and up

Search

Search

CEPH problem

solaris249

Renowned Member

Ashley

Member

solaris249

Renowned Member

spirit

Distinguished Member

udo

Distinguished Member

solaris249

Renowned Member

solaris249

Renowned Member

udo

Distinguished Member

solaris249

Renowned Member

solaris249

Renowned Member

solaris249

Renowned Member

udo

Distinguished Member

solaris249

Renowned Member

udo

Distinguished Member

udo

Distinguished Member

solaris249

Renowned Member