CEPH problem

S

solaris249

Renowned Member
May 9, 2014
9
0
66
I have ceph cluster with 5 node in proxmox and this is my ceph status:
ceph health detail
HEALTH_WARN 2 pgs incomplete; 2 pgs stuck inactive; 2 pgs stuck unclean; 3 requests are blocked > 32 sec; 1 osds have slow requests
pg 1.ce is stuck inactive since forever, current state incomplete, last acting [7,2,1]
pg 1.1b4 is stuck inactive since forever, current state incomplete, last acting [4,5,7]
pg 1.ce is stuck unclean since forever, current state incomplete, last acting [7,2,1]
pg 1.1b4 is stuck unclean since forever, current state incomplete, last acting [4,5,7]
pg 1.ce is incomplete, acting [7,2,1]
pg 1.1b4 is incomplete, acting [4,5,7]
1 ops are blocked > 16777.2 sec
1 ops are blocked > 1048.58 sec
1 ops are blocked > 262.144 sec
1 ops are blocked > 16777.2 sec on osd.7
1 ops are blocked > 1048.58 sec on osd.7
1 ops are blocked > 262.144 sec on osd.7
1 osds have slow requests





my proxmox vm's doesn't work anymore.
help me please.
how can I fix my ceph cluster?
 
Looks like you have maybe some OSD's down?

What does Proxmox show on Ceph -> OSD's
 
It's my osd tree.I haven't any problem with OSD's.all of OSDs are up and in
may I have some another problems with OSD's?
how can I check OSD's for some another problem?


ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 25.47997 root default
-2 5.45999 host pve16
0 3.64000 osd.0 up 1.00000 1.00000
3 1.81999 osd.3 up 1.00000 1.00000
-3 5.45999 host pve18
5 3.64000 osd.5 up 1.00000 1.00000
1 1.81999 osd.1 up 1.00000 1.00000
-4 5.45999 host pve19
4 3.64000 osd.4 up 1.00000 1.00000
2 1.81999 osd.2 up 1.00000 1.00000
-5 3.64000 host pve17
7 3.64000 osd.7 up 1.00000 1.00000
-6 5.45999 host pve15
9 3.64000 osd.9 up 1.00000 1.00000
8 1.81999 osd.8 up 1.00000 1.00000
 
solaris249 said:
It's my osd tree.I haven't any problem with OSD's.all of OSDs are up and in
may I have some another problems with OSD's?
how can I check OSD's for some another problem?


ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 25.47997 root default
-2 5.45999 host pve16
0 3.64000 osd.0 up 1.00000 1.00000
3 1.81999 osd.3 up 1.00000 1.00000
-3 5.45999 host pve18
5 3.64000 osd.5 up 1.00000 1.00000
1 1.81999 osd.1 up 1.00000 1.00000
-4 5.45999 host pve19
4 3.64000 osd.4 up 1.00000 1.00000
2 1.81999 osd.2 up 1.00000 1.00000
-5 3.64000 host pve17
7 3.64000 osd.7 up 1.00000 1.00000
-6 5.45999 host pve15
9 3.64000 osd.9 up 1.00000 1.00000
8 1.81999 osd.8 up 1.00000 1.00000
Click to expand...


isn't osd.6 missing ?

what is your pool replication configuration ? (size?)
 
solaris249 said:
It's my osd tree.I haven't any problem with OSD's.all of OSDs are up and in
may I have some another problems with OSD's?
how can I check OSD's for some another problem?


ceph osd tree
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
-1 25.47997 root default
-2 5.45999 host pve16
0 3.64000 osd.0 up 1.00000 1.00000
3 1.81999 osd.3 up 1.00000 1.00000
-3 5.45999 host pve18
5 3.64000 osd.5 up 1.00000 1.00000
1 1.81999 osd.1 up 1.00000 1.00000
-4 5.45999 host pve19
4 3.64000 osd.4 up 1.00000 1.00000
2 1.81999 osd.2 up 1.00000 1.00000
-5 3.64000 host pve17
7 3.64000 osd.7 up 1.00000 1.00000
-6 5.45999 host pve15
9 3.64000 osd.9 up 1.00000 1.00000
8 1.81999 osd.8 up 1.00000 1.00000
Click to expand...
Hi,
is there enough free space on the OSDs - esp. osd.7 ?

Udo
 
udo said:
Hi,
have you tried to restart osd.7?

Do you use jumbo-frames?

What is the output of
Code: 
ceph -s

ceph -w | head -40
Udo
Click to expand...



ceph -s
cluster e778e423-b7db-4c9a-868c-2fafc8cae285
health HEALTH_WARN
2 pgs incomplete
2 pgs stuck inactive
2 pgs stuck unclean
14 requests are blocked > 32 sec
monmap e7: 5 mons at {0=192.168.200.216:6789/0,1=192.168.200.219:6789/0,2=192.168.200.218:6789/0,3=192.168.200.217:6789/0,4=192.168.200.215:6789/0}
election epoch 890, quorum 0,1,2,3,4 4,0,3,2,1
osdmap e12230: 9 osds: 9 up, 9 in
pgmap v3942635: 512 pgs, 1 pools, 2825 GB data, 714 kobjects
8469 GB used, 17599 GB / 26069 GB avail
509 active+clean
2 incomplete
1 active+clean+scrubbing
client io 5391 B/s wr, 1 op/s
=============================================
ceph -w | head -40
cluster e778e423-b7db-4c9a-868c-2fafc8cae285
health HEALTH_WARN
2 pgs incomplete
2 pgs stuck inactive
2 pgs stuck unclean
14 requests are blocked > 32 sec
monmap e7: 5 mons at {0=192.168.200.216:6789/0,1=192.168.200.219:6789/0,2=192.168.200.218:6789/0,3=192.168.200.217:6789/0,4=192.168.200.215:6789/0}
election epoch 890, quorum 0,1,2,3,4 4,0,3,2,1
osdmap e12230: 9 osds: 9 up, 9 in
pgmap v3942643: 512 pgs, 1 pools, 2825 GB data, 714 kobjects
8469 GB used, 17599 GB / 26069 GB avail
510 active+clean
2 incomplete
client io 17404 B/s wr, 3 op/s

2017-01-02 08:25:15.885199 mon.0 [INF] pgmap v3942643: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 17404 B/s wr, 3 op/s
================================================
how can I restart osd.7 ?
 
It's my new "ceph -w |head -40"
ceph -w | head -40
cluster e778e423-b7db-4c9a-868c-2fafc8cae285
health HEALTH_WARN
2 pgs incomplete
2 pgs stuck inactive
2 pgs stuck unclean
14 requests are blocked > 32 sec
monmap e7: 5 mons at {0=192.168.200.216:6789/0,1=192.168.200.219:6789/0,2=192.168.200.218:6789/0,3=192.168.200.217:6789/0,4=192.168.200.215:6789/0}
election epoch 890, quorum 0,1,2,3,4 4,0,3,2,1
osdmap e12230: 9 osds: 9 up, 9 in
pgmap v3942643: 512 pgs, 1 pools, 2825 GB data, 714 kobjects
8469 GB used, 17599 GB / 26069 GB avail
510 active+clean
2 incomplete
client io 17404 B/s wr, 3 op/s

2017-01-02 08:25:15.885199 mon.0 [INF] pgmap v3942643: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 17404 B/s wr, 3 op/s
2017-01-02 08:25:36.391086 mon.0 [INF] pgmap v3942644: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 2178 B/s wr, 0 op/s
2017-01-02 08:26:08.014058 mon.0 [INF] pgmap v3942645: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 129 B/s wr, 0 op/s
2017-01-02 08:26:34.945883 mon.0 [INF] pgmap v3942646: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 152 B/s wr, 0 op/s
2017-01-02 08:26:37.554544 mon.0 [INF] pgmap v3942647: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 415 B/s wr, 0 op/s
2017-01-02 08:26:42.218942 mon.0 [INF] pgmap v3942648: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 1689 B/s wr, 0 op/s
2017-01-02 08:26:43.238113 mon.0 [INF] pgmap v3942649: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 721 B/s wr, 0 op/s
2017-01-02 08:26:47.465053 mon.0 [INF] pgmap v3942650: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:26:41.223676 osd.1 [INF] 1.145 deep-scrub starts
2017-01-02 08:26:52.465582 mon.0 [INF] pgmap v3942651: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:26:57.801876 mon.0 [INF] pgmap v3942652: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:27:03.259419 mon.0 [INF] pgmap v3942653: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:27:08.026275 mon.0 [INF] pgmap v3942654: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 400 B/s wr, 0 op/s
2017-01-02 08:27:09.045531 mon.0 [INF] pgmap v3942655: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 708 B/s wr, 0 op/s
2017-01-02 08:27:11.837116 mon.0 [INF] pgmap v3942656: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:27:12.846647 mon.0 [INF] pgmap v3942657: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:27:14.875809 mon.0 [INF] pgmap v3942658: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:27:15.884957 mon.0 [INF] pgmap v3942659: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:27:19.906112 mon.0 [INF] pgmap v3942660: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:27:24.907778 mon.0 [INF] pgmap v3942661: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:27:29.906498 mon.0 [INF] pgmap v3942662: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:27:34.957006 mon.0 [INF] pgmap v3942663: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 407 B/s wr, 0 op/s
2017-01-02 08:27:35.975677 mon.0 [INF] pgmap v3942664: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 1351 B/s wr, 0 op/s
2017-01-02 08:27:36.984991 mon.0 [INF] pgmap v3942665: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 2018 B/s wr, 0 op/s
2017-01-02 08:27:38.012789 mon.0 [INF] pgmap v3942666: 512 pgs: 1 active+clean+scrubbing+deep, 509 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 4021 B/s wr, 0 op/s
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
 
osd.7 restarted with this command:systemctl restart -l ceph-osd.7.1483256510.301600030.service
It's my new status:

ceph -s
cluster e778e423-b7db-4c9a-868c-2fafc8cae285
health HEALTH_WARN
2 pgs incomplete
2 pgs stuck inactive
2 pgs stuck unclean
3 requests are blocked > 32 sec
monmap e7: 5 mons at {0=192.168.200.216:6789/0,1=192.168.200.219:6789/0,2=192.168.200.218:6789/0,3=192.168.200.217:6789/0,4=192.168.200.215:6789/0}
election epoch 890, quorum 0,1,2,3,4 4,0,3,2,1
osdmap e12234: 9 osds: 9 up, 9 in
pgmap v3942849: 512 pgs, 1 pools, 2825 GB data, 714 kobjects
8469 GB used, 17599 GB / 26069 GB avail
510 active+clean
2 incomplete
client io 888 B/s wr, 0 op/s
==============================================================
ceph -w |head -40
cluster e778e423-b7db-4c9a-868c-2fafc8cae285
health HEALTH_WARN
2 pgs incomplete
2 pgs stuck inactive
2 pgs stuck unclean
3 requests are blocked > 32 sec
monmap e7: 5 mons at {0=192.168.200.216:6789/0,1=192.168.200.219:6789/0,2=192.168.200.218:6789/0,3=192.168.200.217:6789/0,4=192.168.200.215:6789/0}
election epoch 890, quorum 0,1,2,3,4 4,0,3,2,1
osdmap e12234: 9 osds: 9 up, 9 in
pgmap v3942855: 512 pgs, 1 pools, 2825 GB data, 714 kobjects
8469 GB used, 17599 GB / 26069 GB avail
510 active+clean
2 incomplete
client io 1471 B/s wr, 0 op/s

2017-01-02 08:46:44.183838 mon.0 [INF] pgmap v3942855: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 1471 B/s wr, 0 op/s
2017-01-02 08:46:58.619612 osd.9 [INF] 1.1b8 scrub starts
2017-01-02 08:47:03.747503 mon.0 [INF] pgmap v3942856: 512 pgs: 509 active+clean, 1 active+clean+scrubbing, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:47:08.624507 mon.0 [INF] pgmap v3942857: 512 pgs: 509 active+clean, 1 active+clean+scrubbing, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 167 B/s wr, 0 op/s
2017-01-02 08:47:09.633434 mon.0 [INF] pgmap v3942858: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 695 B/s wr, 0 op/s
2017-01-02 08:47:06.629222 osd.9 [INF] 1.1b8 scrub ok
2017-01-02 08:47:09.028697 osd.8 [INF] 1.64 scrub starts
2017-01-02 08:47:14.184252 mon.0 [INF] pgmap v3942859: 512 pgs: 509 active+clean, 1 active+clean+scrubbing, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:47:19.090304 mon.0 [INF] pgmap v3942860: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail
2017-01-02 08:47:14.947353 osd.8 [INF] 1.64 scrub ok
2017-01-02 08:47:35.103376 mon.0 [INF] pgmap v3942861: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 195 B/s wr, 0 op/s
2017-01-02 08:47:38.628308 mon.0 [INF] pgmap v3942862: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 419 B/s wr, 0 op/s
2017-01-02 08:47:39.637842 mon.0 [INF] pgmap v3942863: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 2709 B/s wr, 1 op/s
2017-01-02 08:47:44.205501 mon.0 [INF] pgmap v3942864: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 1468 B/s wr, 0 op/s
2017-01-02 08:47:43.563875 osd.7 [WRN] 1 slow requests, 1 included below; oldest blocked for > 240.280506 secs
2017-01-02 08:47:43.563883 osd.7 [WRN] slow request 240.280506 seconds old, received at 2017-01-02 08:43:43.283326: osd_op(client.5324163.0:29283 rbd_data.c79d7238e1f29.000000000001c29b [set-alloc-hint object_size 4194304 write_size 419
4304,write 0~4194304] 1.9e7462ce RETRY=3 ack+ondisk+retry+write+known_if_redirected e12233) currently reached_pg
2017-01-02 08:48:08.737794 mon.0 [INF] pgmap v3942865: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 166 B/s wr, 0 op/s
2017-01-02 08:48:24.136285 mon.0 [INF] pgmap v3942866: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 102 B/s wr, 0 op/s
2017-01-02 08:48:35.122317 mon.0 [INF] pgmap v3942867: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 310 B/s wr, 0 op/s
2017-01-02 08:48:38.950515 mon.0 [INF] pgmap v3942868: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 829 B/s wr, 0 op/s
2017-01-02 08:48:39.959111 mon.0 [INF] pgmap v3942869: 512 pgs: 510 active+clean, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 1693 B/s wr, 0 op/s
2017-01-02 08:48:41.001937 mon.0 [INF] pgmap v3942870: 512 pgs: 509 active+clean, 1 active+clean+scrubbing, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 2017 B/s wr, 0 op/s
2017-01-02 08:48:43.639561 mon.0 [INF] pgmap v3942871: 512 pgs: 509 active+clean, 1 active+clean+scrubbing, 2 incomplete; 2825 GB data, 8469 GB used, 17599 GB / 26069 GB avail; 1112 B/s wr, 0 op/s
2017-01-02 08:48:39.056643 osd.4 [INF] 1.f4 scrub starts
2017-01-02 08:48:44.263500 osd.4 [INF] 1.f4 scrub ok
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "_ctypes/callbacks.c", line 314, in 'calling callback function'
File "/usr/lib/python2.7/dist-packages/rados.py", line 2056, in monitor_log_callback
self.callback(arg, line, who, sec, nsec, seq, level, msg)
File "/usr/bin/ceph", line 734, in watch_cb
sys.stdout.flush()
IOError: [Errno 32] Broken pipe
 
udo said:
Hi,
IO-Error sounds not so good.
Any SMART-info which show an problem on a OSD?

The question again - do you use Jumbo-frames?

Are you sure, that the cluster was healthy after removing osd.6?

Udo
Click to expand...
what's jumbo frame?
I haven't used jumbo frame?how can I know that jubo frame is using or not?
I added osd6 again
 
You must log in or register to reply here.
Top Bottom
Back