[solved] Ceph / problem quorum monitor?

D

Dark26

Renowned Member
Nov 27, 2017
273
25
68
47
Bonjour,

I think a have a problem with my ceph setup .i have a lots of calling monitor election, and the filesystem seems to freeze on a couple of seconds every five minutes.

No heavy load or nothing using a lot of ressources.

Code: 
2019-11-08 13:33:23.089062 mgr.p1 (mgr.3881911) 12235 : cluster [DBG] pgmap v12240: 250 pgs: 250 active+clean; 51 GiB data, 148 GiB used, 206 GiB / 357 GiB avail; 341 B/s rd, 850 KiB/s wr, 32 op/s
2019-11-08 13:33:24.235604 mon.p3 (mon.2) 4667 : cluster [INF] mon.p3 calling monitor election
2019-11-08 13:33:24.792722 mon.p2 (mon.1) 6179 : cluster [INF] mon.p2 calling monitor election
2019-11-08 13:33:25.089630 mgr.p1 (mgr.3881911) 12236 : cluster [DBG] pgmap v12241: 250 pgs: 250 active+clean; 51 GiB data, 148 GiB used, 206 GiB / 357 GiB avail; 341 B/s rd, 850 KiB/s wr, 32 op/s
2019-11-08 13:33:27.091265 mgr.p1 (mgr.3881911) 12237 : cluster [DBG] pgmap v12242: 250 pgs: 250 active+clean; 51 GiB data, 148 GiB used, 206 GiB / 357 GiB avail; 2.0 KiB/s rd, 1.6 MiB/s wr, 58 op/s
2019-11-08 13:33:29.514899 mon.p1 (mon.0) 64111 : cluster [INF] mon.p1 calling monitor election
2019-11-08 13:33:29.541213 mon.p1 (mon.0) 64112 : cluster [INF] mon.p1 is new leader, mons p1,p2,p3 in quorum (ranks 0,1,2)
2019-11-08 13:33:30.257728 mon.p1 (mon.0) 64113 : cluster [DBG] monmap e5: 3 mons at {p1=[v2:10.10.5.91:3300/0,v1:10.10.5.91:6789/0],p2=[v2:10.10.5.92:3300/0,v1:10.10.5.92:6789/0],p3=[v2:10.10.5.93:3300/0,v1:10.10.5.93:6789/0]}
2019-11-08 13:33:30.257863 mon.p1 (mon.0) 64114 : cluster [DBG] fsmap cephfs:1 {0=p3=up:active} 2 up:standby
2019-11-08 13:33:30.257913 mon.p1 (mon.0) 64115 : cluster [DBG] osdmap e877: 3 total, 3 up, 3 in
2019-11-08 13:33:30.258558 mon.p1 (mon.0) 64116 : cluster [DBG] mgrmap e302: p1(active, since 6h), standbys: p3, p2
2019-11-08 13:33:30.258881 mon.p1 (mon.0) 64117 : cluster [WRN] overall HEALTH_WARN mon p1 is low on available space
2019-11-08 13:33:29.091869 mgr.p1 (mgr.3881911) 12238 : cluster [DBG] pgmap v12243: 250 pgs: 250 active+clean; 51 GiB data, 148 GiB used, 206 GiB / 357 GiB avail; 1.7 KiB/s rd, 1.0 MiB/s wr, 40 op/s
2019-11-08 13:33:31.093885 mgr.p1 (mgr.3881911) 12239 : cluster [DBG] pgmap v12244: 250 pgs: 250 active+clean; 51 GiB data, 148 GiB used, 206 GiB / 357 GiB avail; 2.0 KiB/s rd, 1.3 MiB/s wr, 55 op/s
2019-11-08 13:33:33.095023 mgr.p1 (mgr.3881911) 12240 : cluster [DBG] pgmap v12245: 250 pgs: 250 active+clean; 51 GiB data, 148 GiB used, 206 GiB / 357 GiB avail; 2.0
2019-11-08 13:33:41.100025 mgr.p1 (mgr.3881911) 12244 : cluster [DBG] pgmap v12249: 250 pgs: 250 active+clean; 51 GiB data, 148 GiB used, 206 GiB / 357 GiB avail; 341 KiB/s wr, 25 op/s
2019-11-08 13:33:46.554264 mon.p2 (mon.1) 6181 : cluster [INF] mon.p2 calling monitor election
2019-11-08 13:33:47.103201 mgr.p1 (mgr.3881911) 12247 : cluster [DBG] pgmap v12252: 250 pgs: 250 active+clean; 51 GiB data, 148 GiB used, 206 GiB / 357 GiB avail; 341 B/s rd, 755 KiB/s wr, 38 op/s
2019-11-08 13:33:48.985540 mon.p1 (mon.0) 64120 : cluster [INF] mon.p1 is new leader, mons p1,p3 in quorum (ranks 0,2)
2019-11-08 13:33:49.223245 mon.p1 (mon.0) 64121 : cluster [WRN] overall HEALTH_WARN mon p1 is low on available space
2019-11-08 13:33:49.223333 mon.p1 (mon.0) 64122 : cluster [INF] mon.p1 calling monitor election
2019-11-08 13:33:49.229959 mon.p2 (mon.1) 6182 : cluster [INF] mon.p2 calling monitor election
2019-11-08 13:33:49.546869 mon.p1 (mon.0) 64123 : cluster [INF] mon.p1 is new leader, mons p1,p2,p3 in quorum (ranks 0,1,2)
2019-11-08 13:33:50.244630 mon.p1 (mon.0) 64124 : cluster [DBG] monmap e5: 3 mons at {p1=[v2:10.10.5.91:3300/0,v1:10.10.5.91:6789/0],p2=[v2:10.10.5.92:3300/0,v1:10.10.5.92:6789/0],p3=[v2:10.10.5.93:3300/0,v1:10.10.5.93:6789/0]}
2019-11-08 13:33:50.244766 mon.p1 (mon.0) 64125 : cluster [DBG] fsmap cephfs:1 {0=p3=up:active} 2 up:standby
2019-11-08 13:33:50.244820 mon.p1 (mon.0) 64126 : cluster [DBG] osdmap e877: 3 total, 3 up, 3 in
2019-11-08 13:33:50.245614 mon.p1 (mon.0) 64127 : cluster [DBG] mgrmap e302: p1(active, since 6h), standbys: p3, p2
2019-11-08 13:33:50.245981 mon.p1 (mon.0) 64128 : cluster [WRN] overall HEALTH_WARN mon p1 is low on available space
2019-11-08 13:33:49.104170 mgr.p1 (mgr.3881911) 12248 : cluster [DBG] pgmap v12253: 250 pgs: 250 active+clean; 51 GiB data, 148 GiB used, 206 GiB / 357 GiB avail; 341 B/s rd, 588 KiB/s wr, 27 op/s
2019-11-08 13:33:51.105954 mgr.p1 (mgr.3881911) 12249 : cluster [DBG] pgmap v12254: 250 pgs: 250 active+clean; 51 GiB data, 148 GiB used, 206 GiB / 357 GiB avail; 341 KiB/s rd, 797 KiB/s wr, 34 op/s
2019-11-08 13:34:17.120860 mgr.p1 (mgr.3881911) 12262 : cluster [DBG] pgmap v12267: 250 pgs: 250 active+clean; 51 GiB data, 148 GiB used, 206 GiB / 357 GiB avail; 1.7 KiB/s rd, 1.3 MiB/s wr, 55 op/s
2019-11-08 13:34:31.773141 mon.p3 (mon.2) 4676 : cluster [INF] mon.p3 calling monitor election
2019-11-08 13:34:31.936381 mon.p2 (mon.1) 6189 : cluster [INF] mon.p2 calling monitor election
2019-11-08 13:34:37.105416 mon.p2 (mon.1) 6190 : cluster [INF] mon.p2 is new leader, mons p2,p3 in quorum (ranks 1,2)
2019-11-08 13:34:37.126205 mon.p2 (mon.1) 6191 : cluster [DBG] monmap e5: 3 mons at {p1=[v2:10.10.5.91:3300/0,v1:10.10.5.91:6789/0],p2=[v2:10.10.5.92:3300/0,v1:10.10.5.92:6789/0],p3=[v2:10.10.5.93:3300/0,v1:10.10.5.93:6789/0]}
2019-11-08 13:34:37.126325 mon.p2 (mon.1) 6192 : cluster [DBG] fsmap cephfs:1 {0=p3=up:active} 2 up:standby
2019-11-08 13:34:37.126387 mon.p2 (mon.1) 6193 : cluster [DBG] osdmap e877: 3 total, 3 up, 3 in
2019-11-08 13:34:37.127290 mon.p2 (mon.1) 6194 : cluster [DBG] mgrmap e302: p1(active, since 6h), standbys: p3, p2
2019-11-08 13:34:37.127674 mon.p2 (mon.1) 6195 : cluster [WRN] Health check failed: 1/3 mons down, quorum p2,p3 (MON_DOWN)
2019-11-08 13:34:37.127712 mon.p2 (mon.1) 6196 : cluster [INF] Health check cleared: MON_DISK_LOW (was: mon p1 is low on available space)
2019-11-08 13:34:37.154143 mon.p2 (mon.1) 6197 : cluster [WRN] overall HEALTH_WARN 1/3 mons down, quorum p2,p3
2019-11-08 13:34:19.121527 mgr.p1 (mgr.3881911) 12263 : cluster [DBG] pgmap v12268: 250 pgs: 250 active+clean; 51 GiB data, 148 GiB used, 206 GiB / 357 GiB avail; 1.7 KiB/s rd, 796 KiB/s wr, 35 op/s
2019-11-08 13:34:21.123450 mgr.p1 (mgr.3881911) 12264 : cluster [DBG] pgmap v12269: 250 pgs: 250 active+clean; 51 GiB data, 148 GiB used, 206 GiB / 357 GiB avail; 7.0 KiB/s rd, 1.2 MiB/s wr, 52 op/s
2019-11-08 13:34:23.124217 mgr.p1 (mgr.3881911) 12265 : cluster [DBG] pgmap v12270: 250 pgs: 250 active+clean; 51 GiB data, 148 GiB used, 206 GiB / 357 GiB avail; 5.7

2019-11-08 13:34:39.132424 mgr.p1 (mgr.3881911) 12273 : cluster [DBG] pgmap v12278: 250 pgs: 250 active+clean; 51 GiB data, 148 GiB used, 206 GiB / 357 GiB avail; 3.0 KiB/s rd, 788 KiB/s wr, 31 op/s
2019-11-08 13:34:41.134606 mgr.p1 (mgr.3881911) 12274 : cluster [DBG] pgmap v12279: 250 pgs: 250 active+clean; 51 GiB data, 148 GiB used, 206 GiB / 357 GiB avail; 23 KiB/s rd, 1.0 MiB/s wr, 46 op/s
2019-11-08 13:34:43.135194 mgr.p1 (mgr.3881911) 12275 : cluster [DBG] pgmap v12280: 250 pgs: 250 active+clean; 51 GiB data, 148 GiB used, 206 GiB / 357 GiB avail; 23 KiB/s rd, 640 KiB/s wr, 29 op/s
2019-11-08 13:34:45.135768 mgr.p1 (mgr.3881911) 12276 : cluster [DBG] pgmap v12281: 250 pgs: 250 active+clean; 51 GiB data, 148 GiB used, 206 GiB / 357 GiB avail; 23 KiB/s rd, 640 KiB/s wr, 29 op/s
2019-11-08 13:34:47.137321 mgr.p1 (mgr.3881911) 12277 : cluster [DBG] pgmap v12282: 250 pgs: 250 active+clean; 51 GiB data, 148 GiB used, 206 GiB / 357 GiB avail; 23 KiB/s rd, 892 KiB/s wr, 43 op/s
2019-11-08 13:34:53.618750 mon.p1 (mon.0) 64130 : cluster [INF] mon.p1 calling monitor election
2019-11-08 13:34:53.970742 mon.p1 (mon.0) 64131 : cluster [INF] mon.p1 is new leader, mons p1,p2,p3 in quorum (ranks 0,1,2)
2019-11-08 13:34:54.447667 mon.p1 (mon.0) 64132 : cluster [DBG] monmap e5: 3 mons at {p1=[v2:10.10.5.91:3300/0,v1:10.10.5.91:6789/0],p2=[v2:10.10.5.92:3300/0,v1:10.10.5.92:6789/0],p3=[v2:10.10.5.93:3300/0,v1:10.10.5.93:6789/0]}
2019-11-08 13:34:54.447821 mon.p1 (mon.0) 64133 : cluster [DBG] fsmap cephfs:1 {0=p3=up:active} 2 up:standby
2019-11-08 13:34:54.447908 mon.p1 (mon.0) 64134 : cluster [DBG] osdmap e877: 3 total, 3 up, 3 in
2019-11-08 13:34:54.449243 mon.p1 (mon.0) 64135 : cluster [DBG] mgrmap e302: p1(active, since 6h), standbys: p3, p2
2019-11-08 13:34:54.449852 mon.p1 (mon.0) 64136 : cluster [WRN] Health check failed: mon p1 is low on available space (MON_DISK_LOW)
2019-11-08 13:34:54.449910 mon.p1 (mon.0) 64137 : cluster [INF] Health check cleared: MON_DOWN (was: 1/3 mons down, quorum p2,p3)
2019-11-08 13:34:54.921956 mon.p1 (mon.0) 64138 : cluster [WRN] overall HEALTH_WARN mon p1 is low on available space
2019-11-08 13:34:49.137873 mgr.p1 (mgr.3881911) 12278 : cluster [DBG] pgmap v12283: 250 pgs: 250 active+clean; 51 GiB data, 148 GiB used, 206 GiB / 357 GiB avail; 21 KiB/s rd, 506 KiB/s wr, 28 op/s


Any idea how to find the source of the problem. ( network / hard drive ? )

Do you need some logs or info ?

Merci

dark26
 
Last edited:
Dark26 said:
2019-11-08 13:34:54.449852 mon.p1 (mon.0) 64136 : cluster [WRN] Health check failed: mon p1 is low on available space (MON_DISK_LOW)
Click to expand...
Check disk space and health on the MON nodes.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back