Degraded data redundancy

M

MoreDakka

Active Member
May 2, 2019
58
13
28
45
I'm currently running a 3 node test cluster with a 10g nic for a storage ceph network on each host.
I'm trying different drive types so removing and adding new OSDs.

These are the steps I've used to remove a drive
- set the OSD to out
- wait for the data to re-jig itself
- delete the OSD
- shut down the host
- add new drive (currently running an h700 controller so I have to R0 the drives...until I get an h200 controller to put into IT mode this will have to do)
- boot host
- add new OSD
- Let data re-jig again.

Now it seems to be stuck and I need to know how to get the ceph cluster back to OK.
As you can see from the log it's been suck at 0.797% since yesterday. What am I missing?

2019-05-07 13:45:06.857993 mon.prox01 [WRN] Health check update: 272/33624 objects misplaced (0.809%) (OBJECT_MISPLACED)
2019-05-07 13:45:11.858338 mon.prox01 [WRN] Health check update: 268/33624 objects misplaced (0.797%) (OBJECT_MISPLACED)
2019-05-07 14:00:00.000168 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-07 15:00:00.000165 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-07 16:00:00.000177 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-07 17:00:00.000164 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-07 18:00:00.000278 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-07 19:00:00.000162 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-07 20:00:00.000161 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-07 21:00:00.000148 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-07 22:00:00.000161 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-07 23:00:00.000183 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-08 00:00:00.000187 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-08 01:00:00.000134 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-08 02:00:00.000190 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-08 03:00:00.000167 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-08 04:00:00.000194 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-08 05:00:00.000170 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-08 06:00:00.000142 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-08 07:00:00.000176 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-08 08:00:00.000134 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-08 09:00:00.000168 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized​

Thanks,
Josh.
 
MoreDakka said:
I'm currently running a 3 node test cluster with a 10g nic for a storage ceph network on each host.
I'm trying different drive types so removing and adding new OSDs.

These are the steps I've used to remove a drive
- set the OSD to out
- wait for the data to re-jig itself
- delete the OSD
- shut down the host
- add new drive (currently running an h700 controller so I have to R0 the drives...until I get an h200 controller to put into IT mode this will have to do)
- boot host
- add new OSD
- Let data re-jig again.

Now it seems to be stuck and I need to know how to get the ceph cluster back to OK.
As you can see from the log it's been suck at 0.797% since yesterday. What am I missing?

2019-05-07 13:45:06.857993 mon.prox01 [WRN] Health check update: 272/33624 objects misplaced (0.809%) (OBJECT_MISPLACED)
2019-05-07 13:45:11.858338 mon.prox01 [WRN] Health check update: 268/33624 objects misplaced (0.797%) (OBJECT_MISPLACED)
2019-05-07 14:00:00.000168 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-07 15:00:00.000165 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-07 16:00:00.000177 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-07 17:00:00.000164 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-07 18:00:00.000278 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-07 19:00:00.000162 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-07 20:00:00.000161 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-07 21:00:00.000148 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-07 22:00:00.000161 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-07 23:00:00.000183 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-08 00:00:00.000187 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-08 01:00:00.000134 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-08 02:00:00.000190 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-08 03:00:00.000167 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-08 04:00:00.000194 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-08 05:00:00.000170 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-08 06:00:00.000142 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-08 07:00:00.000176 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-08 08:00:00.000134 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized
2019-05-08 09:00:00.000168 mon.prox01 [WRN] overall HEALTH_WARN 268/33624 objects misplaced (0.797%); Degraded data redundancy: 452/33624 objects degraded (1.344%), 5 pgs degraded, 8 pgs undersized​
Click to expand...

Figure out more details by
Code: 
ceph status
ceph osd status
pveceph lspools
ceph pg dump
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom