Hi,
I have a 3x node Proxmox cluster running ceph with a mixture of NVMe and SAS hard drives built from some used hardware. I've logged into the dashboard this morning and was greeted with an error in the CEPH dashboard saying 'Possible data damage: 1 pg inconsistent'. I've tried a few things from posts from Google etc with no luck. I have specific rules set so specific pools store data on specific device types i.e., one for NVMe and one for HDD.
I've tried running some commands that may help:
I've got the ceph dashboard enabled so I'll try to add some screenshots from both dashboards:
In comparison, here's the EC pool for NVMe that looks fine:
Thanks,
Lewis
I have a 3x node Proxmox cluster running ceph with a mixture of NVMe and SAS hard drives built from some used hardware. I've logged into the dashboard this morning and was greeted with an error in the CEPH dashboard saying 'Possible data damage: 1 pg inconsistent'. I've tried a few things from posts from Google etc with no luck. I have specific rules set so specific pools store data on specific device types i.e., one for NVMe and one for HDD.
I've tried running some commands that may help:
Code:
ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
-1 43.88187 - 44 TiB 1.8 TiB 1.8 TiB 3.7 MiB 23 GiB 42 TiB 4.04 1.00 - root default
-7 14.62729 - 15 TiB 509 GiB 502 GiB 908 KiB 7.1 GiB 14 TiB 3.40 0.84 - host dev-pve-01
9 hdd 1.09160 1.00000 1.1 TiB 593 MiB 19 MiB 27 KiB 574 MiB 1.1 TiB 0.05 0.01 1 up osd.9
10 hdd 1.09160 1.00000 1.1 TiB 89 MiB 18 MiB 7 KiB 71 MiB 1.1 TiB 0.01 0.00 0 up osd.10
32 hdd 0.54579 1.00000 559 GiB 88 MiB 18 MiB 4 KiB 70 MiB 559 GiB 0.02 0.00 0 up osd.32
33 hdd 0.54579 1.00000 559 GiB 88 MiB 18 MiB 4 KiB 70 MiB 559 GiB 0.02 0.00 0 up osd.33
34 hdd 0.54579 1.00000 559 GiB 166 MiB 38 MiB 4 KiB 129 MiB 559 GiB 0.03 0.01 1 up osd.34
35 hdd 0.54579 1.00000 559 GiB 88 MiB 18 MiB 4 KiB 70 MiB 559 GiB 0.02 0.00 0 up osd.35
36 hdd 0.54579 1.00000 559 GiB 88 MiB 18 MiB 4 KiB 70 MiB 559 GiB 0.02 0.00 0 up osd.36
37 hdd 0.54579 1.00000 559 GiB 88 MiB 18 MiB 4 KiB 70 MiB 559 GiB 0.02 0.00 0 up osd.37
38 hdd 0.54579 1.00000 559 GiB 84 MiB 18 MiB 4 KiB 66 MiB 559 GiB 0.01 0.00 0 up osd.38
39 hdd 0.54579 1.00000 559 GiB 181 GiB 179 GiB 165 KiB 1.9 GiB 378 GiB 32.32 8.00 1 up osd.39
40 hdd 0.54579 1.00000 559 GiB 181 GiB 179 GiB 164 KiB 2.0 GiB 378 GiB 32.34 8.00 1 up osd.40
41 hdd 0.54579 1.00000 559 GiB 84 MiB 18 MiB 4 KiB 66 MiB 559 GiB 0.01 0.00 0 up osd.41
4 nvme 3.49309 1.00000 3.5 TiB 146 GiB 144 GiB 509 KiB 1.9 GiB 3.4 TiB 4.08 1.01 2 up osd.4
5 nvme 3.49309 1.00000 3.5 TiB 74 MiB 18 MiB 4 KiB 56 MiB 3.5 TiB 0.00 0 0 up osd.5
-5 14.62729 - 15 TiB 1015 GiB 1003 GiB 1.7 MiB 12 GiB 14 TiB 6.78 1.68 - host dev-pve-02
8 hdd 1.09160 1.00000 1.1 TiB 265 MiB 38 MiB 11 KiB 227 MiB 1.1 TiB 0.02 0.01 1 up osd.8
11 hdd 1.09160 1.00000 1.1 TiB 200 MiB 18 MiB 10 KiB 182 MiB 1.1 TiB 0.02 0.00 0 up osd.11
12 hdd 0.54579 1.00000 559 GiB 181 GiB 179 GiB 157 KiB 2.0 GiB 378 GiB 32.33 8.00 2 up osd.12
13 hdd 0.54579 1.00000 559 GiB 88 MiB 18 MiB 4 KiB 70 MiB 559 GiB 0.02 0.00 0 up osd.13
14 hdd 0.54579 1.00000 559 GiB 88 MiB 18 MiB 4 KiB 70 MiB 559 GiB 0.02 0.00 0 up osd.14
15 hdd 0.54579 1.00000 559 GiB 181 GiB 179 GiB 163 KiB 2.0 GiB 378 GiB 32.33 8.00 1 up osd.15
16 hdd 0.54579 1.00000 559 GiB 88 MiB 18 MiB 4 KiB 70 MiB 559 GiB 0.02 0.00 0 up osd.16
17 hdd 0.54579 1.00000 559 GiB 88 MiB 18 MiB 4 KiB 70 MiB 559 GiB 0.02 0.00 0 up osd.17
18 hdd 0.54579 1.00000 559 GiB 181 GiB 179 GiB 163 KiB 2.0 GiB 378 GiB 32.33 8.00 1 up osd.18
19 hdd 0.54579 1.00000 559 GiB 88 MiB 18 MiB 4 KiB 70 MiB 559 GiB 0.02 0.00 0 up osd.19
20 hdd 0.54579 1.00000 559 GiB 181 GiB 179 GiB 163 KiB 2.0 GiB 378 GiB 32.34 8.00 1 up osd.20
21 hdd 0.54579 1.00000 559 GiB 88 MiB 18 MiB 4 KiB 70 MiB 559 GiB 0.02 0.00 0 up osd.21
2 nvme 3.49309 1.00000 3.5 TiB 146 GiB 144 GiB 524 KiB 2.0 GiB 3.4 TiB 4.08 1.01 2 up osd.2
3 nvme 3.49309 1.00000 3.5 TiB 145 GiB 144 GiB 513 KiB 1.2 GiB 3.4 TiB 4.06 1.00 1 up osd.3
-3 14.62729 - 15 TiB 292 GiB 288 GiB 1.1 MiB 3.4 GiB 14 TiB 1.95 0.48 - host dev-pve-03
6 hdd 1.09160 1.00000 1.1 TiB 94 MiB 18 MiB 25 KiB 76 MiB 1.1 TiB 0.01 0.00 0 up osd.6
7 hdd 1.09160 1.00000 1.1 TiB 93 MiB 18 MiB 17 KiB 75 MiB 1.1 TiB 0.01 0.00 0 up osd.7
22 hdd 0.54579 1.00000 559 GiB 89 MiB 18 MiB 6 KiB 71 MiB 559 GiB 0.02 0.00 0 up osd.22
23 hdd 0.54579 1.00000 559 GiB 88 MiB 18 MiB 6 KiB 70 MiB 559 GiB 0.02 0.00 0 up osd.23
24 hdd 0.54579 1.00000 559 GiB 88 MiB 18 MiB 6 KiB 71 MiB 559 GiB 0.02 0.00 0 up osd.24
25 hdd 0.54579 1.00000 559 GiB 88 MiB 18 MiB 6 KiB 70 MiB 559 GiB 0.02 0.00 0 up osd.25
26 hdd 0.54579 1.00000 559 GiB 88 MiB 18 MiB 6 KiB 70 MiB 559 GiB 0.02 0.00 0 up osd.26
27 hdd 0.54579 1.00000 559 GiB 88 MiB 18 MiB 6 KiB 70 MiB 559 GiB 0.02 0.00 0 up osd.27
28 hdd 0.54579 1.00000 559 GiB 88 MiB 18 MiB 6 KiB 70 MiB 559 GiB 0.02 0.00 0 up osd.28
29 hdd 0.54579 1.00000 559 GiB 88 MiB 18 MiB 6 KiB 70 MiB 559 GiB 0.02 0.00 0 up osd.29
30 hdd 0.54579 1.00000 559 GiB 88 MiB 18 MiB 6 KiB 70 MiB 559 GiB 0.02 0.00 0 up osd.30
31 hdd 0.54579 1.00000 559 GiB 88 MiB 18 MiB 6 KiB 70 MiB 559 GiB 0.02 0.00 0 up osd.31
0 nvme 3.49309 1.00000 3.5 TiB 145 GiB 144 GiB 514 KiB 1.4 GiB 3.4 TiB 4.07 1.01 3 up osd.0
1 nvme 3.49309 1.00000 3.5 TiB 145 GiB 144 GiB 505 KiB 1.2 GiB 3.4 TiB 4.06 1.00 1 up osd.1
TOTAL 44 TiB 1.8 TiB 1.8 TiB 3.7 MiB 23 GiB 42 TiB 4.04
MIN/MAX VAR: 0/8.00 STDDEV: 11.24I've got the ceph dashboard enabled so I'll try to add some screenshots from both dashboards:
In comparison, here's the EC pool for NVMe that looks fine:
Thanks,
Lewis