Partition on ceph corrupt

Orionis said:
if i understand the 40 pg are lost forever.
Click to expand...
Not necessarily. they MIGHT be lost. the file system cannot guarantee that what you've written there is valid. It will still reconstruct, but your guest (filesystem or files) may not be recoverable.

Orionis said:
any idea ?
Click to expand...
you cant get this by file, only by entire disk (rbd.) If I were you, wait till your filesystem becomes healthy and then go vm by vm.
 
my disk again out the smart result

Code: 
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR--   083   064   006    -    189924569
  3 Spin_Up_Time            PO----   096   096   000    -    0
  4 Start_Stop_Count        -O--CK   094   094   020    -    6195
  5 Reallocated_Sector_Ct   PO--CK   100   100   010    -    0
  7 Seek_Error_Rate         POSR--   089   060   045    -    784817936
  9 Power_On_Hours          -O--CK   052   052   000    -    42826 (32 181 0)
 10 Spin_Retry_Count        PO--C-   100   100   097    -    0
 12 Power_Cycle_Count       -O--CK   100   100   020    -    214
183 Runtime_Bad_Block       -O--CK   100   100   000    -    0
184 End-to-End_Error        -O--CK   100   100   099    -    0
187 Reported_Uncorrect      -O--CK   100   100   000    -    0
188 Command_Timeout         -O--CK   100   100   000    -    0 0 0
189 High_Fly_Writes         -O-RCK   100   100   000    -    0
190 Airflow_Temperature_Cel -O---K   074   049   040    -    26 (Min/Max 25/27)
191 G-Sense_Error_Rate      -O--CK   100   100   000    -    0
192 Power-Off_Retract_Count -O--CK   100   100   000    -    1348
193 Load_Cycle_Count        -O--CK   092   092   000    -    16992
194 Temperature_Celsius     -O---K   026   051   000    -    26 (0 11 0 0 0)
195 Hardware_ECC_Recovered  -O-RC-   083   064   000    -    189924569
197 Current_Pending_Sector  -O--C-   100   100   000    -    0
198 Offline_Uncorrectable   ----C-   100   100   000    -    0
 
[ 12.611537] sas: Enter sas_scsi_recover_host busy: 1 failed: 1
[ 12.611547] sas: ata8: end_device-0:1: cmd error handler
[ 12.611658] sas: ata7: end_device-0:0: dev error handler
[ 12.611674] sas: ata8: end_device-0:1: dev error handler
[ 12.611689] sas: ata9: end_device-0:2: dev error handler
[ 12.611693] sas: ata10: end_device-0:3: dev error handler
[ 12.611707] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 1 tries: 1
[ 12.651538] sas: Enter sas_scsi_recover_host busy: 1 failed: 1
[ 12.651545] sas: ata9: end_device-0:2: cmd error handler
[ 12.651646] sas: ata7: end_device-0:0: dev error handler
[ 12.651661] sas: ata8: end_device-0:1: dev error handler
[ 12.651666] sas: ata9: end_device-0:2: dev error handler
[ 12.651674] sas: ata10: end_device-0:3: dev error handler
[ 12.651687] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 1 tries: 1
Click to expand...

maybe nothing, but whats the make, model, and driver version of your hba? you can get this by lspci -vv and lsmod
 
Code: 
Module                  Size  Used by
veth                   32768  0
ebtable_filter         16384  0
ebtables               40960  1 ebtable_filter
ip_set                 53248  0
ip6table_raw           16384  0
iptable_raw            16384  0
ip6table_filter        16384  0
ip6_tables             32768  2 ip6table_filter,ip6table_raw
iptable_filter         16384  0
bpfilter               16384  0
sctp                  389120  2
ip6_udp_tunnel         16384  1 sctp
udp_tunnel             20480  1 sctp
nf_tables             258048  0
bonding               192512  0
tls                   110592  1 bonding
softdog                16384  2
nfnetlink_log          20480  1
nfnetlink              20480  4 nf_tables,ip_set,nfnetlink_log
ipmi_ssif              40960  0
intel_rapl_msr         20480  0
intel_rapl_common      32768  1 intel_rapl_msr
sb_edac                28672  0
x86_pkg_temp_thermal    20480  0
intel_powerclamp       24576  0
coretemp               24576  0
kvm_intel             360448  27
kvm                  1024000  1 kvm_intel
ast                    57344  0
irqbypass              16384  31 kvm
crct10dif_pclmul       16384  1
drm_vram_helper        24576  1 ast
drm_ttm_helper         16384  2 drm_vram_helper,ast
ghash_clmulni_intel    16384  0
ttm                    86016  2 drm_vram_helper,drm_ttm_helper
aesni_intel           376832  0
drm_kms_helper        311296  4 drm_vram_helper,ast
crypto_simd            16384  1 aesni_intel
cryptd                 24576  2 crypto_simd,ghash_clmulni_intel
cec                    61440  1 drm_kms_helper
rc_core                65536  1 cec
rapl                   20480  0
input_leds             16384  0
fb_sys_fops            16384  1 drm_kms_helper
intel_cstate           20480  0
pcspkr                 16384  0
syscopyarea            16384  1 drm_kms_helper
sysfillrect            20480  1 drm_kms_helper
sysimgblt              16384  1 drm_kms_helper
mei_me                 40960  0
mei                   135168  1 mei_me
acpi_ipmi              20480  0
mac_hid                16384  0
ipmi_si                73728  1
ipmi_devintf           20480  0
ipmi_msghandler       122880  4 ipmi_devintf,ipmi_si,acpi_ipmi,ipmi_ssif
vhost_net              32768  3
vhost                  53248  1 vhost_net
vhost_iotlb            16384  1 vhost
tap                    28672  1 vhost_net
ib_iser                40960  0
rdma_cm               122880  1 ib_iser
iw_cm                  49152  1 rdma_cm
ib_cm                 131072  1 rdma_cm
ib_core               389120  4 rdma_cm,iw_cm,ib_iser,ib_cm
iscsi_tcp              24576  0
libiscsi_tcp           32768  1 iscsi_tcp
libiscsi               69632  3 libiscsi_tcp,iscsi_tcp,ib_iser
scsi_transport_iscsi   139264  5 libiscsi_tcp,iscsi_tcp,ib_iser,libiscsi
drm                   614400  6 drm_kms_helper,drm_vram_helper,ast,drm_ttm_helper,ttm
sunrpc                581632  1
ip_tables              32768  2 iptable_filter,iptable_raw
x_tables               53248  7 ebtables,ip6table_filter,ip6table_raw,iptable_filter,ip6_tables,iptable_raw,ip_tables
autofs4                49152  2
zfs                  3809280  18
zunicode              348160  1 zfs
zzstd                 634880  1 zfs
zlua                  163840  1 zfs
zavl                   16384  1 zfs
icp                   303104  1 zfs
zcommon               102400  2 zfs,icp
znvpair               102400  2 zfs,zcommon
spl                   106496  6 zfs,icp,zzstd,znvpair,zcommon,zavl
btrfs                1548288  0
blake2b_generic        20480  0
xor                    24576  1 btrfs
zstd_compress         233472  1 btrfs
raid6_pq              122880  1 btrfs
libcrc32c              16384  3 btrfs,nf_tables,sctp
simplefb               16384  0
hid_generic            16384  0
usbmouse               16384  0
usbkbd                 16384  0
usbhid                 65536  0
uas                    28672  0
hid                   151552  2 usbhid,hid_generic
usb_storage            77824  2 uas
crc32_pclmul           16384  0
ahci                   49152  4
isci                  167936  10
i2c_i801               36864  0
ehci_pci               20480  0
i2c_smbus              20480  1 i2c_i801
libsas                 98304  1 isci
libahci                49152  1 ahci
lpc_ich                28672  0
ehci_hcd               94208  1 ehci_pci
scsi_transport_sas     45056  2 isci,libsas
igb                   266240  0
i2c_algo_bit           16384  2 igb,ast
dca                    16384  1 igb
 
It's not quite clear exactly what you did at the start when this all happened?

Did you physically remove a disk whilst CEPH was online, all it takes is a write to not have been completed when your running 2/1 and you have a corrupt PG, if you did something on one of the disks outside of CEPH being aware. Or caused the disk to suddenly loose access you now have a copy of the PG 1 and 2 which are different but as you only have 2 copies you can't use a 3rd to decide which is the correct. So CEPH won't bring up the PG.

There is some options that you can tell CEPH to just ignore and "FIX" by pretending there is no incorrect PG's however this is permeant and will destroy the data and never be retrievable again. So before that be good to have a clear idea of what lead up to this issue and if you have any options first.
 
Orionis said:
it's possible than this HDD cache was in fault ?
Click to expand...
not likely.

you have a "combo" sas controller, which often leads to strange behaviors. What kinds of drives are you using? if they are sata, it may be to your benefit to disable SAS in bios and set your operating mode to sata ahci. If they are sas, you need to double check your cabling (which may be necessary even if they are sata.) its also may be necessary to move the cable from the gray "SAS" ports to the black or blue (or red sometimes) SATA ports. Its also possible that your SAS bus does a rescan when the driver loads and no action is necessary at all- keep an eye on your dmesg for repeats of the messages I copied.
 
it might. and it might make things worse if you're not careful.

whats the output of

ceph health detail
ceph osd tree

edit figured this is going to come to play regardless of your answers; read https://ceph.io/geen-categorie/ceph-manually-repair-object/
then read it again.

after that, if you still need help, post a list of all your stuck pgs and we'll go from there.
 
Last edited:
Code: 
HEALTH_WARN noout flag(s) set; 1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set; 3 nearfull osd(s); Reduced data availability: 39 pgs inactive, 39 pgs incomplete; Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull; 41 pgs not deep-scrubbed in time; 42 pgs not scrubbed in time; 2 pool(s) nearfull
[WRN] OSDMAP_FLAGS: noout flag(s) set
[WRN] OSD_FLAGS: 1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set
    osd.0 has flags nodown,noout
[WRN] OSD_NEARFULL: 3 nearfull osd(s)
    osd.0 is near full
    osd.8 is near full
    osd.9 is near full
[WRN] PG_AVAILABILITY: Reduced data availability: 39 pgs inactive, 39 pgs incomplete
    pg 3.12 is incomplete, acting [0,8]
    pg 3.2c is incomplete, acting [5,0]
    pg 3.41 is incomplete, acting [6,0]
    pg 3.48 is incomplete, acting [0,5]
    pg 3.57 is incomplete, acting [0,5]
    pg 3.65 is incomplete, acting [4,0]
    pg 3.68 is incomplete, acting [0,5]
    pg 3.70 is incomplete, acting [5,1]
    pg 3.75 is incomplete, acting [1,5]
    pg 3.8e is incomplete, acting [5,0]
    pg 3.92 is incomplete, acting [0,8]
    pg 3.ac is incomplete, acting [5,0]
    pg 3.c1 is incomplete, acting [6,0]
    pg 3.c8 is incomplete, acting [0,5]
    pg 3.d7 is incomplete, acting [0,5]
    pg 3.e5 is incomplete, acting [4,0]
    pg 3.e8 is incomplete, acting [0,5]
    pg 3.f0 is incomplete, acting [5,1]
    pg 3.f5 is incomplete, acting [1,5]
    pg 3.10e is incomplete, acting [5,0]
    pg 3.112 is incomplete, acting [0,8]
    pg 3.12c is incomplete, acting [5,0]
    pg 3.141 is incomplete, acting [6,0]
    pg 3.148 is incomplete, acting [0,5]
    pg 3.157 is incomplete, acting [0,5]
    pg 3.165 is incomplete, acting [4,0]
    pg 3.168 is incomplete, acting [0,5]
    pg 3.170 is incomplete, acting [5,1]
    pg 3.175 is incomplete, acting [1,5]
    pg 3.18e is incomplete, acting [5,0]
    pg 3.192 is incomplete, acting [0,8]
    pg 3.1ac is incomplete, acting [5,0]
    pg 3.1c1 is incomplete, acting [6,0]
    pg 3.1c8 is incomplete, acting [0,5]
    pg 3.1d7 is incomplete, acting [0,5]
    pg 3.1e5 is incomplete, acting [4,0]
    pg 3.1e8 is incomplete, acting [0,5]
    pg 3.1f0 is incomplete, acting [5,1]
    pg 3.1f5 is incomplete, acting [1,5]
[WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull
    pg 3.4a is active+remapped+backfill_toofull, acting [4,1]
    pg 3.9a is active+remapped+backfill_toofull, acting [4,1]
    pg 3.14a is active+remapped+backfill_toofull, acting [4,1]
[WRN] PG_NOT_DEEP_SCRUBBED: 41 pgs not deep-scrubbed in time
    pg 3.1f5 not deep-scrubbed since 2022-11-23T21:14:08.772646+0100
    pg 3.1f0 not deep-scrubbed since 2022-11-20T21:16:11.613272+0100
    pg 3.1e8 not deep-scrubbed since 2022-07-26T23:13:26.879814+0200
    pg 3.1e5 not deep-scrubbed since 2022-11-20T04:38:11.107513+0100
    pg 3.1d7 not deep-scrubbed since 2022-08-02T22:00:30.906588+0200
    pg 3.1c8 not deep-scrubbed since 2022-08-02T08:37:35.502820+0200
    pg 3.1c1 not deep-scrubbed since 2022-07-31T12:50:22.933559+0200
    pg 3.1ac not deep-scrubbed since 2022-07-28T13:36:48.842499+0200
    pg 3.192 not deep-scrubbed since 2022-08-01T18:31:44.084229+0200
    pg 3.18e not deep-scrubbed since 2022-11-22T15:19:51.327080+0100
    pg 3.175 not deep-scrubbed since 2022-11-23T21:14:08.772646+0100
    pg 3.170 not deep-scrubbed since 2022-11-20T21:16:11.613272+0100
    pg 3.168 not deep-scrubbed since 2022-07-26T23:13:26.879814+0200
    pg 3.165 not deep-scrubbed since 2022-11-20T04:38:11.107513+0100
    pg 3.157 not deep-scrubbed since 2022-08-02T22:00:30.906588+0200
    pg 3.14a not deep-scrubbed since 2024-02-02T07:18:20.347020+0100
    pg 3.148 not deep-scrubbed since 2022-08-02T08:37:35.502820+0200
    pg 3.141 not deep-scrubbed since 2022-07-31T12:50:22.933559+0200
    pg 3.12c not deep-scrubbed since 2022-07-28T13:36:48.842499+0200
    pg 3.112 not deep-scrubbed since 2022-08-01T18:31:44.084229+0200
    pg 3.10e not deep-scrubbed since 2022-11-22T15:19:51.327080+0100
    pg 3.75 not deep-scrubbed since 2022-11-23T21:14:08.772646+0100
    pg 3.70 not deep-scrubbed since 2022-11-20T21:16:11.613272+0100
    pg 3.68 not deep-scrubbed since 2022-07-26T23:13:26.879814+0200
    pg 3.65 not deep-scrubbed since 2022-11-20T04:38:11.107513+0100
    pg 3.57 not deep-scrubbed since 2022-08-02T22:00:30.906588+0200
    pg 3.4a not deep-scrubbed since 2024-01-29T04:29:16.257327+0100
    pg 3.48 not deep-scrubbed since 2022-08-02T08:37:35.502820+0200
    pg 3.41 not deep-scrubbed since 2022-07-31T12:50:22.933559+0200
    pg 3.12 not deep-scrubbed since 2022-08-01T18:31:44.084229+0200
    pg 3.2c not deep-scrubbed since 2022-07-28T13:36:48.842499+0200
    pg 3.8e not deep-scrubbed since 2022-11-22T15:19:51.327080+0100
    pg 3.92 not deep-scrubbed since 2022-08-01T18:31:44.084229+0200
    pg 3.ac not deep-scrubbed since 2022-07-28T13:36:48.842499+0200
    pg 3.c1 not deep-scrubbed since 2022-07-31T12:50:22.933559+0200
    pg 3.c8 not deep-scrubbed since 2022-08-02T08:37:35.502820+0200
    pg 3.d7 not deep-scrubbed since 2022-08-02T22:00:30.906588+0200
    pg 3.e5 not deep-scrubbed since 2022-11-20T04:38:11.107513+0100
    pg 3.e8 not deep-scrubbed since 2022-07-26T23:13:26.879814+0200
    pg 3.f0 not deep-scrubbed since 2022-11-20T21:16:11.613272+0100
    pg 3.f5 not deep-scrubbed since 2022-11-23T21:14:08.772646+0100
[WRN] PG_NOT_SCRUBBED: 42 pgs not scrubbed in time
    pg 3.1f5 not scrubbed since 2022-11-26T05:17:33.120909+0100
    pg 3.1f0 not scrubbed since 2022-11-27T00:42:13.122933+0100
    pg 3.1e8 not scrubbed since 2022-07-31T22:23:46.225507+0200
    pg 3.1e5 not scrubbed since 2022-11-26T09:58:17.790782+0100
    pg 3.1d7 not scrubbed since 2022-08-06T04:22:46.075679+0200
    pg 3.1c8 not scrubbed since 2022-08-06T00:19:41.137858+0200
    pg 3.1c1 not scrubbed since 2022-08-06T14:04:48.123228+0200
    pg 3.1ac not scrubbed since 2022-08-01T09:53:35.617819+0200
    pg 3.192 not scrubbed since 2022-08-05T17:27:23.540156+0200
    pg 3.18e not scrubbed since 2022-11-26T12:26:19.057741+0100
    pg 3.175 not scrubbed since 2022-11-26T05:17:33.120909+0100
    pg 3.170 not scrubbed since 2022-11-27T00:42:13.122933+0100
    pg 3.168 not scrubbed since 2022-07-31T22:23:46.225507+0200
    pg 3.165 not scrubbed since 2022-11-26T09:58:17.790782+0100
    pg 3.157 not scrubbed since 2022-08-06T04:22:46.075679+0200
    pg 3.14a not scrubbed since 2024-02-04T11:43:50.144268+0100
    pg 3.148 not scrubbed since 2022-08-06T00:19:41.137858+0200
    pg 3.141 not scrubbed since 2022-08-06T14:04:48.123228+0200
    pg 3.12c not scrubbed since 2022-08-01T09:53:35.617819+0200
    pg 3.112 not scrubbed since 2022-08-05T17:27:23.540156+0200
    pg 3.10e not scrubbed since 2022-11-26T12:26:19.057741+0100
    pg 3.75 not scrubbed since 2022-11-26T05:17:33.120909+0100
    pg 3.70 not scrubbed since 2022-11-27T00:42:13.122933+0100
    pg 3.68 not scrubbed since 2022-07-31T22:23:46.225507+0200
    pg 3.65 not scrubbed since 2022-11-26T09:58:17.790782+0100
    pg 3.57 not scrubbed since 2022-08-06T04:22:46.075679+0200
    pg 3.4a not scrubbed since 2024-02-04T06:50:28.547969+0100
    pg 3.48 not scrubbed since 2022-08-06T00:19:41.137858+0200
    pg 3.41 not scrubbed since 2022-08-06T14:04:48.123228+0200
    pg 3.12 not scrubbed since 2022-08-05T17:27:23.540156+0200
    pg 3.2c not scrubbed since 2022-08-01T09:53:35.617819+0200
    pg 3.8e not scrubbed since 2022-11-26T12:26:19.057741+0100
    pg 3.92 not scrubbed since 2022-08-05T17:27:23.540156+0200
    pg 3.9a not scrubbed since 2024-02-04T07:10:39.891150+0100
    pg 3.ac not scrubbed since 2022-08-01T09:53:35.617819+0200
    pg 3.c1 not scrubbed since 2022-08-06T14:04:48.123228+0200
    pg 3.c8 not scrubbed since 2022-08-06T00:19:41.137858+0200
    pg 3.d7 not scrubbed since 2022-08-06T04:22:46.075679+0200
    pg 3.e5 not scrubbed since 2022-11-26T09:58:17.790782+0100
    pg 3.e8 not scrubbed since 2022-07-31T22:23:46.225507+0200
    pg 3.f0 not scrubbed since 2022-11-27T00:42:13.122933+0100
    pg 3.f5 not scrubbed since 2022-11-26T05:17:33.120909+0100
[WRN] POOL_NEARFULL: 2 pool(s) nearfull
    pool 'device_health_metrics' is nearfull
    pool 'OMV_data' is nearfull






Code: 
ID  CLASS  WEIGHT    TYPE NAME               STATUS  REWEIGHT  PRI-AFF
-1         19.10312  root default
-3          9.09677      host bsg-galactica
 0    hdd   3.63869          osd.0               up   1.00000  1.00000
 1    hdd   2.72899          osd.1               up   1.00000  1.00000
 2    hdd   1.81940          osd.2               up   1.00000  1.00000
 9    hdd   0.90970          osd.9               up   1.00000  1.00000
-5         10.00635      host bsg-pegasus
 3    hdd   0.90970          osd.3               up   1.00000  1.00000
 4    hdd   3.63869          osd.4               up   1.00000  1.00000
 5    hdd   3.63869          osd.5               up   1.00000  1.00000
 6    hdd   0.90970          osd.6               up   1.00000  1.00000
 7    hdd   0.45479          osd.7               up   1.00000  1.00000
 8    hdd   0.45479          osd.8               up   1.00000  1.00000
 
ok
I had a 2TO hdd , the system working to rebalance .

DO you think the incomplete PG was recover after ?
 
Orionis said:
ok
I had a 2TO hdd , the system working to rebalance .

DO you think the incomplete PG was recover after ?
Click to expand...

No, it will allow you to repair and or tell CEPH they are PG's with lost data and to continue.

But I think at this point you have data that was inflight to the disks that has been lost, hence you have PG's in the state they are, one of the core downsides to a size of 2 vs the recommended 3.
 
ok now i have this

ceph health
Code: 
HEALTH_WARN 2 nearfull osd(s); Reduced data availability: 39 pgs inactive, 39 pgs incomplete; 42 pgs not deep-scrubbed in time; 42 pgs not scrubbed in time; 2 pool(s) nearfull
[WRN] OSD_NEARFULL: 2 nearfull osd(s)
    osd.0 is near full
    osd.8 is near full
[WRN] PG_AVAILABILITY: Reduced data availability: 39 pgs inactive, 39 pgs incomplete
    pg 3.12 is incomplete, acting [0,8]
    pg 3.2c is incomplete, acting [5,0]
    pg 3.41 is incomplete, acting [6,0]
    pg 3.48 is incomplete, acting [10,5]
    pg 3.57 is incomplete, acting [0,5]
    pg 3.65 is incomplete, acting [10,5]
    pg 3.68 is incomplete, acting [0,5]
    pg 3.70 is incomplete, acting [5,10]
    pg 3.75 is incomplete, acting [1,5]
    pg 3.8e is incomplete, acting [5,0]
    pg 3.92 is incomplete, acting [0,8]
    pg 3.ac is incomplete, acting [5,0]
    pg 3.c1 is incomplete, acting [6,0]
    pg 3.c8 is incomplete, acting [10,5]
    pg 3.d7 is incomplete, acting [0,5]
    pg 3.e5 is incomplete, acting [10,5]
    pg 3.e8 is incomplete, acting [0,5]
    pg 3.f0 is incomplete, acting [5,10]
    pg 3.f5 is incomplete, acting [1,5]
    pg 3.10e is incomplete, acting [5,0]
    pg 3.112 is incomplete, acting [0,8]
    pg 3.12c is incomplete, acting [5,0]
    pg 3.141 is incomplete, acting [6,0]
    pg 3.148 is incomplete, acting [10,5]
    pg 3.157 is incomplete, acting [0,5]
    pg 3.165 is incomplete, acting [10,5]
    pg 3.168 is incomplete, acting [0,5]
    pg 3.170 is incomplete, acting [5,10]
    pg 3.175 is incomplete, acting [1,5]
    pg 3.18e is incomplete, acting [5,0]
    pg 3.192 is incomplete, acting [0,8]
    pg 3.1ac is incomplete, acting [5,0]
    pg 3.1c1 is incomplete, acting [6,0]
    pg 3.1c8 is incomplete, acting [10,5]
    pg 3.1d7 is incomplete, acting [0,5]
    pg 3.1e5 is incomplete, acting [10,5]
    pg 3.1e8 is incomplete, acting [0,5]
    pg 3.1f0 is incomplete, acting [5,10]
    pg 3.1f5 is incomplete, acting [1,5]
[WRN] PG_NOT_DEEP_SCRUBBED: 42 pgs not deep-scrubbed in time
    pg 3.1f5 not deep-scrubbed since 2022-11-23T21:14:08.772646+0100
    pg 3.1f0 not deep-scrubbed since 2022-11-20T21:16:11.613272+0100
    pg 3.1e8 not deep-scrubbed since 2022-07-26T23:13:26.879814+0200
    pg 3.1e5 not deep-scrubbed since 2022-11-20T04:38:11.107513+0100
    pg 3.1d7 not deep-scrubbed since 2022-08-02T22:00:30.906588+0200
    pg 3.1c8 not deep-scrubbed since 2022-08-02T08:37:35.502820+0200
    pg 3.1c1 not deep-scrubbed since 2022-07-31T12:50:22.933559+0200
    pg 3.1ac not deep-scrubbed since 2022-07-28T13:36:48.842499+0200
    pg 3.192 not deep-scrubbed since 2022-08-01T18:31:44.084229+0200
    pg 3.18e not deep-scrubbed since 2022-11-22T15:19:51.327080+0100
    pg 3.175 not deep-scrubbed since 2022-11-23T21:14:08.772646+0100
    pg 3.170 not deep-scrubbed since 2022-11-20T21:16:11.613272+0100
    pg 3.168 not deep-scrubbed since 2022-07-26T23:13:26.879814+0200
    pg 3.165 not deep-scrubbed since 2022-11-20T04:38:11.107513+0100
    pg 3.157 not deep-scrubbed since 2022-08-02T22:00:30.906588+0200
    pg 3.14a not deep-scrubbed since 2024-02-02T07:18:20.347020+0100
    pg 3.148 not deep-scrubbed since 2022-08-02T08:37:35.502820+0200
    pg 3.141 not deep-scrubbed since 2022-07-31T12:50:22.933559+0200
    pg 3.12c not deep-scrubbed since 2022-07-28T13:36:48.842499+0200
    pg 3.112 not deep-scrubbed since 2022-08-01T18:31:44.084229+0200
    pg 3.10e not deep-scrubbed since 2022-11-22T15:19:51.327080+0100
    pg 3.75 not deep-scrubbed since 2022-11-23T21:14:08.772646+0100
    pg 3.70 not deep-scrubbed since 2022-11-20T21:16:11.613272+0100
    pg 3.68 not deep-scrubbed since 2022-07-26T23:13:26.879814+0200
    pg 3.65 not deep-scrubbed since 2022-11-20T04:38:11.107513+0100
    pg 3.57 not deep-scrubbed since 2022-08-02T22:00:30.906588+0200
    pg 3.4a not deep-scrubbed since 2024-01-29T04:29:16.257327+0100
    pg 3.48 not deep-scrubbed since 2022-08-02T08:37:35.502820+0200
    pg 3.41 not deep-scrubbed since 2022-07-31T12:50:22.933559+0200
    pg 3.12 not deep-scrubbed since 2022-08-01T18:31:44.084229+0200
    pg 3.2c not deep-scrubbed since 2022-07-28T13:36:48.842499+0200
    pg 3.8e not deep-scrubbed since 2022-11-22T15:19:51.327080+0100
    pg 3.92 not deep-scrubbed since 2022-08-01T18:31:44.084229+0200
    pg 3.9a not deep-scrubbed since 2024-02-04T07:10:39.891150+0100
    pg 3.ac not deep-scrubbed since 2022-07-28T13:36:48.842499+0200
    pg 3.c1 not deep-scrubbed since 2022-07-31T12:50:22.933559+0200
    pg 3.c8 not deep-scrubbed since 2022-08-02T08:37:35.502820+0200
    pg 3.d7 not deep-scrubbed since 2022-08-02T22:00:30.906588+0200
    pg 3.e5 not deep-scrubbed since 2022-11-20T04:38:11.107513+0100
    pg 3.e8 not deep-scrubbed since 2022-07-26T23:13:26.879814+0200
    pg 3.f0 not deep-scrubbed since 2022-11-20T21:16:11.613272+0100
    pg 3.f5 not deep-scrubbed since 2022-11-23T21:14:08.772646+0100
[WRN] PG_NOT_SCRUBBED: 42 pgs not scrubbed in time
    pg 3.1f5 not scrubbed since 2022-11-26T05:17:33.120909+0100
    pg 3.1f0 not scrubbed since 2022-11-27T00:42:13.122933+0100
    pg 3.1e8 not scrubbed since 2022-07-31T22:23:46.225507+0200
    pg 3.1e5 not scrubbed since 2022-11-26T09:58:17.790782+0100
    pg 3.1d7 not scrubbed since 2022-08-06T04:22:46.075679+0200
    pg 3.1c8 not scrubbed since 2022-08-06T00:19:41.137858+0200
    pg 3.1c1 not scrubbed since 2022-08-06T14:04:48.123228+0200
    pg 3.1ac not scrubbed since 2022-08-01T09:53:35.617819+0200
    pg 3.192 not scrubbed since 2022-08-05T17:27:23.540156+0200
    pg 3.18e not scrubbed since 2022-11-26T12:26:19.057741+0100
    pg 3.175 not scrubbed since 2022-11-26T05:17:33.120909+0100
    pg 3.170 not scrubbed since 2022-11-27T00:42:13.122933+0100
    pg 3.168 not scrubbed since 2022-07-31T22:23:46.225507+0200
    pg 3.165 not scrubbed since 2022-11-26T09:58:17.790782+0100
    pg 3.157 not scrubbed since 2022-08-06T04:22:46.075679+0200
    pg 3.14a not scrubbed since 2024-02-04T11:43:50.144268+0100
    pg 3.148 not scrubbed since 2022-08-06T00:19:41.137858+0200
    pg 3.141 not scrubbed since 2022-08-06T14:04:48.123228+0200
    pg 3.12c not scrubbed since 2022-08-01T09:53:35.617819+0200
    pg 3.112 not scrubbed since 2022-08-05T17:27:23.540156+0200
    pg 3.10e not scrubbed since 2022-11-26T12:26:19.057741+0100
    pg 3.75 not scrubbed since 2022-11-26T05:17:33.120909+0100
    pg 3.70 not scrubbed since 2022-11-27T00:42:13.122933+0100
    pg 3.68 not scrubbed since 2022-07-31T22:23:46.225507+0200
    pg 3.65 not scrubbed since 2022-11-26T09:58:17.790782+0100
    pg 3.57 not scrubbed since 2022-08-06T04:22:46.075679+0200
    pg 3.4a not scrubbed since 2024-02-04T06:50:28.547969+0100
    pg 3.48 not scrubbed since 2022-08-06T00:19:41.137858+0200
    pg 3.41 not scrubbed since 2022-08-06T14:04:48.123228+0200
    pg 3.12 not scrubbed since 2022-08-05T17:27:23.540156+0200
    pg 3.2c not scrubbed since 2022-08-01T09:53:35.617819+0200
    pg 3.8e not scrubbed since 2022-11-26T12:26:19.057741+0100
    pg 3.92 not scrubbed since 2022-08-05T17:27:23.540156+0200
    pg 3.9a not scrubbed since 2024-02-04T07:10:39.891150+0100
    pg 3.ac not scrubbed since 2022-08-01T09:53:35.617819+0200
    pg 3.c1 not scrubbed since 2022-08-06T14:04:48.123228+0200
    pg 3.c8 not scrubbed since 2022-08-06T00:19:41.137858+0200
    pg 3.d7 not scrubbed since 2022-08-06T04:22:46.075679+0200
    pg 3.e5 not scrubbed since 2022-11-26T09:58:17.790782+0100
    pg 3.e8 not scrubbed since 2022-07-31T22:23:46.225507+0200
    pg 3.f0 not scrubbed since 2022-11-27T00:42:13.122933+0100
    pg 3.f5 not scrubbed since 2022-11-26T05:17:33.120909+0100
[WRN] POOL_NEARFULL: 2 pool(s) nearfull
    pool 'device_health_metrics' is nearfull
    pool 'OMV_data' is nearfull

and ceph tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 20.92252 root default
-3 10.91617 host bsg-galactica
0 hdd 3.63869 osd.0 up 1.00000 1.00000
1 hdd 2.72899 osd.1 up 1.00000 1.00000
2 hdd 1.81940 osd.2 up 1.00000 1.00000
9 hdd 0.90970 osd.9 up 1.00000 1.00000
10 hdd 1.81940 osd.10 up 1.00000 1.00000
-5 10.00635 host bsg-pegasus
3 hdd 0.90970 osd.3 up 1.00000 1.00000
4 hdd 3.63869 osd.4 up 1.00000 1.00000
5 hdd 3.63869 osd.5 up 1.00000 1.00000
6 hdd 0.90970 osd.6 up 1.00000 1.00000
7 hdd 0.45479 osd.7 up 1.00000 1.00000
8 hdd 0.45479 osd.8 up 1.00000 1.00000


Code: 
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP     META     AVAIL     %USE   VAR   PGS  STATUS
 0    hdd  3.63869   1.00000  3.6 TiB  3.2 TiB  3.2 TiB   54 MiB  6.6 GiB   461 GiB  87.62  1.29  230      up
 1    hdd  2.72899   1.00000  2.7 TiB  1.4 TiB  1.4 TiB    2 KiB  2.9 GiB   1.4 TiB  49.76  0.73   91      up
 2    hdd  1.81940   1.00000  1.8 TiB  1.1 TiB  1.1 TiB    2 KiB  3.0 GiB   697 GiB  62.59  0.92   76      up
 9    hdd  0.90970   1.00000  932 GiB  616 GiB  614 GiB   55 MiB  1.7 GiB   316 GiB  66.08  0.97   40      up
10    hdd  1.81940   1.00000  1.8 TiB  862 GiB  860 GiB    1 KiB  1.7 GiB  1001 GiB  46.24  0.68   67      up
 3    hdd  0.90970   1.00000  932 GiB  659 GiB  657 GiB    1 KiB  1.3 GiB   273 GiB  70.70  1.04   43      up
 4    hdd  3.63869   1.00000  3.6 TiB  2.6 TiB  2.6 TiB    4 KiB  4.7 GiB   1.0 TiB  71.43  1.05  169      up
 5    hdd  3.63869   1.00000  3.6 TiB  2.4 TiB  2.4 TiB    2 KiB  4.0 GiB   1.3 TiB  64.82  0.95  185      up
 6    hdd  0.90970   1.00000  932 GiB  740 GiB  738 GiB    1 KiB  1.3 GiB   192 GiB  79.39  1.17   51      up
 7    hdd  0.45479   1.00000  466 GiB  372 GiB  371 GiB    1 KiB  587 MiB    94 GiB  79.85  1.17   23      up
 8    hdd  0.45479   1.00000  466 GiB  430 GiB  429 GiB   58 MiB  671 MiB    36 GiB  92.30  1.36   32      up
                       TOTAL   21 TiB   14 TiB   14 TiB  167 MiB   28 GiB   6.7 TiB  68.03

the osd 10 is the new 2to i add on galactica
but my osd is alway nearfull

do i enter a command like reweigh ? or other to liberated the osd full ?
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back