Recent content by hahosting

as an fyi this is the output when activating the osd root@vms-ceph112:~# journalctl -u ceph-osd@1203.service -n 50 -- Journal begins at Fri 2023-06-30 13:25:22 BST, ends at Sat 2023-11-25 14:19:28 GMT. -- Nov 25 14:18:56 vms-ceph112 ceph-osd[116397]: 20: (RocksDBStore::do_open(std::ostream&...

interestingly i have run an fsck on the command you sent root@vms-ceph112:/var/lib/ceph/osd# ceph-bluestore-tool fsck --path /var/lib/ceph/osd/ceph-1203 fsck success And i get a success

yes: root@vms-ceph112:/var/lib/ceph/osd/ceph-1203# ls block block.wal ceph_fsid fsid keyring ready type whoami root@vms-ceph112:/var/lib/ceph/osd/ceph-1203#

im afraid the new disk shows no valid partions and wont activate. I have put the broken osd back in and it does show albeit wont start. If i try to extract the pg as above i get root@vms-ceph112:/var/lib/ceph/osd# ceph-objectstore-tool --op export --pgid 8.11 --data-path...

The disk has been cloned to a new one and heading back to the DC with it now so will update shortly

Thanks for your detailed reply. Yes all our pools are now 3/2 except this old one. We have set different crush rules for the different size disks for the various different pools we have created. So the 10tb disks are in a 3/2 backup crush rule pool whereas the 2tb disks are in a crush rule...

great stuff yes i found the above commands before drive removal but without the disk mounted i couldnt do anything with it. I will update here when drive returns

ok thats great thanks for your help. I have set noscrub and nodeep-scrub for now I think at this point all i can do is wait for the recovered osd device to return and see if a. it can be activated and if not b. i can extract the pg's i cant even start what looks like all the vm's with disks...

is it worth disabling scrubbing at this point?

if we decide data is lost can i just delete the dead pg's and see what does and doesn't start from a vm point of view and restore what doesn't from backup?

yes i agree thankyou, we are trying to recover the data from the disk currently to a new device. If i can recover the data is it possible to extract the missing pg's and add them back to the cluster or am i just wasting my time?

root@vms-ceph112:/home/richard.admin# ceph pg dump_stuck stale PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 8.1e0 stale+active+undersized+degraded [1203] 1203 [1203] 1203 8.11 stale+active+undersized+degraded [1203] 1203...

yes the query command just sits forever. I will try it again to be sure

yes 1203 being the issue. we have removed the disk and its reading from a recovery point of view but very slowly so im assuming at this point its a bad disk and being on an old 2/1 has ruined my day. I am trying to get the drive recovered to a new one and perhaps it might activate?

root@vms-ceph112:/home/richard.admin# ceph osd dump | grep ratio full_ratio 0.95 backfillfull_ratio 0.9 nearfull_ratio 0.85 root@vms-ceph112:/home/richard.admin#