cannot see the contents of my CEPH storage
- Thread starter damon1
- Start date
Hmm, what did you do?
Can you post the output of
Code:
ceph status
ceph osd tree
ceph osd pools ls
pvesm statusTo get some initial information about the current state of your cluster
Argh, sorry, had an s to much in the
It seems that you have three pools, two for cephFS and the extra RBD ceph pool.
Anyway, can you redo it and then check the rbd list command you tried before but each time with a single ceph pool as argument:
Are those 4 nodes (prox1 to prox4) all nodes in this Proxmox VE cluster?
And finally, let's try to do that what the Webinterface would do indirectly:
Lots of questions, I know, but this is a bit strange as the Ceph cluster itself is definitively OK and healthy, so something must be wrong elsewhere.
On another note: For such a smallish cluster 4 monitors are overkill, 3 would be enough. 4 Normally do not help much here, as then three need to be always active to be quorate.
ceph osd pool ls command..It seems that you have three pools, two for cephFS and the extra RBD ceph pool.
Anyway, can you redo it and then check the rbd list command you tried before but each time with a single ceph pool as argument:
Code:
rbd list ceph-pool-name1
rbd list ceph-pool-name2
...Are those 4 nodes (prox1 to prox4) all nodes in this Proxmox VE cluster?
And finally, let's try to do that what the Webinterface would do indirectly:
pvesm list MyCephLots of questions, I know, but this is a bit strange as the Ceph cluster itself is definitively OK and healthy, so something must be wrong elsewhere.
On another note: For such a smallish cluster 4 monitors are overkill, 3 would be enough. 4 Normally do not help much here, as then three need to be always active to be quorate.
Last edited:
Hi Damon,
so the cephfs _data and _metadata are both related to, well, CephFS and seem OK.
But, there's a pool missing - while your ceph status talks about three pools, the list command only output two..
Can you remember what things you did on the cluster? Was it just working fine a while ago or did it never wroked at all?
you mean the fourth MDS and fourht manager? I mean, they won't hurt you really, but in such a setup you also won't really need them..
so the cephfs _data and _metadata are both related to, well, CephFS and seem OK.
But, there's a pool missing - while your ceph status talks about three pools, the list command only output two..
Can you remember what things you did on the cluster? Was it just working fine a while ago or did it never wroked at all?
you mean the fourth MDS and fourht manager? I mean, they won't hurt you really, but in such a setup you also won't really need them..
to be honest I am sure it is my fault but i cannot remember what I did.
I am thinking about removing the ceph and rebuilding it with cache teiring as per these instructions.
http://tacoisland.net/2019/01/06/hyperconverged-hybrid-storage-on-the-cheap-with-proxmox-and-ceph/
I have found speed to be very slow on the ceph drive
Read 139.4 MB/s
Write 4.089 MB/s
CrystalDiskMark
Thanks for all the help.
Have a great day.
Damon
I am thinking about removing the ceph and rebuilding it with cache teiring as per these instructions.
http://tacoisland.net/2019/01/06/hyperconverged-hybrid-storage-on-the-cheap-with-proxmox-and-ceph/
I have found speed to be very slow on the ceph drive
Read 139.4 MB/s
Write 4.089 MB/s
CrystalDiskMark
Thanks for all the help.
Have a great day.
Damon
OK - little harder than I thought.
Is there a list of things to do to remove CEPH?
I tried to remove all the monitors, OSD's etc..
pveceph purge
sudo apt-get --purge remove ceph-base
and a few other things i found on the internet but ceph is hanging in there.
currently migrating and then rebooting each node {edit} which did no good.
thanks
damon
Is there a list of things to do to remove CEPH?
I tried to remove all the monitors, OSD's etc..
pveceph purge
sudo apt-get --purge remove ceph-base
and a few other things i found on the internet but ceph is hanging in there.
currently migrating and then rebooting each node {edit} which did no good.
thanks
damon
Last edited:
I really wish I had found this FIRST
I'm not a expert.. but you should:
1 - Stop/Out all OSDs
2 - Remove all OSDs
3 - Remove ALL Mons (except the master)
4 - Remove ALL Managers (except the master)
5 - Execute on each OSD node: pveceph purge
6 - On last node (master mon/mgr): stop all ceph services, and execute: pveceph purge
( Note:
sudo systemctl stop ceph\*.service ceph\*.target
)
If you wanna run CEPH again, you need to remove all conf files on /etc/ceph/ and /var/lib/ceph first...
what do you do if you have deleted all the Monitors and Manages?
I'm not a expert.. but you should:
1 - Stop/Out all OSDs
2 - Remove all OSDs
3 - Remove ALL Mons (except the master)
4 - Remove ALL Managers (except the master)
5 - Execute on each OSD node: pveceph purge
6 - On last node (master mon/mgr): stop all ceph services, and execute: pveceph purge
( Note:
sudo systemctl stop ceph\*.service ceph\*.target
)
If you wanna run CEPH again, you need to remove all conf files on /etc/ceph/ and /var/lib/ceph first...
what do you do if you have deleted all the Monitors and Manages?
I understand were you come from, but this is actually the same.
10.10.10.X/24 tells that the last 8 bit are irrelevant (all from the same network), so while often 10.10.10.0/24 is used, and IMO it is the nicer notation, it is semantically the same as 10.10.10.4/24 or the like.
But yes, I agree, this webinterface component may need some improvements to show this better.