ceph not working monitors and managers lost

[GoZippy]

monitor services seem up - but not talking.


I mean maybe they are talking but no MDS or FS there anymore... seems the monitors see same number of tasks... show active... idk.. been deep diving into the ceph boards trying to learn more - seems this has happened a few times to others but no resolution and they end up just scrapping and reinstalling... i would like to save my VM's if at all possible on the CPool1 pool

 

[markusd]

Mon900 has the IP 10.0.90.0
That can`t work i fear...

 

[GoZippy]

Mon900 has the IP 10.0.90.0
That can`t work i fear...

Why? It is within the reserved space... what do you think is wrong with it? Has been working for a year till the whole thing died...

 

[alexskysilk]

This is not a sane config.

1. You're comingling your ceph, proxmox cluster, vm, and internet traffic.
2. you have a spof storage node. if 900 is down your whole ceph cluster will be down. You'd be better served by using 900 as a storage only node serving iscsi or nfs.
3. you have OSDs on a raid volume. even if you're passing the volumes as individualt "raid0" the raid controller is intercepting and reassigning the queue according to its own policy.

Honestly, I'd look to rebuild the whole cluster according to best practices and restore your payload from backup. I know you already know this and trying to avoid it

 

[GoZippy]

trying not to lose the vms I have on the ceph osds

 

[GoZippy]

Got all monitors to start I think... still no quorum.

node 900 was weird - it is the only one that has 2 targets mons... shows ceph-mon@node900.service as well as ceph-mon@stack1.service


so I stop all
then restart with mon900 and whammo - monitor900 finally active...

now I am trying to figure out how stact1.service got on node 900

and I still have the critical warning for disk space monitor node 1 and 7



so looking for way to add more space to the pve root where the ceph data lives... sows 60% available space left so not sure why it is giving me fits...

stack1



node7



Q: any safe way to resize pveroot or best practice with regard here? It is a 80GB SSD for OS and pveroot on all machines... should be plenty of space - no?

I am thinking ceph just got nutty when I did the dist upgrade and it ran low on space on root partition and ceph went into shutdown thinking no space left...

I cleaned the install folders and deleted some log files to recover some space on root but still seems to think there is no space left....

I added
[mon] mon compact on start = true

to /etc/pve/ceph.conf but it does not seem to have saved any space or indicated any compaction on restart...

 

[GoZippy]

This is not a sane config.

1. You're comingling your ceph, proxmox cluster, vm, and internet traffic.

Yes - you're right. I have 6 x 1gb network cards in each of the r210ii servers and can isolate the traffic later.. right not it was all working fine for my limited home use... I have 2x24port unmanaged switches (10/100/1000) and am looking for ideas on cards to install to up the node to node traffic - but will probably just make do with what I have... bond 2 for corosync and proxmox management and bond 2 for ceph only traffic... still looking for better info on how best to isolate those.. right now I just use the one port and ethernet cable for everything - and it is plenty for what I do right now... which is just playing and testing some things. I do want to isolate traffic when I get the ceph stuff worked out and save my VM's on those OSDs

2. you have a spof storage node. if 900 is down your whole ceph cluster will be down. You'd be better served by using 900 as a storage only node serving iscsi or nfs.

Yes - it is insane... node900 is currently the spof - but I have 4 more machines identical to it and can bring them online and grow the cluster as needed - I just done NEED it right now and would rather save some money... it is an r510 with 8 1TB SATA drives.. nothing crazy. I have a ton of dell DL585 just sitting in a stack I can play with too..

I wanted to use ceph for VMs and the HP DL585 machines as NAS but was hopeful that I could install ceph on those too as dedicated NAS.. I am playing and learning with old servers I got my hands on...

Node 900 is alive and working - just not in quorum after that dist upgrade and stuff - like I said - I think ceph freaked out when it downloaded the upgrade packages and storage space went down to basically nothing on node7 and stack1 machines... I had to delete logs and autoremove old packages to get the update to complete... then the ceph pool died on me... guessing ceph started seeing limited space (below 30%) and freaked out... now trying to avoid split-brain and get node900 talking to the cluster again so it can bring those vms it hosts back in and start up again.

3. you have OSDs on a raid volume. even if you're passing the volumes as individualt "raid0" the raid controller is intercepting and reassigning the queue according to its own policy.

On node900? Yes - the r510 server has 8 drives in it... I have them all as raid9 individually. I tested a ton of reads and writes and it seemed to work fine that way so I did not have to go buy another controller to fit in there... seemed to just slow IO a little but otherwise was fine... not sure how ceph and raid0 fight though... read some issues and warnings but no real detail on why... raid0 individual disk is closest thing I could find on that controller to straight pass-thru. Got any recommendations on controllers for it that are not expensive I can replace later or add to the other ones I have sitting idle?

Honestly, I'd look to rebuild the whole cluster according to best practices and restore your payload from backup. I know you already know this and trying to avoid it

yeah... but I need to save the VM'm on those OSDs on that node900 and that were balanced over to it from other proxmox node VM's

 

[alexskysilk]

in that case, I wish you best of luck. Once you recover I suggest you rethink your priorities- based on what you describe I don't think you're well served by a cluster. at all. just put all your VMs on the r510 and call it a win. then build another box with the rest of the drives as a backup device.

 

[GoZippy]

in that case, I wish you best of luck. Once you recover I suggest you rethink your priorities- based on what you describe I don't think you're well served by a cluster. at all. just put all your VMs on the r510 and call it a win. then build another box with the rest of the drives as a backup device.

well that wasn't that helpful tbh... when cluster was running (it still is running - just not ceph pools) I was having no issues at all and it was just churning along like it should.. and I was learning... so that IS worth my time.

Anyhow - if anyone has ideas on how to help me get node 900 back in please let me know...

also on the best practice on resize partitions... need to move some more space over to the pveroot where ceph lives

 

[GoZippy]

So I see monmap is correct
I see fsmap cephfs on node2 with 2 up standby
I see mgmap stack1 active with node2 and 7 stanby
but still show mon.node900 down not in quorum...

Ceph commands still fail... no quorum is shown on proxmox gui but logs show quorum of stack1, node2 and node7 all exist - mon.node 900 down

and I have critical low space on mon node7, mon stack1

you think the io blocking due to critical low space is causing the crux of the issue? Still, not sure why node900 cannot rejoin though...