using ceph with proxmox

[offerlam]

Hi all,

We really like the idea that using ceph with proxmox we now use the entire server for virtualization and cluster. The downside is though that we use ressources on cep that could be used for proxmox and vise versa.

When you go through the ceph documentation they say to use dedicated hardware for ceph which is not what we are doing if we use it with proxmox.

I think they say this more to be on the safe side.

My question is. You guys who are using proxmox with ceph on the same node or nodes. How well does it run? do you have SQL servers running fine? do you servers for the entire infrastructor running this way? are there any thing one should know before planing this?

I know its a open question.. so i was hoping we could treat it as such..

 

[wahmed]

For day to day operation it works fine assuming there are plenty of resources to go around. But when Ceph goes into rebalancing mode thats when those CPU and Memory starts to churn really. In those cases your heavyweight VMs or Ceph or Both will suffer due to CPU/Memory shortage. You can also tune your Ceph to do slow recovery which will mitigate resource shortage.
As you said it is an open question, it really comes down to environment by environment scenario.
We have Proxmox+Ceph scenario. For example, lets say we have 40 Proxmox nodes cluster. Out of those 40, 15 of them Proxmox+Ceph(OSD+MDS). The other 25 purely Proxmox + some Ceph MONs. We occasionally move VMs from pure Proxmox to Proxmox+Ceph in some of the following scenario:
1. Multiple Proxmox node failures requiring move of temporary overflow of VMs to Proxmox+Ceph nodes.
2. VMs taking so little resources that Ceph performance will not suffer.
3. Even distribution of VMs to utilize All available CPU cores.

You also(must) take into consideration how many PGs you are going to have. Each Ceph PG consumes some resources. So higher the PG regardless the size of the cluster, the more resources they will consume. Carefully planned PG quantity should keep thing running smoothly. If you are not using yet, this is a great link for calculating PGs:
http://ceph.com/pgcalc/

 

[offerlam]

Hi Wasim,

Thanks for sharing your experinces..

So the reason SOME of your proxmox nodes also has ceph is to make sure that those VMs that are CPU and RAM hungry can be run on dedicated hardware IE no ceph and i case of failures you can move a vm to a node that is not also handling ceph. Is this correctly understood?

I must say i can't say what the future brings? for now we will use our proxmox for owncloud (googledrive, onedrive, dropbox like solution) and email through zimbra. im not sure if any of these service are consider tuff? our zimbra servers uses one cpu and 4gig of ram and out owncloud is using 1 cpu and 1gig of ram.

I haven't been able to find the abriviation PG stands for? I have found the phcalc site on my own but kind of hard to understand when you don't know what PG stands for

 

[wahmed]

So the reason SOME of your proxmox nodes also has ceph is to make sure that those VMs that are CPU and RAM hungry can be run on dedicated hardware IE no ceph and i case of failures you can move a vm to a node that is not also handling ceph. Is this correctly understood?

You are somewhat correct. We use proxmox nodes all around simply because it is just convenient to see all nodes in cluster from GUI. Other than that, you are correct. In emergency we can move some VM to Proxmox+Ceph if need to be.

Both owncloud and Zimbra are lightweight. So i think you are ok. Although we dont use Zimbra, we do use 2 dozens Zarafa email servers serving several hundreds email accounts. And about 8 owncloud deployment. All running with similar resource allocation. 1 CPU and between 2 to 4 GB Ram.
I must say i can't say what the future brings? for now we will use our proxmox for owncloud (googledrive, onedrive, dropbox like solution) and email through zimbra. im not sure if any of these service are consider tuff? our zimbra servers uses one cpu and 4gig of ram and out owncloud is using 1 cpu and 1gig of ram. No issue at all. I think you will beginning to see performance hits when you use Microsoft servers like Exchange, SQL etc.

I haven't been able to find the abriviation PG stands for? I have found the phcalc site on my own but kind of hard to understand when you don't know what PG stands for

Sorry i should have abbreviated. Since you use Ceph i assumed you would know the terminology. PG stands for Placement Group. This is a good read on PG:
http://ceph.com/docs/master/rados/operations/placement-groups/

In simple terms each Placement Group holds data block which consumed CPU/Ram resources. So higher the number of PG, higher will be resource consumption. Good rule of thumb is start with smaller PG number. Then grow as needed. You can only increase PG number, not decrease.

 

[offerlam]

You are somewhat correct. We use proxmox nodes all around simply because it is just convenient to see all nodes in cluster from GUI. Other than that, you are correct. In emergency we can move some VM to Proxmox+Ceph if need to be.




Sorry i should have abbreviated. Since you use Ceph i assumed you would know the terminology. PG stands for Placement Group. This is a good read on PG:
http://ceph.com/docs/master/rados/operations/placement-groups/

In simple terms each Placement Group holds data block which consumed CPU/Ram resources. So higher the number of PG, higher will be resource consumption. Good rule of thumb is start with smaller PG number. Then grow as needed. You can only increase PG number, not decrease.

Hi Wasim,

Thanks for answering.. and so quick

I haven't rolled out ceph yet... sorry if that wasn't clear.. im thinking of doing it. Why i made this threaed.

do you manage ceph soly from proxmox gui? or is it not sufficient when it comes to larger implementation?

 

[wahmed]

do you manage ceph soly from proxmox gui? or is it not sufficient when it comes to larger implementation?

Ceph management through Proxmox GUI is very limited for the right reason. You can manage OSD, Pool, MON and check status through GUI. But all other advanced tasks such as benhmarking, pg repair, retrieving/injecting Ceph configuration and many many more must be done through CLI. I dont see foresee Proxmox team adding many more features in Proxmox GUI for Ceph. I rather have them concentrate on Proxmox itself than trying to integrate 100% Ceph stuff in Proxmox.
Once you know the Ceph commands it is really no big deal to use CLI. There are few options for full Ceph GUI namely ceph-dashboard, Kraken, Calamri. I think Calamari has much more management option in their GUI for Ceph. I use ceph-dashboard full time for Ceph health monitoring. It is read only GUI to only show Ceph status as following:


To manage Ceph cluster of any sizes small or large, i would suggest to get familiar with Ceph commands.