Ceph Resources

[Rizki Rivai]

We tried ceph backend storage on our cluster lab. So far we like reliability of it.

We are prepared to move into production. We need more information about resource needs on mixing ceph with VM.

Any documentation/guide on limiting CPU, Memory, network for ceph process in order to minimize VM steal cpu time for ceph or vice versa or its better to separate ceph cluster and proxmox?

We intent to have arount 15 nodes in single proxmox cluster.

 

[alexskysilk]

Generally speaking, you dont want to limit ceph from ANY of its required resourced or you'll be asking for tears.

The absolutely safest thing to do is to not comingle your storage and compute nodes, but under normal operations ceph doesnt really tax your system very much except on rebalancing (which should only happen on rare occasions.) As long as you're keeping your ceph interface completely separate from all other interfaces (corosync, internet, backup, ?) you should not need any further modifications.

 

[Rizki Rivai]

if we use 10G network we still to separate physical interface?

 

[Alwin]

As @alexskysilk said, don't limit Ceph on its resources, as this will slow down your Ceph cluster. In general, you can use Ceph on the nodes directly, it greatly depends though on your workload on those nodes. Ceph is very latency sensitive, on a hyper converged setup it is very important to not overcommit your nodes, as this will slow down every aspect of your cluster.

One of the good parts with Ceph and Proxmox is, that you can expand starting with three nodes. So, should you see the cluster performance decreasing along the way, you can always move the Ceph services away from the PVE nodes. As you sure know, good monitoring is a must for such setups.

 

[alexskysilk]

if we use 10G network we still to separate physical interface?

yes.

The reason for this should be made readily apparent by @Alwin's answer; while an interface may have plenty of bandwidth for multiple functions, latency is a whole other matter. A stable cluster should have no less than
1 high speed link for cluster traffic (corosync)
1 high speed link for ceph traffic
1 link for api traffic (VLANned), container traffic, etc. If you will be using nfs for slow storage (eg templates, backups) I strongly advise to separate it to another interface than the proxmox api traffic interface.

Do a little bit of digging on the forum; there are many good discussions about optimal network configuration for a successful production cluster.