Ceph pool tweaking....

Vassilis Kipouros

Active Member
Nov 2, 2016
54
6
28
47
I have managed to succesfully deploy a test cluster over three sites
connected with 100mbit fiber.

All three nodes have 3 osds each, and there is the default pools for cephfs-data and cephfs-metadata.

The performance of this one pool stretched across the 100mbit links is low but acceptable for my test case.

So my question is if its possible create 4 pools,
- one that stretches across all OSD's (and is slow) and
- three pools that will be local to each node utilizing the local OSD's?

How do I do it? Can someone point me to the right direction or give a simple example?

Thank you in advance...
 
Been reading a lot on Ceph documentation but looks confusing.
I was hoping for some Proxmox specific instructions because I've broken
my previous cluster by manually doing ceph commands.
 
This just calls for a disaster to happen!

On recovery (usually this is one reason, why you have Ceph in the first place) the data movement will max out your bandwidth. Further, Ceph has no locality, this means a Ceph client will try to reach any OSD that is primary for a PG, irregardless of locality. The three MONs will lose quorum when IO needs rise.

While having a pool only on one node works, it will not be fast and clients still need to connect to a MON (possibly not the local one). The local IO may also amplify latency or recovery issues for the rest of the cluster. This is better suited for a hardware/software RAID controller.

TL;TR
Don't run a Ceph cluster over low bandwidth/high latency networks. Create separate clusters and use rbd-mirror to replicate data. Or as an alternative use our storage replication (pvesr).
https://pve.proxmox.com/pve-docs/chapter-pvesr.html
 
Thank you for your reply Alwin.

Is it possible to have 3 nodes per site in 3 sites all clustered with proxmox
and create local seperate ceph cluster on each site? managed by proxmox?
 
In regards to your described environment, no. Corosync has the similar latency requirements as Ceph.
 
Without using ceph, my current described environment works without a hitch (proxmox wise)
I can fully manage the vms and cts on each site without a problem.

So the question remains, can a proxmox cluster manage multiple ceph clusters?

Or should I isolate the proxmox clusters per site?
Can we have multiple datacenters on the proxmox gui?
 
Don't run a Ceph cluster over low bandwidth/high latency networks. Create separate clusters and use rbd-mirror to replicate data. Or as an alternative use our storage replication (pvesr).
My earlier comment can be extended to Proxmox VE clusters (corosync) and is not limited to the storage part.

So the question remains, can a proxmox cluster manage multiple ceph clusters?
No.

Or should I isolate the proxmox clusters per site?
Yes.

Can we have multiple datacenters on the proxmox gui?
No.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!