Ceph Database Deployment Geo-distribution & Latency Question

[drdenis]

I love Proxmox's ability to make it super easy to set up a three-node HA cluster with Ceph. I like to use it for my VMs and SQL databases that require a lot of IOPS. This way, if one of the nodes goes down, my SQL DB VM can be quickly redeployed to another node.

That way, if one of the nodes goes down, my SQL DB can be quickly redeployed to another node. But what if the datacenter goes down? To avoid such a problem, I thought about clustering three servers not in the same rack, but in geographically different locations. Can I do this without losing too much performance? In other words, can Proxmox and Ceph handle high latency (100ms+) well? How can I optimize?

 

[esi_y]

In other words, can Proxmox and Ceph handle high latency (100ms+) well?

It cannot.

https://pve.proxmox.com/wiki/Cluster_Manager#_cluster_network

How can I optimize?

You cannot.

https://forum.proxmox.com/threads/high-latency-clusters.141098/

 

[drdenis]

It cannot.

https://pve.proxmox.com/wiki/Cluster_Manager#_cluster_network



You cannot.

https://forum.proxmox.com/threads/high-latency-clusters.141098/

Thank you for your reply! What is the right solution for me?

Alternatively, I could set up my Postgres DBs in HA directly with Patroni on different Proxmox HA clusters.

What do you think?

 

[esi_y]

Thank you for your reply! What is the right solution for me?

I am not sure I am qualified to answer THAT, actually.

Alternatively, I could set up my Postgres DBs in HA directly with Patroni on different Proxmox HA clusters.

I would definitely be looking at HA on that level instead, especially with a DB. The other thing that got me surprised was you mentioning high IOPS, Ceph and 3 nodes in one sentence above.

What do you think?

Lots of times, of course in my opinion, PVE is not the best choice for the problem. I would go for k8s for this.

 

[drdenis]

I would definitely be looking at HA on that level instead, especially with a DB. The other thing that got me surprised was you mentioning high IOPS, Ceph and 3 nodes in one sentence above.

What confuses you about it?

Lots of times, of course in my opinion, PVE is not the best choice for the problem. I would go for k8s for this.

How is k8s better for my project?

 

[esi_y]

What confuses you about it?

3 OSDs on NVMe SSDs would give you ... <10k IOPS?

How is k8s better for my project?

For a database it would be *actually* high availability. Did you try to dry run such deployment and compare it (with a "node down" scenario on PVE)?

EDIT: I do not know what you are looking to deploy, but have a look at e.g. https://cloudnative-pg.io/

 

[drdenis]

Thanks, I will take a look. IOPS should be fine if you scale to more OSDs.

 

[esi_y]

Thanks, I will take a look. IOPS should be fine if you scale to more OSDs.

What I tried to say was that CEPH is not deployed to increase IOPS, it's for resilience. If you look at use case like DB, you would be better off with DB native replicas and (even on SATA SSD) be benefiting from up to 10x the IOPS (100x on NVMe) and latencies ~200uSec (or ~50uSec). Benchmark CEPH in your deployment. If it's for the sake of deploying CEPH, that's your call, but I would not run proper production DBs on PVE/CEPH, at all.

Quick search yields e.g.: https://www.reddit.com/r/ceph/comments/ubjshn/databases_on_ceph/

 

[drdenis]

What I tried to say was that CEPH is not deployed to increase IOPS, it's for resilience. If you look at use case like DB, you would be better off with DB native replicas and (even on SATA SSD) be benefiting from up to 10x the IOPS (100x on NVMe) and latencies ~200uSec (or ~50uSec). Benchmark CEPH in your deployment. If it's for the sake of deploying CEPH, that's your call, but I would not run proper production DBs on PVE/CEPH, at all.

Quick search yields e.g.: https://www.reddit.com/r/ceph/comments/ubjshn/databases_on_ceph/

Thank you very much!

 

[Falk R.]

I would not run proper production DBs on PVE/CEPH, at all.

I wouldn't leave that as a generalization.NWellNatOf courseNaturalOf courseOf courseOf courseOf course Of course you can run productive databases on Ceph very well, but the reason for using Ceph is never the database.
I have Streched PVE+Ceph clusters at customers to achieve high availability for all workloads and databases also run very well on them.
If I have a high-end database such as SAP HANA, for example, I always run it natively on a server without virtualization, with NVMe's of course.

You have to consider the size of the database and the general purpose.

 

[esi_y]

I wouldn't leave that as a generalization.

For me generally, I simply would not do that - the only benefit I can think of is "sort of highly available" setup which does not require any special DB expertise at the cost of low IOPS.

Of course you can run productive databases on Ceph very well, but the reason for using Ceph is never the database.

I mean, it would work if it can take the load. I would call such setup sub-optimal.

I have Streched PVE+Ceph clusters at customers to achieve high availability for all workloads and databases also run very well on them.
If I have a high-end database such as SAP HANA, for example, I always run it natively on a server without virtualization, with NVMe's of course.

Now forget these two extreme cases and compare it with e.g. k8s with operator like CNPG linked above. I just feel like the only thing stopping people from running it (properly) is lack of experience / focus on DBs.

 

[Falk R.]

I find K8s somewhat overengineered for a simple SQL DB (probably MS SQL on Windows) and somewhat more difficult to implement with MS SQL.

 

[esi_y]

I find K8s somewhat overengineered for a simple SQL DB (probably MS SQL on Windows) and somewhat more difficult to implement with MS SQL.

Fair enough answer. I just want the OP to understand he is 1) not getting some better performance with CEPH; 2) he might be better off running it locally (albeit in a VM).

Keep in mind the CEPH is not there just for the DB. Yes, I do not deploy MS SQL (I think you would have guessed by now ), but I am in the faction where either I do it "the highway" or it's not critical (small workload SQLite of course is not in need of any of that) or ... I would recommend people serious about this but not wanting to become DB admins to go fully managed route.