Proxmox Ceph Cluster "path of data/storage"

[AngryAdm]

Hi, I have a question about the path of the storage.

If I create 3 nodes with CEPH. Node A,B and C.
I have a VM running on node A and I have 3 replicas, and host as failure domain. Thusly A's vdisks will be placed on node A, B and C concurrently

Will the VM grab data from storage that is local to node A or will it go over the public network and pick some data from everywhere?

4 nodes.
Now 3 replicas data is no longer assuredly placed on 3 nodes but can be distrubuted along all the 4 nodes.
Will Proxmox+CEPH combo ensure that the VM's data has a full replica on node A on which it runs?

 

[Stefan_R]

No it won't, but this is also not how ceph works in general. To ensure consistency, every write needs to be confirmed by the qurom in the cluster anyway, so if the data is local or not should not be of concern. Ceph will auto-balance to the best of it's abilities, but as an administrator, thinking of data locality is the wrong approach to provisioning Ceph.

 

[AngryAdm]

My question did not concern writes, it is about fetching data. which is fastest if done from local storage, obviously

 

[Stefan_R]

Yes, and as I said, that's not the right way to think about Ceph deployment. If you absolutely require local data access for performance (which, given how fast Ceph can be on 10G+ networks, is rather unlikely), Ceph is the wrong tool for the job. Or put differently: If you're intra-cluster networking is not fast enough for your needs (independant of actual storage location), you will not be happy with Ceph.

And yes, your question does in fact concern writes as well, since Ceph is a shared system. So even reads need some form of quorum, which is dependent on prior writes.

 

[AngryAdm]

Well, we will see how PowerBI treats a 10G network. 4 nodes 16 OSD's each