Ceph questions

[svendsen]

Hi,

I'm about to look into Ceph storage for my PVE cluster and have some questions I hope you guys can help with
I'm having a 3-node PVE3 "cluster" with local disks only. (I use cluster function just to have single web UI)
I'm looking into upgrading to PVE5 and then maybe start looking into Ceph as part of the reinstallation/migration. I have only dual 1Gb NICs in all servers and I'm (for now) trying to avoid upgrading to 10Gb, IB, etc. as the server load is not that high (at least I assume)

So, questions:

1) How easy can I monitor disk I/O on my existing Proxmox nodes to make final decission about Ceph and 1 vs 2 vs 10Gb NIC/bonds? In the webUI I can only see cpu/load, memory and network graphs.

2) What would be expected disk I/O on a 3-node PVE+ceph cluster? (Assuming low latency as the servers are just next to eachother)

3) Is scale-out better than scale-up with Ceph? (in other words, will a 5-node Ceph cluster have better performance than a 3-node cluster?

4) Will Ceph also use the replicated blocks as part of the readable storage? Fx if you have 3-node cluster and have 1 replica per block, then in theory if Ceph is balanced, you would have 2/3 of your datastore on each node minimizing network usage as Ceph only need to get 1/3 of the blocks from another node, right? Or can't you make these kind of assumptions?

Any answers and comments are welcome!
THANKS!

 

[wolfgang]

Hi

1) How easy can I monitor disk I/O on my existing Proxmox nodes to make final decission about Ceph and 1 vs 2 vs 10Gb NIC/bonds? In the webUI I can only see cpu/load, memory and network graphs.

atop is good for long analyses

2) What would be expected disk I/O on a 3-node PVE+ceph cluster? (Assuming low latency as the servers are just next to eachother)

It dependence on the nic(speed) and the disks.

3) Is scale-out better than scale-up with Ceph? (in other words, will a 5-node Ceph cluster have better performance than a 3-node cluster?

It dependence dependence on the count of OSD, but general yes.

4) Will Ceph also use the replicated blocks as part of the readable storage? Fx if you have 3-node cluster and have 1 replica per block, then in theory if Ceph is balanced, you would have 2/3 of your datastore on each node minimizing network usage as Ceph only need to get 1/3 of the blocks from another node, right? Or can't you make these kind of assumptions?

A Ceph pool should use size 3 which means the placement group exists 3 time, but you will only read from one copy.