Ceph SAS OSD's to SSD Question

[sahostking]

If we have a currently running Ceph Cluster with the following:

7 nodes with the following setup:

Dell R610 Servers
64 GB Memory
1 x 480GB PM863a SSD for Proxmox OS
5 x 600GB Enterprise 10K SAS Disks for OSDs
10Gb Ethernet Network
Dell H200 Card

Lets say these nodes are doing ok running only around 3 or 4 VMs and performance is ok.

Now if we wanted to switch this over later to say Full Enterprise SSDs and upgrade the servers with a difference of :

Dell R610 Servers
64 GB Memory
1 x 480GB PM863a SSD for Proxmox OS
5 x 480GB SM863a SSD for OSDs
10Gb Ethernet Network
Dell H200 Card

Would that be an issue at any point?

Could we just easily switch off one node at a time replace the Disks with the SSD disks and then bring it back up. And re-add the OSDs which would be different?
What is the proper procedure? Any pointers.

Note this is all currently in a lab for testing and we just ordered 15 SM863a's on ebay and just awaiting their arrival in a few weeks possibly. I just wanted more information before I break this when it arrives

Thanks for your help.

 

[aaron]

Normal procedure would be to first set the OSDs on the node about to be upgraded to Out. Then wait for the rebalance to be finished (Ceph cluster healthy). Then stop and destroy these OSDs. Shutdown the node and replace them with the faster SSDs. Once the node is back up, create the new OSDs with the SSDs, wait until the Ceph cluster is health again as it will rebalance again.

Keep doing the same with the remaining nodes.

If the rebalance takes too long, you could also stop all that by first setting norecover, noout, rebalance to on, then stop and destroy the OSDs, restart the server with the replaced drives, recreate the OSDs and then take away the OSD flags so the cluster can rebalance with the new OSDs. Once healthy, set the flags again and proceed with the next node.

The latter will leave the cluster in a degraded state though for the time that node is offline.

 

[sahostking]

ooooo - This is sooo easy then

I cant wait to test this out. Thanks for the response.

 

[aaron]

One more thing, with a 7 node cluster and 5SSD OSDs per node the 10Gbit network will most likely become the bottleneck. In our 2018 Ceph benchmark paper we did test for exactly that.

 

[sahostking]

Isnt that dependant on the amount of data?

The server has 4 x 1GB Ethernet ports
I could bond the dual 10Gb Network ports on the network cards as they come with 2 x 10Gb ports in each network card.

 

[sahostking]

One more thing, with a 7 node cluster and 5SSD OSDs per node the 10Gbit network will most likely become the bottleneck. In our 2018 Ceph benchmark paper we did test for exactly that.

I wanted to ask this question. Is this not related to the amount of data being written and read and only if it is maxed?

Looking at iotop each VM we will be hosting does around 25 to 50MB/s during busy periods of the day and we have around 15 VMs. So surely you are meaning it will only become the bottleneck if we require insane amount of of IOPS.