Ceph hardware / installation recommendations

[tcz]

Hi Everybody,

we are discussing internally our new proxmox ceph cluster with 4 nodes (ceph) and 5 nodes (compute).

Our plans are either
- 4x 2TB + 3x 4TB SSD's (the 4TB SSD's are currently used in our existing storage) or
- 7x 4TB SSD's

Is anybody already using a "mixed" SSD setup? Or should we go only with same size SSD's? What TBW should this SSD have?

Also we are not sure what type of SSD would fit best on a raid-z1 for the OS itself. We read many different opinions about size / TBW / durability / liability necessary for this SSD. What are the real / production values for that?

Thanks!

 

[alexskysilk]

Is anybody already using a "mixed" SSD setup? Or should we go only with same size SSD's? What TBW should this SSD have?

As long as your nodes are balanced (as in, have the same total capacity) mixed sizes should work. only you can answer the TBW question since only you know how much you're going to write, but bear in mind that rebalances can be pretty write intensive.

Also we are not sure what type of SSD would fit best on a raid-z1 for the OS itself

This doesnt matter much, so long as you're only using them to boot; if you intend to use them for other purposes/services you may want to mention it. Also, no real reason to have a raidz1; a simple mirror should suffice.

We read many different opinions about size / TBW / durability / liability necessary for this SSD. What are the real / production values for that?

I imagine if you read the different opinions you'd have the backing logic too. In my view, pve nodes should be fungible and therefore the boot storage is not really very important.