Building a separate ceph storage cluster

[ejmerkel]

We're evaluating different shared storage and we're contemplating using ceph. We've tried this in the past with a hyper- converged set up and it didn't go so well so we're wanting to build a separate ceph cluster to integrate with our proxmox cluster. We've read the benchmark document and have an idea of the best hardware/network but we are wondering what we should use to build the cluster?

1. Should we just build the cluster using proxmox nodes with no VM's on it and use the GUI?
2. Should we try to build it by hand using something like ceph-deploy? I am a little leery of that route because we are not ceph experts by any stretch of the imagination.
3. Should we look to a vendor to build a ceph cluster for us and provide support?
4. Something other option?

I am just looking for advice. I appreciate it!


P.S. Forgot to mention, we are looking for 50TB of usable storage to begin with and scale up to 100TB in 6 months. Over the next year or two we'll probably want to be at 150-200TB.

Best regards,
Eric

 

[mira]

The easiest solution would probably be to use PVE for the setup.
With Basic or higher subscriptions, you will be able to get support via my.proxmox.com, should there be any issue.

If it is part of a bigger cluster, you can configure it so that no VMs/CTs can be created on the Ceph nodes.
If it is a standalone cluster, you can still manage it easily through the Web GUI.


As to other solutions, I don't have experience with those. But there might be some nice solutions by other providers. This of course means no support from our side for the Ceph cluster.

 

[czechsys]

For OP:
4. Rook or any configuration management solutions

Anyway, we are calculation external Ceph storage for our PVE too.
PVE Staff, are there some requirements for external clusters? For example, version difference etc? PVE documentation handles hyperconverged/updates mainly.

 

[mira]

By default on PVE 7 the Debian Ceph packages are used. Which are 14.2 in bullseye. So anything that is compatible with that version.
If you require support for newer versions, you can always add our Ceph repositories and run apt update and apt full-upgrade to get Octopus or Pacific versions of the already installed Ceph packages.