I would like to know if there is a real difference between installing Ceph externally and installing Ceph that is already installed on Proxmox.
Difference between Ceph already integrated into Proxmox and installing Ceph externally
- Thread starter UtilisateurCXS
- Start date
The main benefits of installing Ceph together with Proxmox VE are that you save costs and make administration (much) easier, while also being able to scale up your infrastructure quickly if you need to.
For example, a common setup is to have two-node Proxmox VE cluster with an extra QDevice and some kind of external storage (e.g. some kind of iSCSI box). The QDevice can also be on top of the external storage. This setup is rather cheap and usually enough if your needs aren't that high.
However, the problem with such a setup is that it's hard to scale up once you have higher demands for compute and/or storage. Instead, if you start off with a three-node Proxmox VE cluster and install Ceph alongside PVE, you can scale up both your compute and storage needs up more easily. If you need more compute resources, you can add more PVE nodes to your cluster; if you need more storage, you add more OSDs to each cluster node, for example. The overall setup is also easier, since you can administrate most of Ceph via the Proxmox VE GUI. This is called Hyper-Converged Infrastructure (see our wiki or Wikipedia on HCI). Also, since you start off with three nodes, you don't need an extra QDevice for quorum. (For completeness' sake, I want to mention that you should always have an odd number of nodes in your cluster if you don't want to use a QDevice.)
If you decided to set up an external Ceph cluster instead, you need to make sure that it's redundant as well—in other words, your external Ceph cluster needs to have at least three nodes. So, with the two-node PVE cluster + QDevice + minimal external Ceph cluster you would end up with 5 nodes (and a QDevice) in total that you would have to administrate; with a three-node PVE cluster + minimal external Ceph cluster you would end up with 6 nodes in total.
With Ceph installed on top of a three-node PVE cluster, you only have, well, three nodes. And you can use our tooling to administrate Ceph. So it really is mostly about saving costs and ease of administration. Maybe some Ceph veterans here have additional insights
For example, a common setup is to have two-node Proxmox VE cluster with an extra QDevice and some kind of external storage (e.g. some kind of iSCSI box). The QDevice can also be on top of the external storage. This setup is rather cheap and usually enough if your needs aren't that high.
However, the problem with such a setup is that it's hard to scale up once you have higher demands for compute and/or storage. Instead, if you start off with a three-node Proxmox VE cluster and install Ceph alongside PVE, you can scale up both your compute and storage needs up more easily. If you need more compute resources, you can add more PVE nodes to your cluster; if you need more storage, you add more OSDs to each cluster node, for example. The overall setup is also easier, since you can administrate most of Ceph via the Proxmox VE GUI. This is called Hyper-Converged Infrastructure (see our wiki or Wikipedia on HCI). Also, since you start off with three nodes, you don't need an extra QDevice for quorum. (For completeness' sake, I want to mention that you should always have an odd number of nodes in your cluster if you don't want to use a QDevice.)
If you decided to set up an external Ceph cluster instead, you need to make sure that it's redundant as well—in other words, your external Ceph cluster needs to have at least three nodes. So, with the two-node PVE cluster + QDevice + minimal external Ceph cluster you would end up with 5 nodes (and a QDevice) in total that you would have to administrate; with a three-node PVE cluster + minimal external Ceph cluster you would end up with 6 nodes in total.
With Ceph installed on top of a three-node PVE cluster, you only have, well, three nodes. And you can use our tooling to administrate Ceph. So it really is mostly about saving costs and ease of administration. Maybe some Ceph veterans here have additional insights
For me there was one additional reason to try pve-ceph: it is extremely easy to setup because everything is integrated in the WebGUI we are already firm with.