How to configure Ceph in a Proxmox cluster

[Asteroid]

There are three Proxmox hosts. The goal is to create a Proxmox cluster with a single Ceph repository. Ceph is already installed on one of the hosts, Public/Cluster networks are configured, and a monitor is created.
Is it possible at this stage to assemble a cluster of these three hosts, and then install and configure Ceph in Datacenter? Will the fact that Ceph was already installed on the first host and the initial configuration was performed affect the final result?

 

[jsterr]

There are three Proxmox hosts. The goal is to create a Proxmox cluster with a single Ceph repository. Ceph is already installed on one of the hosts, Public/Cluster networks are configured, and a monitor is created.
Is it possible at this stage to assemble a cluster of these three hosts, and then install and configure Ceph in Datacenter? Will the fact that Ceph was already installed on the first host and the initial configuration was performed affect the final result?

Im not sure If I understand your question. So you have installed on node, and installed proxmox ceph on it via ui or proxmox ceph commands? now you want to cluster the three nodes and install ceph on the other ones as well. This should work, if the three nodes have the same network config etc. so they can ping each other etc.

Ceph is always installed via host view, not datacenter view as you need ceph installation on each of the hosts. The other two hosts should see the already initialized ceph conf from node1, so they just install packages and integrate themselfes. you need to manually create monitor and mgr after that on the other two hosts.

 

[Asteroid]

Thanks jsterr for such a detailed answer.
On the first host, ceph was installed via the Web. On the remaining two hosts, Ceph is not yet installed (not configured accordingly).
I wonder why there is a separate Ceph branch in Datacenter if it is configured on every host anyway.

 

[scyto]

try this if you are looking for an example of how to setup ceph on 3 nodes https://gist.github.com/scyto/76e94832927a89d977ea989da157e9dc

 

[jsterr]

Thanks jsterr for such a detailed answer.
On the first host, ceph was installed via the Web. On the remaining two hosts, Ceph is not yet installed (not configured accordingly).
I wonder why there is a separate Ceph branch in Datacenter if it is configured on every host anyway.

You can view the ceph dashboard (proxmox ceph dashboard) via that datacenter view as it contains information about the ceph cluster system, not the single ceph node configuration.

 

[Asteroid]

try this if you are looking for an example of how to setup ceph on 3 nodes https://gist.github.com/scyto/76e94832927a89d977ea989da157e9dc

Thank you! Good instructions.

Now all that remains is to think carefully about the storage architecture. Different servers have disks of different sizes.
1 server: 2 x 140 Gb
2 server: 2 x 300 Gb
3 server: 2 x 70 Gb
4 server: 2 x 70 Gb

 

[Asteroid]

You can view the ceph dashboard (proxmox ceph dashboard) via that datacenter view as it contains information about the ceph cluster system, not the single ceph node configuration.

Yesterday I deployed three test Proxmox servers at home. Then I first assembled a cluster from them, and then confused the process of installing Ceph from the dashboard of the cluster’s web interface. Some difficulties arose, it was too late, I didn’t figure it out yet. Today, according to the scyto instructions, I first installed ceph on each of the hosts on other equipment, and then configured Ceph.

 

[jsterr]

Thank you! Good instructions.

Now all that remains is to think carefully about the storage architecture. Different servers have disks of different sizes.
1 server: 2 x 140 Gb
2 server: 2 x 300 Gb
3 server: 2 x 70 Gb
4 server: 2 x 70 Gb

That wont work out very well, I would only use this for testing not for production. You will likely run into issues when one disk fails, as data needs to be recreated on on of the nodes to regain SIZE = 3 (3 replicas per cluster). Ceph will set lower weighting on disks that have less space, but yeah if a 300GB disk fails, how will you make sure that those 200-250GB that were maybe on it can fit on the other disks ...

 

[scyto]

Some difficulties arose, it was too late, I didn’t figure it out yet.

yeah i hit same sort of issues, what i documented was my 3rd try based on everything i learn't from the first two - i tore down ceph competely twice (never had to rebuilt the cluster) - basically the UI geets easily locked and confused, which is why some things in my notes are done at command line for ceph. tl;dr you need 100% stable cluster and networking config before thinking about doing ceph.

 

[scyto]

Different servers have disks of different sizes.

definitely, i don't know enough to advise, there are some OSD and PG calculators out there, but most of the time i couldn't answer the input questions, lol!