Ceph - Expantion, more disks and servers.

[potetpro]

Greetings.

We are about to get Proxmox for out new cluster solution (see image below), and i need to make sure i understand this completely before we commit to this investment. The Backup server will be full image backup, to a backup server running alongside the Proxmox servers. The backup server will be mirroring using glusterfs to a offsite data center for disaster recovery.




At first, lets say two SSD per. server. with the default settings from the wiki guide on how to setup ceph.
One OSD per. disk, One monitor pr server. Size 3/ minimum 2 , pg 128. So my questions are:
-Does ceph understand to spread the data over 3 server, minimum 2, and not put the data on multiple ssd's on the same server and call it a day, and then the server fails ? (i am guessing this is thought out)

-When i need to add storage to Ceph, can i just disks, create new OSD's on those disks, and they will be usable to the exsisting pool?

-What does the "add storage" do when creating a pool? is this needed when adding more disks?
-When i create a pool with size 3 and minimum 2, does this mean that pool occupy 3 OSD's, and only use them? or is it spread out over the entire cluster?
The reason i ask is because the video in the wiki regarding ceph setup, he creates two pool (one for images, and one for containers) and he has 3 servers with 2 disks in each, and if i try to create two pools in my current setup with 3 server and 1 disk in each, i get
mon_command failed - pg_num 128 size 3 would mean 768 total pgs, which exceeds max 750 (mon_max_pg_per_osd 250 * num_in_osds 3)wqU"

Still trying to figure out how the placement groups, size and minimum in ceph relates to the numbers in the ceph calculator below:
ceph.com/pgcalc/


Thanks in advance

 

[Alwin]

-Does ceph understand to spread the data over 3 server, minimum 2, and not put the data on multiple ssd's on the same server and call it a day, and then the server fails ? (i am guessing this is thought out)

This is the default. You can check this in your crushmap at the rules section.

-When i need to add storage to Ceph, can i just disks, create new OSD's on those disks, and they will be usable to the exsisting pool?

Yes, as long as the pool is working with a rule that includes those OSDs (by default all). Best practice is to keep the disk size and count even balanced on each host.

-What does the "add storage" do when creating a pool? is this needed when adding more disks?

The 'add storage' adds the newly created pool as a storage to Proxmox. This is just for convenience and only needs to be considered on pool creation.

-When i create a pool with size 3 and minimum 2, does this mean that pool occupy 3 OSD's, and only use them? or is it spread out over the entire cluster?

Yup, Ceph is clever and takes alle OSDs for storage. This is also a reason why you need to calculate how many PGs for your pools are needed. https://ceph.com/pgcalc/

The reason i ask is because the video in the wiki regarding ceph setup, he creates two pool (one for images, and one for containers) and he has 3 servers with 2 disks in each, and if i try to create two pools in my current setup with 3 server and 1 disk in each, i get
mon_command failed - pg_num 128 size 3 would mean 768 total pgs, which exceeds max 750 (mon_max_pg_per_osd 250 * num_in_osds 3)wqU"

You don't need two pools, container and VMs can run on the same pool. The reason Ceph complains is that the amount PGs in the current configuration is way more then needed and will only take a big resource overhead. See the calculator above for the pool sizing, you can increase the PG count later on but not decrease it (limitation of luminous).

Still trying to figure out how the placement groups, size and minimum in ceph relates to the numbers in the ceph calculator below:
ceph.com/pgcalc/

The PGs are buckets for objects, as it would be very computational intensive to calculate the storage location for a single object. So Ceph calculates the location of a PG and gets its objects. The 100% capacity is split by the number of pools and depending on its copy count a certain amount of PGs is needed for each pool. To make it even more complex, the "optimal" count of PGs per OSD should be 100 PGs and the value of the nearest power of two is taken.

 

[potetpro]

Thank you very much for your answer