Scenarios for Proxmox + Ceph

[marcelovvm]

I would like to ask a question that I am having. I have the following scenario, 6 HP Proliant DL360/380 G7 servers that I am wanting to create a Proxmox + Ceph cluster. All these servers have the same configurations: 2x Xeon E5630 Quad Core CPU, 48GB RAM, 4x 480GB SSD (connected using LSI SAS 9211-8i non-raid) and dual 10GBE SFP+ network card. I understand virtualization well (today these servers are with ESXi), but very little about SDS (ZFS, Ceph, etc.). Researching Proxmox + Ceph I found that I have two scenarios for my future architecture:

Scenario A: use all 6 servers with Proxmox + Ceph and create an SDS with 4 OSDs per each server using all 6 servers.

Scenario B: use 3 servers with Proxmox + Ceph and create an SDS with 4 OSDs per each server using the 3 servers AND use 3 servers with only Proxmox to host my VMs.

My environment: 15-20 VMs between W7, W10/11, Windows Server and Linux. My VMs use 4/8/16GB of RAM and they all have a 100GB virtual disk. All 10GBE boards have two SFP+ ports, but today I only use one exclusive one for VMs. The servers have 4 integrated 1GB NICs that I use for management and vMotion (ESXi).

1) What would be the best scenario A or B? Why?

2) How many Ceph monitors should I install in scenarios A or B?

p.s. I know the servers are old but they serve the purpose perfectly, I'm just looking to use Proxmox as ESXi no longer supports these servers.

Live long and prosper,
Marcelo Magalhães
Rio de Janeiro - Brazil

 

[jsterr]

I would like to ask a question that I am having. I have the following scenario, 6 HP Proliant DL360/380 G7 servers that I am wanting to create a Proxmox + Ceph cluster. All these servers have the same configurations: 2x Xeon E5630 Quad Core CPU, 48GB RAM, 4x 480GB SSD (connected using LSI SAS 9211-8i non-raid) and dual 10GBE SFP+ network card. I understand virtualization well (today these servers are with ESXi), but very little about SDS (ZFS, Ceph, etc.). Researching Proxmox + Ceph I found that I have two scenarios for my future architecture.

Nice! Im always happy people learning about ceph.

Scenario A: use all 6 servers with Proxmox + Ceph and create an SDS with 4 OSDs per each server using all 6 servers.

Are the servers all in the same room? usually u want a uneven number, it doesnt matter if all servers are in the same room.

Scenario B: use 3 servers with Proxmox + Ceph and create an SDS with 4 OSDs per each server using the 3 servers AND use 3 servers with only Proxmox to host my VMs.

Yeah you can also do that, you can have ceph nodes that are not having osds, but still access the storge of the first 3 nodes. Are these ssds enterprise ssds with powerloss protection?

My environment: 15-20 VMs between W7, W10/11, Windows Server and Linux. My VMs use 4/8/16GB of RAM and they all have a 100GB virtual disk.

Its total 288GB RAM (6x 48GB) whats your total ram usage you plan to use in vms? Storage is thin provisioned in ceph, so if your not using 100GB there will be less used, even though you have 100GB provisioned in vm.

10GBE boards have two SFP+ ports, but today I only use one exclusive one for VMs. The servers have 4 integrated 1GB NICs that I use for management and vMotion (ESXi).

I would use:

- 1Gbit Port Management (+ Corosync Fallback)
- 1Gbit Port Corosync (Cluster Management)
- 1x 10Gbit for VMs (no redundancy though)
- 1x 10Gbit for Ceph (no redundancy though)

OR

2x 10Gbit LACP and separate Ceph / VM Traffic via Linux-VLANs.

1) What would be the best scenario A or B? Why?

Both are valid setups, but the more osds the more performance. You also balance storage-load over more nodes if you have 6 nodes with osds instead of 3. All nodes still need access to ceph network, so the only difference on Setup B is that you dont create osds there.

Regarding redundancy youll have more optiouns with setup A, as you could potentially go to a 4/2 (4 Replicas across 4 Hosts, with Minimum 2 copys always available). Even if your staying on 3:2 you could potentially loose more then one node without downtime, but only if ceph complete recovered from the first host that failed. With 6 servers and a 3:2 setup (3 Replicas, 2 MIN requirement to make the pool worK) you could loose 1 server, ceph recovers, loose another, ceph recovers. until you have only 2 servers left (and aslong your workloads can be run on that less servers). On 3:2 you can never loose 2 servers at the same time without loosing storage access.

2) How many Ceph monitors should I install in scenarios A or B?

Depends on the location of the servers. If all are in the smae room, MIN 3 maximum 5, I would go with 3. If your thinking about splitting between rooms theres a lot more things to consider.

p.s. I know the servers are old but they serve the purpose perfectly, I'm just looking to use Proxmox as ESXi no longer supports these servers.

Yeah, welcome to Proxmox Ceph

 

[marcelovvm]

Nice! Im always happy people learning about ceph.



Are the servers all in the same room? usually u want a uneven number, it doesnt matter if all servers are in the same room.



Yeah you can also do that, you can have ceph nodes that are not having osds, but still access the storge of the first 3 nodes. Are these ssds enterprise ssds with powerloss protection?



Its total 288GB RAM (6x 48GB) whats your total ram usage you plan to use in vms? Storage is thin provisioned in ceph, so if your not using 100GB there will be less used, even though you have 100GB provisioned in vm.



I would use:

- 1Gbit Port Management (+ Corosync Fallback)
- 1Gbit Port Corosync (Cluster Management)
- 1x 10Gbit for VMs (no redundancy though)
- 1x 10Gbit for Ceph (no redundancy though)

OR

2x 10Gbit LACP and separate Ceph / VM Traffic via Linux-VLANs.



Both are valid setups, but the more osds the more performance. You also balance storage-load over more nodes if you have 6 nodes with osds instead of 3. All nodes still need access to ceph network, so the only difference on Setup B is that you dont create osds there.

Regarding redundancy youll have more optiouns with setup A, as you could potentially go to a 4/2 (4 Replicas across 4 Hosts, with Minimum 2 copys always available). Even if your staying on 3:2 you could potentially loose more then one node without downtime, but only if ceph complete recovered from the first host that failed. With 6 servers and a 3:2 setup (3 Replicas, 2 MIN requirement to make the pool worK) you could loose 1 server, ceph recovers, loose another, ceph recovers. until you have only 2 servers left (and aslong your workloads can be run on that less servers). On 3:2 you can never loose 2 servers at the same time without loosing storage access.




Depends on the location of the servers. If all are in the smae room, MIN 3 maximum 5, I would go with 3. If your thinking about splitting between rooms theres a lot more things to consider.



Yeah, welcome to Proxmox Ceph

Hi jsterr,

Yes. The 6 servers are on the same rack in the same room.

Based on your answers and researching more on the subject, I concluded that scenario A will be better for my case, since if I opted for scenario B and a server from the Ceph cluster OR a server from the VM cluster failed, I would only be left with one two, without good redundancy. Already choosing scenario A, the 6 servers with Proxmox + Ceph I can lose up to 4 servers (!!!) so that there is no redundancy (assuming a replication of 3 times).

However, I had a doubt when you talk about RAM memory (already knowing that Ceph is a big RAM memory eater). All 6 servers have 48GB RAM (288GB RAM total) and 4 x 480GB SSDs. So at the end of the Ceph settings I will have 24 OSDs of 480GB each. In my research I found several mentions of the formula 1GB of RAM for 1TB of OSD. In my accounts I would need 12GB of RAM! Just that?

You ask about my VMs RAM. Today my 15 VMs allocate 150GB/160GB of RAM, that is, in total I have more than 100GB of RAM free. Another doubt is whether the RAM amount account should be made for the whole cluster, even if the servers were not so similar to each other (mine are identical - and they are all in the same cluster)?

For the allocation of network cards, I found your first suggestion better:
- 1Gbit Port Management (+ Corosync Fallback)
- 1Gbit Port Corosync (Cluster Management)
- 1x 10Gbit for VMs (no redundancy though)
- 1x 10Gbit for Ceph (no redundancy though)

My 15 VMs today use 1.5TB (Thin Provision - ESXi), adding the total allocated spaces, not the space in use (much smaller). Finally, how can I calculate the total size (with replicas) of my Ceph storage?

Live long and prosper,
Marcelo Magalhães
Rio de Janeiro - Brazil