HA Proxmox with CEPH

[KHosting]

Hello,

I am trying to deploy a HA Hypervisor with Proxmox to host a few VMs (50+)

I was reading about CEPH on https://pve.proxmox.com/wiki/Ceph_Server and it seem like what I really need. Now my concerns are;

1. Must all the 3 server be exact replicas in CPU (& Cores) Disk and RAM?
2. Is deploying CEPH in Proxmox Cluster sufficient for HA so that if a server fails, the VMs live move on any of the other servers?

Thank you.

 

[udo]

Hello,

I am trying to deploy a HA Hypervisor with Proxmox to host a few VMs (50+)

I was reading about CEPH on https://pve.proxmox.com/wiki/Ceph_Server and it seem like what I really need. Now my concerns are;

1. Must all the 3 server be exact replicas in CPU (& Cores) Disk and RAM?

Hi,
no - but you should have enough RAM (to speed up ceph too).
For live migration it's recommendet to use the same CPU, but mostly it's works well with different CPU (but not any time).
For 50+ VMs the IO from an 3-node-ceph cluster looks not sufficient for me (full SSD?)... you should try it before.

2. Is deploying CEPH in Proxmox Cluster sufficient for HA so that if a server fails, the VMs live move on any of the other servers?

Thank you.

Should work. Ceph can handle normaly such things good (but I don't have HA experiences).

Udo

 

[KHosting]

Thank you @udo.

Yes, I plan to do all full SSD on all 3 nodes.

The reason I was asking if 3-node-ceph is sufficient HA is because I can see there is just another HA how-to here https://pve.proxmox.com/wiki/High_Availability_Cluster_4.x

I don't know if this MUST be done to achieve real HA or the 3-node-ceph is just ok.

 

[udo]

Thank you @udo.

Yes, I plan to do all full SSD on all 3 nodes.

The reason I was asking if 3-node-ceph is sufficient HA is because I can see there is just another HA how-to here https://pve.proxmox.com/wiki/High_Availability_Cluster_4.x

I don't know if this MUST be done to achieve real HA or the 3-node-ceph is just ok.

Hi,
why not - I read nothing there why it should not work. at least 3 nodes and shared/distrubuted storage like ceph! that is what you plan.

Udo

 

[markusd]

Hi,
I think you need a lot of cpu-power if you plan to put io-intensive VM`s and ceph-osd`s on the same nodes.
I am interested too, if someone has got tipps how to balance this.

Markus

 

[jinjer]

Last time I attempted to use the same nodes for ceph and VM, I used 4 nodes, 4 disks per node and just one VM per node. The result was disastrous. Ceph would eat all resources not leaving anything for proxmox and kvm.

I wad advised at that time (around 3.1 era) that it was not a good idea to use the same nodes for both ceph and kvm, and that ceph integration in proxmox was there just to ease deployment of ceph.

I'm not sure if this is still the case. I have never looked at ceph again (as it had other issues at the time).

Be carefull. Your project looks overly optimistic.

 

[KHosting]

I see your angle on the memory and cpu usage if Ceph and VM are on the same nodes.

So what would you suggest as an optimal design. Essentially, I need to be able to host VMs and ensure "high availability" or some form of quick fail-overs in case of a node failure due to whatever reason.

I am asking because I have no pre-experience in this and hoping to get it right.

Please let me know what you think will work well based on your experiences.

 

[jinjer]

Ceph is CPU and DISK hog. Specially DISK hog. It has to run on it's own nodes. I've seen loads up to 100 during ceph reconstruction and these are considered "normal" in order to squeeze the last 0.x% from the disks.

So, best recommendation is to do as everybody: have disk nodes and compute nodes separate.

No free lunch.

 

[KHosting]

Last time I attempted to use the same nodes for ceph and VM, I used 4 nodes, 4 disks per node and just one VM per node. The result was disastrous. Ceph would eat all resources not leaving anything for proxmox and kvm.

I wad advised at that time (around 3.1 era) that it was not a good idea to use the same nodes for both ceph and kvm, and that ceph integration in proxmox was there just to ease deployment of ceph.

I'm not sure if this is still the case. I have never looked at ceph again (as it had other issues at the time).

Be carefull. Your project looks overly optimistic.

Hello Jinjer,

What kind of CPU and RAM size do you have on each of the 4 nodes ?