2 node cluster + CEPH
- Thread starter Dimon4ik
- Start date
Minimum for Ceph is 3 nodes but not recomended for production.
You should use at least 6 nodes, 2 osd each and an enteprise ssd for bluestore db. Hardware, CPU 1 core for each OSD, 1GB RAM for each 1TB of OSD, 3 gigabit network cards, one for proxmox network, two for ceph network (bond).
Do not run VM on thesame server with Ceph / OSD's.
For the compute nodes, use minimum, 4 network cards. One for public (internet), one for Proxmox network, two for Ceph network (bond).
You should use at least 6 nodes, 2 osd each and an enteprise ssd for bluestore db. Hardware, CPU 1 core for each OSD, 1GB RAM for each 1TB of OSD, 3 gigabit network cards, one for proxmox network, two for ceph network (bond).
Do not run VM on thesame server with Ceph / OSD's.
For the compute nodes, use minimum, 4 network cards. One for public (internet), one for Proxmox network, two for Ceph network (bond).
Not true, you can use three nodes in a production setup. See the benchmark paper for performance details.
https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark-2018-02.41761/
Depending on your hardware, you can also use less server and more OSDs per server (see the link above). As you stated for the hardware, ceph needs those resources solely available, otherwise its performance will suffer. With more OSDs per server, you will also need higher bandwidth, as the OSDs will max out the 2x1Gb.
No true, this is very usual in a hyper converged setup. But as stated above, you need to have resources solely for each service available.
I agree, it is very important to physically separate your network. This will not only benefit security, but also reduce the interference, latency and provide dedicated bandwidth for each service.
# POC Environment — Can have a minimum of 3 physical nodes with 10 OSD’s each. This provides 66% cluster availability upon a physical node failure and 97% uptime upon an OSD failure. RGW and Monitor nodes can be put on OSD nodes but this may impact performance and not recommended for production.
# Production Environment — a minimum of 5 physically separated nodes and minimum of 100 OSD @ 4TB per OSD the cluster capacity is over 130TB and provides 80% uptime on physical node failure and 99% uptime on OSD failure. RGW and Monitors should be on separate nodes.
Source: https://ceph.com/geen-categorie/zero-to-hero-guide-for-ceph-cluster-planning/
I must admit, 100 osds in minimum 5 physically nodes it's a bit insane.
) Nevertheless this is from the ceph.com website. True, but that was a minimum configuration of a node.
I agree with you, but ceph osd and monitor daemons needs a lot of resources. If you run them on an monster server and the resources needed by the VM's are low, sure you cand run them on the Ceph node. But surely, I don't recommend this practice.
Yes, ceph was always made with the intention of having huge amounts of data. As, they are talking about > 100 TB, a hyper converged scenario is out of question (and I wonder if redhat and suse, emphasize big clusters more).Source: https://ceph.com/geen-categorie/zero-to-hero-guide-for-ceph-cluster-planning/
I must admit, 100 osds in minimum 5 physically nodes it's a bit insane.
It is then also clear, why one node doesn't need a high amount of resources.
I agree, that it is very dependent on the workload you want to run, for many people it is enough to have three nodes in a hyper converged setup, as their workload is not demanding, but they want the extra failure safety (even HA). As you need to have at least three nodes for quorum, not only for ceph, also for PVE. It is a very quick and convenient way to go with decent performance.
But we can sure put some emphasize on it, that it is not recommended for every workload/demand!