ceph 3-cluster with 2-copies per object

vkhera

Member
Feb 24, 2015
192
14
18
Maryland, USA
I've read through both the books and the online docs for Proxmox over the last couple of weeks. I am planning to set up a cluster to replace 4 physical servers, one of which contains about 5 virtual machines on it already which I manually manage.

My plan is to get 3 servers each with SSD for boot/OS, and 4 x 1TB drives for data on a Ceph cluster, 32GB RAM and 10Gb ethernet for the inter-node communications. Every tutorial I see shows the ceph redundancy at 3 copies. I was planning to use a 2 copy redundancy. I will have backups go to an NFS file server. All of the VMs will be KVM based and running mostly FreeBSD for web site development purposes, and one for the office file share. I need a total of about 2TB of disk today, with room to grow.

I know if two nodes go down I will lose data that is not backed up. Anything else? Will there be a noticeable speed impact on the performance of the VMs? Some of the files will have to come from a remote node, but at the 10Gb speed I'm thinking it will be ok.

Is there some other fatal flaw in doing this? Will it make upgrading hardware (ie replace a whole node with bigger faster hardware) difficult?
 
I've read through both the books and the online docs for Proxmox over the last couple of weeks. I am planning to set up a cluster to replace 4 physical servers, one of which contains about 5 virtual machines on it already which I manually manage.

My plan is to get 3 servers each with SSD for boot/OS, and 4 x 1TB drives for data on a Ceph cluster, 32GB RAM and 10Gb ethernet for the inter-node communications. Every tutorial I see shows the ceph redundancy at 3 copies. I was planning to use a 2 copy redundancy. I will have backups go to an NFS file server. All of the VMs will be KVM based and running mostly FreeBSD for web site development purposes, and one for the office file share. I need a total of about 2TB of disk today, with room to grow.

I know if two nodes go down I will lose data that is not backed up. Anything else? Will there be a noticeable speed impact on the performance of the VMs?
Hi,
for write speed replica 2 is faster then replica 3 (because less writes). Read speed is roughly the same (still now) because the data is read from the primary only.
Some of the files will have to come from a remote node, but at the 10Gb speed I'm thinking it will be ok.
Many files came from the remote-nodes (2/3)!
Is there some other fatal flaw in doing this? Will it make upgrading hardware (ie replace a whole node with bigger faster hardware) difficult?
No difference between 2 and 3 replicas (but 3 replicas are recommended).
I use replica 2 till 60 OSDs - now with 84 OSDs I switched to replica 3.
One benefit of ceph is the easy extension/migration.

Don't fill your OSDs too much (Ceph don't like it) and use the right mount options/ceph params.

Udo
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!