Recommended setup

Bobbbb

Well-Known Member
Jul 13, 2018
70
1
48
28
Hi everyone,

I currently have 1 server, running about 20 sites.
I would like the change that to a proxmox server, and virtual the existing server and create some kind of HA.

I currently have 2 servers with 4 x 480 ssd drives.

my plan is , RAID 5 for proxmox and Ceph with the additional drive.
once thats done, i will get (or use an existing ..) smaller server and allocate a drive for the ceph, and in future perhaps add more.

is that a good setup?i believe HA is currently not an option with ZFS and i am required to use ceph.

any other ideas are also welcome!
thanks a lot,
Bob
 
or is raid not even required because in case of a drive failure HA will just fail over to another cluster?
 
I currently have 2 servers with 4 x 480 ssd drives.
...
my plan is , RAID 5 for proxmox and Ceph with the additional drive.

Some notes regarding your config:

1. you need a minimum of 3 nodes for ceph.
2. ceph with one drive is not going to give you useful results.
3. parity raid is not well suited for virtual disks; it will work but performance may not be adequate.
4. RAID5 is dangerous. When a drive fails you will be operating without parity, which leaves you exposed to data corruption untl the disk is replaced and rebuilt. RAID10 would function better at the cost of 480GB less usable space then raid5.
 
Some notes regarding your config:

1. you need a minimum of 3 nodes for ceph.
2. ceph with one drive is not going to give you useful results.
3. parity raid is not well suited for virtual disks; it will work but performance may not be adequate.
4. RAID5 is dangerous. When a drive fails you will be operating without parity, which leaves you exposed to data corruption untl the disk is replaced and rebuilt. RAID10 would function better at the cost of 480GB less usable space then raid5.

THANKS FOR THE REPLY!

1) my end goal is to have 3 nodes with 3 ceph, (and maybe more after that...)
i just need to get started.
currently i have 2 servers with 4 x 480gb SSD so need to decide on the best configuration to get started.

I agree that raid 10 is always best practice, but (and please correct me if i am wrong...:))
i will not be able to have proxmox and ceph with 4 drives and raid 10?
2) isn't parity good enough, considering that i'll have a 3 node cluster?

EDIT: unless i do raid 1 for the proxmox, and the rest without raid?
surely its ok if the ceph drives are not raided?
 
you would not normally have a raid backed vdisk store AND ceph backed. pick one.

1) my end goal is to have 3 nodes with 3 ceph, (and maybe more after that...)
i just need to get started.
currently i have 2 servers with 4 x 480gb SSD so need to decide on the best configuration to get started.
If you intent to roll out ceph, 3 nodes is where you start.

i will not be able to have proxmox and ceph with 4 drives and raid 10?
You can have a RAID10 volume with any multiple of 2 drives. Some controllers will allow you to make a raid1e volume with any number of drives, which is similar to the way ceph does replication groups.

2) isn't parity good enough, considering that i'll have a 3 node cluster?
Let me make it simpler. parity and replication are two methods of creating fault tolerance. Whats "good enough" is dependent on your tolerance for performance, downtime and/or data loss. If you'll deploy a ceph pool, it will come down to dual or triple replication.

EDIT: unless i do raid 1 for the proxmox, and the rest without raid?
Exactly.
 
thanks again.

stupid question, is ceph = object storage solution that many companies offer?
looking at getting some space from digital ocean.
 
thanks again.

stupid question, is ceph = object storage solution that many companies offer?
looking at getting some space from digital ocean.
anyone knows?
Can i add digital ocean spaces or amazon S3 storage as external RBD?
 
stupid question, is ceph = object storage solution that many companies offer?
https://ceph.com

Ceph can do object storage but that is only one of its interface capabilities.

Can i add digital ocean spaces or amazon S3 storage as external RBD?
Not easily. anything is possible with enough persistence- but dont expect it to have useful (performant) application.
 
And I guess that most of this sites have some kind of DB ?

Yes that is correct.
what is the recommendation regarding the DB?

My 3rd proxmox node, will be at another location, so will have 2 proxmox nodes at the same data canter and a 3rd one in another country.
will that be ok for ceph?

once everything is moved to the new cluster , i will add another node.
 
My 3rd proxmox node, will be at another location, so will have 2 proxmox nodes at the same data canter and a 3rd one in another country.
will that be ok for ceph?

I do not use ceph, but I think that it will be a problem to use a 3rd node in another DC/country, because you will need a lot of bandwidth(for ceph), and PMX cluster need a very good latency. If this is your usage scenario, I can guess that it will not work. It could work with async replication data from your DC(PMX cluster) -> remote DC(PMX another cluster or non-cluster standalone node). So in the remote DC will be a remote DC disaster recover. But if you need to write data in both DC at the same time, you will need to use some kind of sql multi-node cluster(like percona mysql cluster).
 
Yes I was thinking about using percona (we actually currently using it), just had some bad experience with it (although the environment wasn't very stable).

My plan is that the DC's that are in the same DC will fail over to each other, with fast LAN between them, only in the very unlikely event that both of them will die, the 3rd will be used.
 
are there any other storage solutions that can work other then ceph?
my original plan was to use ZFS but i see HA is not supported.
 
Yes I was thinking about using percona (we actually currently using it), just had some bad experience with it (although the environment wasn't very stable).
I run percona-cluster over zfs(at least for 3 years) and I can say that for me is rock solid.
 
are there any other storage solutions that can work other then ceph?
my original plan was to use ZFS but i see HA is not supported.

You can use zfs if it is OK to have async replication via zfs(shedule for each 5 min) from node X1 -> node X2. And with percona-cluster on both nodes/VMs you will not lose (almost)any data. You will need to use a moniroring tool like monit, who will start VMx using the last snapshot who was replicated(less than 2 min).

Another option is to use lizardfs(ditributed and/or replicated) - now I am evaluate this tool. lizardfs could be used in a HA enviroment under PMX.
 
the monit will run on the proxmox host?

so i like your idea....but i am also confused now.
lol

do you recommend zfs over ceph based on my requirements and what i have available?
 
the monit will run on the proxmox host?
Yes.

VMx(node1)---- >zfs replicate ----> VMx(node2/monit)

VMx(node1) is up and running, VMx(node2) is down(because they have the same ID). Monit check at each 90 seconds(or whatever do you want) if VMx(node1) is up. When VMx(node1) is down, monit it will change VMx(node2) ID, and the name of the vHDD(using the changed ID), then it will start this new VMx on node2 with a new ID, with the data after the last successful zfs sync. It could take less then 2 min the all process.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!