Ceph, Proxmox cluster. OS on single disk or RAID?

[driftux]

Hi,
for each server I have 2 HHD and 1 SSD disks.

I wonder do I really need to put Proxmox OS on ZFS raid1 with two disks or just use a single HDD disk, because in HA cluster environment in case of server (virtual machine) failure another server (VM) automatically will take place? So this means I don't even need Raid for OS, am I right? I plan a several dozen nodes cluster and want to minimize the budget. I don't have hardware raid controller. This will be my first cluster.

If I put Proxmox OS on HDD (or ZFS RAiD1), and Ceph OSD on SSD drive, would my HDD won't slow down the performance of virtual machines? I choose between OpenVZ and KVM machines to put on OSD, but I don't know which will be better in this scenario. Will those two technologies being on SSD, can get negative impact that the main Proxmox OS is on slow HHD drive. Could someone point me to the right direction?

 

[Dominic]

When you don't have a RAID mirror then the death of your OS disk means your node is down and the guests (VMs and containers) will automatically start on another node (if you have everything set up correctly). It is usually more important that your VMs and containers are on SSDs than your Proxmox VE installation. You can install both VMs and containers. By the way Proxmox VE now uses LXC instead of OpenVZ.

I plan a several dozen nodes cluster

Then one node going down should not be a problem. But then you will also need to invest quite some money into network infrastructure and the cost of an additional small SSD for the operating system will be negligible.

Take a look at our benchmarks to see why you should get enterprise grade SSDs.

 

[driftux]

Ok, thanks for the answer. I slowly get the whole picture. I didn't think that lower grade SSD is so low on IOPS.
I also noticed that XFS, according to your document will use 50% of free RAM.

Actually my servers are quite old and each has only 16GB RAM. Investing in better network cards will be costly, and I'm on very tight on budget. I thought that in some way I could live with 1GB network cards and switch. I mean, I take disadvantage that in case of server failure 6 hours all the systems could work 30% slower because of sync.

Actually each server will have 1 OSD on 500GB SSD and 1 OSD on 2TB HDD (5400RPM). Will I survive with such parameters in production environment, how do you think?

 

[Dominic]

Virtual guests should start even if your hardware is slow. The main question is if the speed they achieve is fast enough for your workload - and this only you can test.

What could become a problem is for example if you have a single network for Ceph & PVE Cluster (Corosync): If storage traffic congests your network and corosync traffic doesn't get through in time your cluster will be unstable.

 

[driftux]

I see that you understand what you are talking. This is very useful information.Each server will have 2 networks cards. I read that Ceph needs to cards for internal and public traffic. So I safe here I guess.
Would you suggest to go with LXC or KVM on Ceph, knowing that each server is with only 16GB RAM and E3 CPU? Of course in production environment stability is very important.

 

[Dominic]

Separating Ceph traffic (= separate cards) is a good idea.

LXC is stable and should be low on resource consumption.

 

[driftux]

Ok, understood. Thanks for your time.