Getting started with a small cluster but trying to get it right from the start

[steffen]

I have been running VM Ware for a few years now and the main Problem is getting complete snapshots efficently and effectively done without the several thousand dollar package.

A few years back I got the Basic Package, but let's not spend time talking about that.

I have been testing on the side with Proxmox. Had a 2.x machine running for my own testing on some old hardware (doing great - install was out of the box / iso CD). I installed a Proxmox 3.1 (free repository) ontop of Debian Wheezy on an external Server and the KVMs are running quite nice.

At the office on my testing grounds I started building a test cluster (not HA) which all seems to point the way that Proxmox is going to suit my needs just fine.

So now I am just looking for a little advice. I intend to pull up three external Servers. Mixed KVM and OpenVZ. Not HA. I do not intend to set up a Storrage Server. I was playing with the Round Robin Idea ... as this will not be a nightly thing, rather a on the WE thing, in other words I will still be running nightly backups of important data, but the KVM Snapshot / Backup ist rather a convenience thing.

I will be using Intel® Core™ i7-4770 CPUs. Having 32 GB Ram on each machine.

I was planning on using two SATA Drives in each machine running Software Raid with Level 1 ... but I just got done reading another thread and this seems like a horrible Idea for Proxmox.

I was toying with the Idea to setup an OpenVPN Net and running a forth Node here at my office. The advantage being that I could set up here and then "deploy" the KVM, but as this is nothing I will be doing daily or weekly for that mather I se no advantage.

Last Info you would need is what in hell am I planing on running on this setup. Well it is mainly KVMs running some variation of Debian inside (or OpenVZ) of which more than 50% are running some version of Tomcat with some testing or staging funktion for out Programmers. The other 50% are either running a few small websites for customers or spezialised applications inside tomcats for customers.

For anything running way higher loads I still plan to set up a single separate server as I have done in the past. The above is mainly there to relive me of the VM Ware ESXi pains I am having or to suit low traffic customers pürojects not justifying renting a whole server.

I hope I am not boring anyone, will appreciate any hints or constructive comments.

 

[mir]

For a cluster I would recommend network bonding between the nodes. 2 x 1 Gbps minimum.
Disk: Go for RAID10. Fastest read/write and highest IOPS of all raids.

If your budget can afford it add a hardware RAID (LSI 2008 is a god choice)

 

[blackpaw]

I was playing with the Round Robin Idea

Pardon my ignorance, but what do you mean by that?

 

[steffen]

Pardon my ignorance, but what do you mean by that?

http://whatis.techtarget.com/definition/round-robin

In my case meaning "taking turns" ... so that Machine A does its backup onto Machine B. Machine B in turn does its backup on Machine C. And yes, Machine C would then in turn do its backup onto Machine A.

Engl. "round robin" (1730) = dt. "Petition", "Denkschrift", "Rundbrief" geht auf die Verballhornung von frz. "ruban rond" = dt. "rundes Band" zurück (vgl. engl. "ribbon"). Ursprünglich war es ein Verfahren französischer Bauern im 17., 18. Jh. Um bei Petitionen keinen Wortführer erkennen zu lassen unterschrieb man im Kreis, so dass nicht erkennbar war, wer als erstes unterschrieben hatte. Im englischen wurde aus frz. "ruban rond", engl. "round ruban", das dann zu "round robin" verballhornt wurde.

 

[blackpaw]

Ah, that makes sense in that context.

Are you not planning on using a distributed filesystem?

 

[steffen]

Thank you "mir" for the fast reply. As these are machines which will not really produce money themselves and surely could be handled otherwise (with more work on my part as well as other "standing around" hardware, etc. I am somewhat tied to keeping it low profile. But let's see if I can get this calculated to see if it fits better that way. I worked out two scenariors:

Scenario Nr. 1 we'll call Spartacus:

I will get three machines (http://www.hetzner.de/hosting/produkte_rootserver/ex40), costing me 49 Euro a month (plus 49 Euro onetime fee). With this I will need a so called "Flexi Pack" to get a subnet, costing 15 Euro a Month plus 16 Euros for the 14 usable IPs (extra). Considering I am getting Proxmox lisenced choosing Basic, that would add another 16,58 Euro. Making it 96,58 per machine.

Totalling:

147 Euro one time fee.
289,74 Euro monthly fee.


In the End I get roughly 13 KVM machines. (one IP for Proxmox host, one IP for stearing the subnet as I do not get Mac Adresses for every IP). Based on the IPs. However as I am not having KVMs running without any juice and I am trying to divide them evenly this would only give me 7 KVMs (Intel® Core™ i7-4770 = 4 cores = 8 threads, leaving one theoretically for the Host) with each KVM having 4 GB of RAM.

The drive is a 2 TB drive, calculating with 1800 GB free space this would give each KVM 257 GB, which still needs to be split depending on how many Backups I want to keep of each machine. So lets not be greedy and say 85 GB size of the KVM.

So in theory we would have one thread, 4 GB Ram and 85 GB sitting on a Software Raid, saving at night to another machine for 13,80 Euro per KVM giving me 21 KVM machines. (Disregarding the one time fee).

Scenario Nr. 2 we'll call Doughboy:

Here I would get two machines (http://www.hetzner.de/hosting/produkte_rootserver/px90), costing me 99 Euro a month (plus 99 Euro onetime fee). With this I will need a so called "Flexi Pack" to get a subnet, costing 15 Euro a Month plus 16 Euros for the 14 usable IPs (extra). Considering I am getting Proxmox lisenced choosing Basic, that would add another 16,58 Euro.

Totalling:

198 Euro one time fee.
293,16 Euro monthly fee.

Here as well I get roughly 13 KVM machines based on the IPs. However having a different CPU I get 11 KVMs per machine using the theory used above (Intel® Xeon® E5-1650 v2). Would give each KVM roughly 5,5 GB Ram.

You guessed it, the drive is a 2 TB drive as well, calculating with 1800 GB free space this would give each KVM 163 GB, which still needs to be split depending on how many Backups I want to keep of each machine. Doing it like above this would be 54 GB size of the KVM.

Here we would have one thread, 5,5 GB Ram, Software Raid and saving at night to another machine for 13,30 Euros per KVM giving me 22 KVM machines. (Disregarding the one time fee).

Scenario Nr. 3 we'll call Spartacus on mount Vesuv:

This will be exactly the same as Secenario Nr.1 but we are adding a hardware Raid as well as two more drives and making it a raid 10 with 4 TB in the End. This will add to the monthly fee 45 Euros monthly per machine.

Totalling:

147 Euro one time fee.
424,74 Euro monthly fee.

Having faster disk access this brings me to one thread, 4 GB Ram and 170 GB sitting on a Hardware Raid, saving at night to another machine for 20,23 Euro per KVM giving me 21 KVM machines. (Disregarding the one time fee).

Scenario Nr. 4 we'll call Doughboy in New Mexico:

This will be exactly the same as Secenario Nr.2 but we are adding a hardware Raid as well as two more drives and making it a raid 10 with 4 TB in the End. This will add to the monthly fee 45 Euros monthly per machine.

Totalling:

198 Euro one time fee.
383,16 Euro monthly fee.

Having faster disk access this brings me to one thread, 5,5 GB Ram and 108 GB sitting on a Hardware Raid, saving at night to another machine for 17,42 Euro per KVM giving me 22 KVM machines. (Disregarding the one time fee).

For a cluster I would recommend network bonding between the nodes. 2 x 1 Gbps minimum.
Disk: Go for RAID10. Fastest read/write and highest IOPS of all raids.

If your budget can afford it add a hardware RAID (LSI 2008 is a god choice)

I hope I am not boring anyone to death with this. The Raid Controller being used is a 9260-4i if all my Information is right. If that controller is fine then it looks like Scenario 3 or 4 to me. I can live with 100 GB drive space per machine. The extra Ram in Scenario 4 could be nice for some Tomacats running inside, but they would not be the make or break for this case.

Any comments, suggestions, etc. are again more than welcome.

 

[steffen]

Ah, that makes sense in that context.

Are you not planning on using a distributed filesystem?

To get a bit more detailed on what is running around in my head I added a couple of scenarios up top to the thread. Having a look at that you will notice that I am sitting at an external Hoster and am counting on every cent. So having a distributed filesystem is really nice, sure, but not in my price range in this case as it would require an external, expensive machine with loads of drives.

As I am not running a HA Cluster I do not really see the use, other than tremndous speed up if I ever want to switch one KVM from one to antoher machine. But honestly I am not planning on switching them around a lot, in fact, I would be just fine if they stayed where I put them for years to come.

 

[steffen]

*Bump* ... any other takers? Comments?