New Setup 4 nodes

[geormanth]

Hi,

I am prepare to setup a 4 nodes dell poweredge 6100.
My plan is to make a 4x proxmox cluster including ceph storage cluster.
Setup per node:
2 X quadcore xeon
32GB RAM
2X gb lan
2X 2terra HDD (one node has 2X3T)
1X 240GB SSD

My questions are:
1. Can I use the same ssd for proxmox and ceph journal?
2. Can I use 3 nodes with the same size storage (2x 2T) and one with different size (2x3T) to use them as ceph disks?
3. What opinion do you have for the following ssd
SSD KINGSTON SUV400S37/240G SSDNOW UV400 240GB

Any suggestion will be grateful.

Thanks in advance for this useful forum.

 

[fireon]

Looks likes great. But you will have not much fun with your disk 3 Disk (they are SATA or SAS15?) When they are SATA i think your cluster will be very very slow... depending on your VMs they are running. So what? Linux KVM, Linux LXC or Windows? How many users should work on it. Will be there an Linux/Windows Terminalserver? And only 2Gb lan is also not enough. Use 10GB with only SSDs or 15kSAS disk. And yes... maybe 8 and more SATAdisks on Raid10 should we also running fine per node.

So it is really really depending on things that should running on.

 

[geormanth]

The l machine has 12 bays for sata disks 3 to each node, I don't know yet if it has separate slot to put OS ssd inside.
I already have the sata disks.
I don't have the ssd's.
My needs are:
5-10 users
4-5x LXC
3x Linux KVM
4x Windows KVM

About raid10 scenario you mention, I think ceph and raid is not so good combination, but I never testit.
Until now I used soft raid and GlusterFS only for backups and templates.

 

[LnxBil]

A good "go to list" for Ceph Journal Device SSDs is this list:
http://www.sebastien-han.fr/blog/20...-if-your-ssd-is-suitable-as-a-journal-device/

Your SSD is not listed there, but that only means that no one tried it (or did not report it there). Bad devices are also listed there, so go and buy a good one.

As @fireon already explained, you will not have a fast system. I tried Ceph with 8 SATA Disks per node and it was very, very slow. Do not use Ceph on only a few extremly-slow SATA disks. If you use disks, use SAS 10k or 15k as @fireon already described.

Why not go with local storage and no HA? A single Proxmox Host with local disks (or even ZFS) will yield much more throughput. With ZFS you can setup a kind of standby Proxmox VE server which is e.g. 15min behind with asynchronous replication.

 

[alexskysilk]

Looks likes great. But you will have not much fun with your disk 3 Disk (they are SATA or SAS15?) When they are SATA i think your cluster will be very very slow

this.

you will not be happy with the performance of a ceph cluster with this few OSDs. also, you'll lose 1 disk slot for boot disk reducing your OSD count to 2/node- if you're ok with having a non redundant boot device. You'll be far better off with an external storage SAN.

I don't know yet if it has separate slot to put OS ssd inside.

It does not.

 

[geormanth]

So, please be advice which will be a good solution to manage all the disks, replications etc?
I don't have needs for ha, I prefer load balancing instead with external san for extra backup.
The problem is the disks has total 18T capacity plus 1T ssd and the external storage is only 5T (4X2T-Raid5).

 

[alexskysilk]

Not enough information for a recommendation.

Lets ignore the hardware you have for a minute. What are you trying to accomplish? what is your use case? What are your painpoints, eg Whats the slowest acceptable performance for a given function? how much failure is acceptable (eg, how long is it ok to be down? whats the consequences of losing a virtual machine completely, etc.)

 

[geormanth]

Not enough information for a recommendation.

Lets ignore the hardware you have for a minute. What are you trying to accomplish? what is your use case? What are your painpoints, eg Whats the slowest acceptable performance for a given function? how much failure is acceptable (eg, how long is it ok to be down? whats the consequences of losing a virtual machine completely, etc.)

Until now I didnt loose anything at all, cause I had all my data in two mirror servers with hardware raid1 disks and external nas as third backup all of them replicate with rsync.
I want to expand my cluster from 2 machines to 4 cause my needs increasing for cpu and storage.
So for now I have
2 RIP Servers (print servers) win2008 (KVM)
2-3 imposition servers winxp (KVM)
2 asterix elastix (KVM)
2 fileservers most of files are big (DTP Services) so I don't want to loose anything at all. (LXC)
2 access control (KVM)
2 ms access 2010 web share (KVM)
2 pfsense (KVM)
2 erp (LXC)
2 ns (LXC)
2 zimbra servers. (LXC)
plus 2-3 winxp (KVM) for different proposes.


As you can see everything is double and sometimes triple, I thing for my case is better to have load balancing instead of HA.
Now I want to duplicate the windows machines and increase capacity of file servers and if possible to have a better solution for the replication instead of rsync.

 

[alexskysilk]

In that case, its a fairly simple proposition.

each node gets its own storage; I'd put 2 disks per node, ZFS mirror (seperate boot disk is optional.) Make sure to split your VMs/CTs in such a way that no 2 mirror instances are on the same node, and that each node has at least 25% of is resources (CPU, RAM, disk space) free.

The exception is your fileservers. you really want each be on their own device with their own disks; these can be big spinning rust and you can use parity raid- and they can double as backup space for your hypervisor stores.

 

[geormanth]

In that case, its a fairly simple proposition.

each node gets its own storage; I'd put 2 disks per node, ZFS mirror (seperate boot disk is optional.) Make sure to split your VMs/CTs in such a way that no 2 mirror instances are on the same node, and that each node has at least 25% of is resources (CPU, RAM, disk space) free.

The exception is your fileservers. you really want each be on their own device with their own disks; these can be big spinning rust and you can use parity raid- and they can double as backup space for your hypervisor stores.

So what do you thing each node has 2 hdd and one ssd for os and vm's.
I don't want the file servers to be on their own device but lxc and manage hdd's from all nodes.
About your last sentence "these can be big spinning rust and you can use parity raid- and they can double as backup space for your hypervisor stores" are youkind enough to tell me more details, I don't understand well enough what do you mean.

 

[alexskysilk]

So what do you thing each node has 2 hdd and one ssd for os and vm's.

I think we're at cross purposes here. the HDDs I'm talking about are for virtual machine use. you dont want those to sit on a single device, as it exposes you to total failure. a mirror is very cheap insurance. No point in the SSD in this context, unless you're deploying 2 of them instead of the HDDs.

I don't want the file servers to be on their own device but lxc and manage hdd's from all nodes.

I dont know or understand why. More importantly, there is no simple way of accomplishing this- and complexity is rarely a desired solution. in short- this isnt doable. If your sole reason is that you have your C6100 and you're going to use it come hell or highwater- I cant stop you, but its the wrong shaped tool for your problem.

are youkind enough to tell me more details, I don't understand well enough what do you mean.

a server with a bunch of 3.5" disks. you could possibly attach an external storage chassis to one of your C6100 nodes with an external HBA for this purpose, but unless you already have the enclosure you might as well get the whole server.

 

[PigLover]

Regardless of the combination of ssd/hdd, etc, I don't believe you'll find a satisfying solution with a single C6100 with 12 3.5" drives (3 per node) and 1gbe networking. I know the C6100 well and I just don't think you'll get there.

The only reasonable way to get decent performance from Ceph with a small number of OSDs is with all OSDs on SSD.

Compromising almost all sense of best practices, the closest you'll come: three nodes (of four) with a small HDD or SSD for boot and two SSD for OSDs - giving you a three node cluster with 6x OSD for VMs. Then take the third node and build it out for file storage only - use all three drives for large-ish HDDs, do a RaidZ across all three drives and boot from the array (the Proxmox installer will support this). Publish the remaining disk space from this node as an NFS share and use it for ISOs, backups, and general file storage as needed by your VMs.

With 1gbe networking supporting Ceph you'll see a fair amount of IOWait. That's life. If you use the default 3x replication you'll always have a local copy to read from but writes will suffer a bit. Any one of the three nodes with Ceph can be offline and you should still have reliable disk service (though the file server node is a single point of failure for anything that requires file IO from it).

 

[geormanth]

I want to be completely independent from hardware, not use any scsi cards for disks just normal sata disks, not care if ssd for proxmox and VMs fails just clone and make a new one, not care if I loose one node or two nodes.
We have from proxmox backup solution for VM's and with clonezilla or something similar we can have the node to bring it back in any failure case.
Until now with 2 identical VM's per case and proxmox (clonezilla image for backup nodes) solved my problem and is time to expand this 4 nodes plus 1 nas.
I understand that Ceph is not for my hardware, but how I can manage all the storage centraly and be completely independent from hardware?