Recommendations for 3-node cluster on existing hardware

[IT Manager BIS]

Hello,

We have 3 identical servers, with the following specs:
Asus RS300-E8-PS4
Intel Xeon E3-1320v3
16Gb RAM
4 Enteprise SATA HDs, 2TB, with Hardware RAID LSI Logic / Symbios Logic MegaRAID SAS 2108
3x1Gb Ethernet ports bonded to get more bandwidth

We have been using them as a very simple Proxmox VE 3.x cluster, no distributed storage, and has so far met our needs really well.
We are now looking into upgrading to Proxmox VE 4.1, and we have moved all our load to only 2 nodes, so we can simply reinstall and start from scratch, one node at a time.

I am looking for some recommendations on "best practices", "what would you do", etc.
I have seen some threads about Ceph, DRBD9, ZFS sync, etc... for achieving live migration with just 3 nodes, but I would appreciate some advice so we know which direction to move forward and start researching.
So to all the Proxmox admins here, what would you do with our hardware?
Thanks!

 

[gosha]

Hi!

My cluster - 3 identical servers (HP DL380 G8):



Ceph block stotage: 18 OSD (6 OSD on each server) over 10Gbit/s-network:



Live migration work for qemu VM's manually.
For HA VM's - automatic live migration impossible.
For LXC-containers - migration via stop/start VM only.

 

[IT Manager BIS]

Thanks! I'll dig into Ceph in more detail, it seems.

 

[IT Manager BIS]

Hey, quick question: what's the network connectivity for your cluster?
Our servers only have standard 1Gb Ethernet. I've been using a vSwitch with IntPorts for management, with VLANs on the switch.
How are you handling the Ceph storage traffic?

 

[gosha]

The recipe is simple. IMHO for the greater number of active working VMs in the cluster - the more disk activity, and then more ceph network bandwidth is needed. For 1Gbit/s adapters better use bonding.
In my ceph-network 10Gb adaters used.

 

[alex purser]

Hi!

My cluster - 3 identical servers (HP DL380 G8):

View attachment 3736

Ceph block stotage: 18 OSD (6 OSD on each server) over 10Gbit/s-network:

View attachment 3737

Live migration work for qemu VM's manually.
For HA VM's - automatic live migration impossible.
For LXC-containers - migration via stop/start VM only.

 

[alex purser]

Hi Gosha would you be able to share the components you have used in your DL380 G8 - i intend to use a DL360 G8 and want to get the correct disk controller - i dont thing the P420i will go to HBA mode so i need an alternate HBA card.. also a recomended 10Gbs card would be great also.
Thank you in advance
Alex

 

[damon1]

Hi Alex,
if you want to look at 10GB I found it wasn't as expensive as I thought.
it depends on what 10GB switch you want to use.

I got one of these as they accept both cables (and are cheap at $850 for what you get) SFP+ the weird looking one and standard RJ45.
Prices are in Australian dollars
QNAP QSW-1208-8C 10GbE 12 Port Unmanaged Switch
for my 4 node set up I use one 10GB port for ceph and then the other from Proxmox so I needed 8 ports as a minimum.
I use one of the other ports for a 10GB NAS for Proxmox BACKUP and another to connect to another windows box via 10GB.
Which is a few years old so there are probably better, but it works well.

but for half the price you can get a 1GB and 10GB switch (4 ports)
TP-Link JetStream T1700G-28TQ 24 Port Gigabit Smart Switch w/4 x 10Gb SFP+

then you buy the 10GB cards - like these - very reasonably priced at under $100 per card and most are prepare to haggle. I got 2 for $85.80 delivered.
15581201-B21 HP NC550SFP 586444-001 Dual Port 10GBE PCI-E NETWORK SERVER ADAPTER
or
Dell Intel X540-T2 10GbE Dual Port Server Adapter RJ45 X540T2 K7H46 Full Height

the SFP+ cables are pretty cheap as well - $20 for a 3m or $15 for a 1m cable
SFP-H10GB-CU3M for Cisco, 10GBASE-CU SFP+ DAC, Twinax Copper Cable 3M
SFP+ to SFP+ 1M/2M/3M 10Gbit Optic Cable Direct Attach Cooper Twinaxial Cable
You will need to get rid of your CAT5 cable anyway and move to CAT6a ($10 for 3meters).

I don't know anything about disk controllers and don't use them in our boxes, straight SATA 3 seems to do the job into 1 SSD and 2 HDD for Ceph per box. Remember Ceph wont work with Raid controlled arrays. Not sure if it would make a difference as the 1GB network is going to be the bottleneck anyway. 10GB is much better. These are my disk speeds with the above hardware and cables (cat6A and SPF+)




Ceph runs like a dream.
If a server goes down then the VM's move to a new host quickly and reboot ( about 2 minutes ).
Live migration - no problems and really quick, under a minute.
If you need to take a node down, move the VM's to the other nodes (live) and turn the node OFF, add memory, change cards, clean it whatever you need to do - plug it back in when you are done and start it up. CEPH will re-balance everything (mostly) automatically (takes a while) without any downtime to you.
I am using prox 6.1-5 (latest) and find it pretty bullet proof, obviously some things go wrong which is I why am here, to post a question but I saw yours and thought I might be able to help.

thanks
Damon

 

[alex purser]

Hi Alex,
if you want to look at 10GB I found it wasn't as expensive as I thought.
it depends on what 10GB switch you want to use.

I got one of these as they accept both cables (and are cheap at $850 for what you get) SFP+ the weird looking one and standard RJ45.
Prices are in Australian dollars
QNAP QSW-1208-8C 10GbE 12 Port Unmanaged Switch
for my 4 node set up I use one 10GB port for ceph and then the other from Proxmox so I needed 8 ports as a minimum.
I use one of the other ports for a 10GB NAS for Proxmox BACKUP and another to connect to another windows box via 10GB.
Which is a few years old so there are probably better, but it works well.

but for half the price you can get a 1GB and 10GB switch (4 ports)
TP-Link JetStream T1700G-28TQ 24 Port Gigabit Smart Switch w/4 x 10Gb SFP+

then you buy the 10GB cards - like these - very reasonably priced at under $100 per card and most are prepare to haggle. I got 2 for $85.80 delivered.
15581201-B21 HP NC550SFP 586444-001 Dual Port 10GBE PCI-E NETWORK SERVER ADAPTER
or
Dell Intel X540-T2 10GbE Dual Port Server Adapter RJ45 X540T2 K7H46 Full Height

the SFP+ cables are pretty cheap as well - $20 for a 3m or $15 for a 1m cable
SFP-H10GB-CU3M for Cisco, 10GBASE-CU SFP+ DAC, Twinax Copper Cable 3M
SFP+ to SFP+ 1M/2M/3M 10Gbit Optic Cable Direct Attach Cooper Twinaxial Cable
You will need to get rid of your CAT5 cable anyway and move to CAT6a ($10 for 3meters).

I don't know anything about disk controllers and don't use them in our boxes, straight SATA 3 seems to do the job into 1 SSD and 2 HDD for Ceph per box. Remember Ceph wont work with Raid controlled arrays. Not sure if it would make a difference as the 1GB network is going to be the bottleneck anyway. 10GB is much better. These are my disk speeds with the above hardware and cables (cat6A and SPF+)

View attachment 14287


Ceph runs like a dream.
If a server goes down then the VM's move to a new host quickly and reboot ( about 2 minutes ).
Live migration - no problems and really quick, under a minute.
If you need to take a node down, move the VM's to the other nodes (live) and turn the node OFF, add memory, change cards, clean it whatever you need to do - plug it back in when you are done and start it up. CEPH will re-balance everything (mostly) automatically (takes a while) without any downtime to you.
I am using prox 6.1-5 (latest) and find it pretty bullet proof, obviously some things go wrong which is I why am here, to post a question but I saw yours and thought I might be able to help.

thanks
Damon

thank you Damon i will take a look, appreciate your response here, cheers, alex

 

[jtracy]

Hi Alex,
if you want to look at 10GB I found it wasn't as expensive as I thought.
it depends on what 10GB switch you want to use.

Honestly if you don't plan on growing you can connect each node directly without needing a switch for CEPH. You need multi port 10G or even 25G or 40G cards to make it work. You need N-1 ports on the card so for a 3 node cluster you can get away with a 2 port card. Read this wiki post for more information