Advice on Shared Storage for Proxmox Cluster

[tom]

yes, I heard a lot of good things about nexenta - personally I never tried it.

they promote the open source community edition as "Unlimited, FREE version for up to 18TB of storage" but then saying that its not allowed to use in production - this is quite uncommon and I am pretty sure a lot did not get this and run in production with this version.

 

[rugby]

That language about it not being for production is new, I've never seen it before you pointed it out. I have a buddy who's heavily into it and has done a ton of setups, I'll ping him and see if he's noticed it as well.

 

[charnov]

We priced out what we wanted to do with Nexenta and while for a SAN it was a steal (I worked a project that used an IBM SAN from 8 years ago that was $2.5 million for 20TB) it was still over $7000 in licensing to get a HA cluster going in addition to the hardware. It came close to the LeftHand SAN we submitted to management that was rejected as too expensive (although performance on the Nexenta was much better) so we rolled our own with Napp-It and it has been chugging along ever since.

 

[rugby]

charnov, how is the performance on Napp-it vs. Nexenta?

 

[charnov]

They are both based on the Illumos/OpenIndiana (mostly... it's getting there) and use ZFS so it's about the same. Nexenta is MUCH easier to work with if the Solaris way is new to you (and it is very different from Linux). Nexenta also has superb support which is worth every penny. We simply did not have the budget for it. Try out Napp-IT. It runs on the same hardware so you may always switch. If it is a nicely behaved IO card (like the LSI cards with IT firmware) and you set it up as JBOD, you should... should be able to go from one distro to the other and do an attach of the ZFS pool if you change your mind. It can be tricky, though.

The great thing with Napp-IT is that you can get much deeper into the system and have fine grained control.... that's also the bad thing. Nexenta makes it super easy and reliable to manage.

 

[gruppe5]

Hi Tom,

tnx for ur advice in licensing for Nexenta CE. You are absolutely right, but it was new to me, also. OK, then we could walk with OpenIndiana and napp-IT. The Interface is not that professional, but it works perfectly, too.

We also watch out your tests with Intel Modular Server. It sounds perfectly to build a stable HA Cluster. So if we need 3 Modules/Servers to build a minimum Cluster, such as compatible must be the hardware equipment? BTW: Could we build the Master with a minimum RAM and small Processor while the 2 Nodes with high performance (dual 6 core and at least 48 Gig of RAM each) equipment?

It would be nice to read more about your tests and experiences with the Intel Modular Server solution.

Greetings
Udo

 

[charnov]

The reason why you need three for the HA cluster is so that you don't get a split-brain issue during cut over, i.e. one goes down and the other nodes take over and then it comes back up before the transition finishes and there is no way for the other node to know => multiple running copies of VMs, corrupted shared storage, and other horrors. With three, you always have a quorum. Also, that third one may be the hosting server, too, so they should all be similar or at least able to handle the load. You can do it with two, but it's not fun: http://pve.proxmox.com/wiki/Two-Node_High_Availability_Cluster

I believe the Intel Modular Server is certified for HA by Proxmox. It's a beast of a machine. http://pve.proxmox.com/wiki/Intel_Modular_Server_HA

 

[rugby]

charnov,

Thanks for the reply. I've been wrestling with Nexenta the past few days and have discovered a few bugs in the UI that prohibit things from moving forward. I think I'll give OpenIndiana a try and see if it goes smoother. I don't have the budget to pick up the licensing for Nexenta anyways, and the issues I've had cannot occur in a production environment without major stability occurrences.