What is the best storage solution for our needs?

[willieb1172]

Hi,

It's everything but written in stone that we've decided to use Proxmox for our new business venture. I've spent quite some time researching Proxmox, testing within VMWare Fusion 5, and have researched storage solutions to an extent but haven't tested. With all of the information available on this forum and the Proxmox wiki, I'm still not sure what's best for our particular deployment, local, distributed, shared, etc. I will try to give you an idea of our requirements and if you'd like please read it and make some suggestions based on your experience. This will be a legit business with paying customers so it's crucial we do this right. We will have Proxmox paid support. We want to consider keeping startup costs to a minimum, allowing us to grow as we go so to speak. This could mean starting out with local storage or distributed then migrating to shared storage later if that makes more sense for a larger deployment. The business will be cloud based storage with steaming capabilities, audio and video with on the fly transcoding. So we have high storage and CPU needs.

For a server we will start out with 1 with local storage. All will be containers (OpenVZ) with Debian 7.

The first server will have (12) 4TB SATA drives in a hardware RAID 5 configuration. It has 4 Nodes, each of which will have 12 Cores (2 x 6 Core L5639 2.13GHz) and 24GB DDR3 RAM.

This gives 48 Cores, 96GB RAM, 44TB of storage.

Long term (multiple servers) the requirements are:

• Container failover to another server node if a node fails (High Availability)
• Full data redundancy for containers and customer data (RAID 5 and Duplication of customer data)
• Live container migration to other nodes for manually load balancing nodes (Can Proxmox do auto load-based migrations?)
• Container backup off-site with automation (this may not be needed with HA)
• Customer data mirror (backup) with automation
• Hardware RAID 5 or 6
• I'm sure there's more, that's all I can think of off the top of my head.

We need to consider our long term goals before we begin, so we will understand the migration path from 1 server to the requirements above. For example we may start out with local storage for the 4 nodes in 1 server, but what about when we buy another server? Would it be best to do some type of distributed shared solution (ceph?) for local storage, or a 10Gbit NAS? If we do the 10Gbit NAS we must consider the migration path from local to NAS. Really I'd like to keep it simple if it makes sense.

So based on the info above, do you guys have any suggestions on which path to take short term and long term? Thanks!

 

[Raymond Burns]

I would recommend NAS. ZFS NAS to be exact. I have one in a production environment with 4-node cluster, and it is handling the demands pretty well.
Additionally, a shared NAS has all the benefits of Proxmox as far as moving storage, live migration, live backups, failover, etc.; and you have the added capability of having a failover NAS setup. So essentially, a basic 5 server setup would be ideal (2 storage servers, 3 Proxmox servers), all fully redundant.
I have yet to find a Hardware RAID that can keep up with my VM demands, and I was using PERC-7 cards. Moving to ZFS, I don't have the I/O wait issue.
HOWEVER, there is no IO Load shown for containers over NAS. I don't know if there is a roadmap for this, but as of now, this is the biggest bummer we have with this setup.

 

[willieb1172]

I would recommend NAS. ZFS NAS to be exact. I have one in a production environment with 4-node cluster, and it is handling the demands pretty well.
Additionally, a shared NAS has all the benefits of Proxmox as far as moving storage, live migration, live backups, failover, etc.; and you have the added capability of having a failover NAS setup. So essentially, a basic 5 server setup would be ideal (2 storage servers, 3 Proxmox servers), all fully redundant.
I have yet to find a Hardware RAID that can keep up with my VM demands, and I was using PERC-7 cards. Moving to ZFS, I don't have the I/O wait issue.
HOWEVER, there is no IO Load shown for containers over NAS. I don't know if there is a roadmap for this, but as of now, this is the biggest bummer we have with this setup.

So per your suggestion it looks like we could use (2) ZFS NAS storage servers connected to a 4 node Proxmox server. That sounds like a great setup. I'm wondering if I should go with an actual NAS, or just a server with a NAS distro such as FreeNAS or unRAID...? Any suggestions for hardware? I'll need at least at least 40TB-50TB per NAS server. I will start re-searching NAS and ZFS.

Thank you very much for your input, it was quite helpful. Anyone else is welcome to chime in as well.

 

[briceburg]

WillieB,

I STRONGLY recommend RAID-10 -- especially if you're running VMs underneath. RAID-5 isn't worth the headache; and worse yet if you're calculating parity with your CPUs and not a dedicated hardware raid controller.

I love ceph; although recommend setting everything up in a test environment first && maybe sticking w/ a traditional NAS/NFS share to start? Then try moving your containers to shared storage.

Auto-load balancing containers sounds pretty awesome but I'd want to define the policies per VM. Not sure it's built in but you could incorporate scripts into your monitoring infrastructure that talks to the proxmox API.

~ Brice

 

[mir]

I would recommend Omnios for server OS with napp-it ontop (web gui) alternative is zfsguru (FreeBSD 9.1 and web gui). The ISCSI provider in Omnios (comstar) is superior to FreeBSD (Istgt). For hardware most seems to agree on SuperMicro with Intel Xeon and by your requirements for storage I would buy a lot of RAM >= 64 GB. Also consider SSD for SIL and LARC2. Don't waist your money on Hardware RAID since ZFS prefers direct access to the disks. Also as mentioned: If you want speed go for RAID 10 and use a lot of mirrored vdevs for your RAID 10 pools for you 2 pools with 10 mirrored vdevs (2x2-3TB) each. BTW Western Digital has some excellent 2 TB disks which is only containing one plater -> excellent access time and higher throughput.

As a sidenote: A storage plugin for ZFS is soon to enter your installations which will give you support for live snapshots and linked clones in proxmox on ZFS storage servers.

 

[e100]

12 Drives in RAID5 is, IMHO, a disaster waiting to happen.
I would either do RAID6 or two RAID5 volumes of six drives each.
12 disks where if two fail you a re SOL is not the sort of reliability you desire. Also consider that when you have 12 disks it is likely that one will fail shortly after the first fails. With 4TB disks it will take a long time to rebuild the first failed disk making the window where you are vulnerable to a 2nd failure quite long.

If you are doing RAID5 or 6 you want a hardware RAID card, the parity will kill you, especially when a disk fails and it needs rebuilt.
I have lots of Areca 1880 and a couple 1882 RAID cards, the 1882 (dual core) can write RAID5 to six SSD disks faster than my DRBD over 10G Infiniband setup allows.

Someone suggested ZFS/NAS setup, no experience, I mostly use DRBD and mostly only KVM machines.
I see you are planning mostly openvz, giant filesystems with zillions of files take forever to fsck.
No idea if this applies to ZFS or not, I have a couple of machines with 3TB ext4 filesystems on old slow Perc card and slow disks, I hate it when they do a fsck. Some might argue that fsck is not necessary with journaled filesystems, I disagree, everyone has their own opinions tho.

 

[mir]

Someone suggested ZFS/NAS setup, no experience, I mostly use DRBD and mostly only KVM machines.
I see you are planning mostly openvz, giant filesystems with zillions of files take forever to fsck.

Since ZFS is a copy-on-write file system with checksums on every block there is no need for fsck. What talks against openvz is the lack of live snapshots and linked clones in the current implementation. Ploop filesystem for openvz provides these features but ploop is saidly not currently on the roadmap for proxmox.

 

[wahmed]

I fully support suggestions made above by mir about OmniOS/ZFS. ZFS in OmniOS+Nap-It is an excellent enterprise quality product which will give very high level of redundancy at very minimal cost. I will tell you one thing though, Hardware RAID is old school. Many people will go crazy about this comment, i know But, with ZFS you have 0 need to spend $$ on hardwares. I have torture tested ZFS setup and it can withstand just about any failure. ZFS offers software level redundancy.
Here is an example, you have two storage servers. One server with 24 HDD in dedicated hardware RAID 6/10 whatever your top dollar could get. 2nd server without any hardware raid but with ZFS. When both servers goes up in smoke, with ZFS you take out all your HDD and put them into a "completely different" machine with OmniOS+NapIt installed, auto import your HDD group and you are up an running. In this scenario with Server #1, you better have an identical RAID card on hand at all times. Big disadvantage with ZFS, "you need lot of RAM". "YOU NEED LOT OF RAM"!! Period!

My personal preference is CEPH. Proxmox does not support OpenVZ CTs on CEPH RBD Storage yet. CEPH is distributed file system. Imagine hardware raid but on Server scale. Instead of one of your HDD going down, imagine one of your Storage node with 24 HDD going down. But your cluster still keeps running because you have other 1 or 2 storage node in the cluster. With CEPH Cluster to increase your overall storage space, just add another node with HDD. With ZFS i yet do not know if multiple nodes can work together as one. CEPH also does not require any hardware RAID system, since the nodes themselves acts as RAID.

As it already has been pointed out, if you are doing it to serve your paid customer, you must setup at least 2 node storage and 3 node proxmox. Hope this gives you some more idea.

 

[RobFantini]

"with ZFS you take out all your HDD and put them into a "completely different" machine with OmniOS+NapIt installed"

I've done that using debian bsd zfs to freenas to debian linux zfs . At home we've been using the same old 8TB array for 4 years. AFAIK there is no need to use the same operating system. The main issue I've had is forgetting to do an export before moving the disks or replacing the operating system. I think it is very hard to kill zfs, exspecially using raidz2 or z3 .

It has been a few years since I researched zfs ,and the reason we are not using zfs for production storage is that it was [ is? ] a single point of failure. Can zfs be set up so that two or more systems can have high available storage? If so how long has that system been in production ?

For now we use DRBD. Have done so since 2007 . Split brain recovery is always scary , especially with a Primary/Primary set up. For the future sheepdog/glusterfs/ceph etc look good.

 

[wahmed]

Can zfs be set up so that two or more systems can have high available storage? If so how long has that system been in production ?

For Napp-It there are commercially available options such as described here http://napp-it.org/extensions/index_en.html. But nothing point-n-click method as far as i know.

For FreeNAS, the only commercially available option is to buy fully assembled Storage appliances from their site http://web.ixsystems.com/. They do not seem to have any option to purchase license for FreeNAS to setup high availability. There are some cases people had been able to setup HA between FreeNAS nodes using Heartbeat, coro, but no easy task. Not enough support. For sure not an option for paid service platform.

 

[Raymond Burns]

NAS4Free has HAST as far as high availability.
http://wiki.nas4free.org/doku.php

Commercially, we are looking into RFS-1 for OmniOS http://www.high-availability.com/omnios-high-availability-cluster/
It is a single point of failure, but with ZFS sends regularly, you have a great resource to fall back on, and, like you say, it's hard to kill ZFS.

I'm not aware of anything that runs as fast as ZFS. I've vaguely looked at some IOPs benchmarks of others, but the ZFS always seem to stand out.

 

[mir]

Commercially, we are looking into RFS-1 for OmniOS http://www.high-availability.com/omnios-high-availability-cluster/

How deep have you looked in to this?
For pricing I have heard rumors that we are talking 4500-5000$ a year. Can you verify that?