Shared storage options for small HA cluster?

[athompso]

I see that PVE 3.1 now has support for many types of shared storage. As of this moment, I count:

• iSCSI (both mounted and direct LUN)
• NFS
• Sheepdog
• GlusterFS (somehow different from NFS?)
• Ceph/RBD

I also see that as of PVE 3.1, all of those *except* Sheepdog are available in the GUI. Yet sheepdog appears to be the simplest, most redundant (maybe only at scale?), and lowest-overhead option of the bunch. Is there something wrong with using sheepdog that I don't know about? (I don't care about OpenVZ containers at this point, only KVM.) I'm need to provision a 4-to-8 node cluster to begin with, all identical, presumably each one would be a sheepdog node. I'm not providing 99.999% uptime, but data loss would be very bad.

Is it possible to run glusterfs or ceph *on* the PVE nodes, like sheepdog?

(It should be possible to run containers on NFS mounted from a VM stored in sheepdog, if I suddenly decide I need containers. Ugly, but possible.)

Thanks,
-Adam Thompson
athompso@athompso.net

 

[athompso]

FYI: I know that sheepdog is technically experimental, but: 1) Proxmox have been calling it experimental since 2010; 2) stable trees now exist with patches; 3) providing paid support for sheepdog is kind of like getting paid support for KVM itself - IMHO that's what Proxmox is here for.

FWIW, I'm thinking that better sheepdog integration could be a huge value-add for Proxmox, at least to the mostly-KVM users (which IIRC according to a poll was ~50% of all users) in the small-to-medium size cluster market. If I can afford more than 16 nodes, I should be able to afford a dedicated iSCSI/FC/NFS SAN, too. However, for those of us doing it on the cheap (e.g. used Dell C6100s) sheepdog is a VERY attractive option!
-Adam

 

[dietmar]

FYI: I know that sheepdog is technically experimental, but: 1) Proxmox have been calling it experimental since 2010;

IMHO the code is simply not stable enough.

2) stable trees now exist with patches;

Those stable releases are maintained for only 3 months?!

3) providing paid support for sheepdog is kind of like getting paid support for KVM itself - IMHO that's what Proxmox is here for.

Sure, and you want that we support your whole storage infrastructure for free ...

FWIW, I'm thinking that better sheepdog integration could be a huge value-add for Proxmox

What do you miss? It is already fully integrated - we only disabled the GUI for now.

 

[athompso]

IMHO the code is simply not stable enough.

How unstable is it? Random reboots? Random lockups? Random data corruption? Random data loss? If so, how often? I can live with the occasional hiccup, if it merely requires a reboot or minor manual intervention. I probably can't live with random data loss or corruption.

Those stable releases are maintained for only 3 months?!

Quoting Hitoshi from earlier this month, "We will try to keep the branches updated until corresponding branchs of QEMU become outdated." If that's been changed to 3 months, then I do see a problem. If they are in fact tracking qemu's -stable, then I don't see the problem. Most recent info I can find is here and since 0.6, there is now a stable tree.

Sure, and you want that we support your whole storage infrastructure for free ...

That would be nice, yes .
Seriously, it does not have to be free. This could be an option to increase revenue as an add-on package - not add-on functionality (that would be problematic), but add-on support. Sheepdog does not appear to have a commercial sponsor right now - that could be Proxmox. You already have a few distinctive features (nice GUI, OpenVZ) - easy-to-set-up *redundant* shared storage is the missing piece for small-to-medium installations.

I think I would likely be happy to pay 50%-100% more if shared-redundant-storage-on-PVE-node was available as a supported option. (+50% at each level of support seems reasonable for an important increase in supported features as long as it was an optional add-on.)

I do note that making ceph setup seamless and able to run on the PVE nodes directly would also address this problem. Right now I look at ceph and I get scared away because it looks massive, complex, difficult, fragile and over-engineered for <16 nodes. Also, the ceph page on the PVE wiki tells us not to run ceph nodes on the PVE nodes, with no explanation why. I assume it's because of CPU & RAM load... but what if I have CPU and RAM to spare, and/or would rather buy some extra nodes than set up a SAN?

What do you miss? It is already fully integrated - we only disabled the GUI for now.

The GUI, firstly. From what you've said, I assume the GUI is disabled due to your concerns about stability?
Support, secondly. The packages are there, they appear trivial to enable and configure, but every time someone mentions using sheepdog, your reply seems/feels like a knee-jerk response. We can't tell how much testing Proxmox is doing on sheepdog, and you aren't telling us. Everyone on the forum who's talked about it seems to generally agree that it doesn't perform amazingly well, but I don't see *anyone* mentioning severe problems with it in the last 12 months or so. (Caveat: I'm not subscribed to the sheepdog mailing list yet. Browsing the archives the last several months doesn't reveal any showstopper bugs.)

Thank you for the reply so far,
-Adam

 

[spirit]

Hi,
I'm testing sheepdog since 1 year now. (I have also made the sheepdog plugin code for proxmox).

Only part in gui disabled is the storage add. (you just need to edit storage.cfg), then you can create image/snapshot/clone/... with gui.

About stability, sheepdog himself is stable (no corruption), the only problem I have had was multicast network problem due to a cisco bug and bridges.

The only problem I was seeing since 1 year was the data format change. (changed 2 times), and this could break the cluster.
Hopefully I think it shouldnt change anymore for a while, and stable branches will be maintain.