Zero-shared-storage Cluster: Possible?

[athompso]

It looks like it ought to be possible to build a PVE cluster without using shared storage, but I'm seeing conflicting information here and in the wiki.

Is it possible to build a cluster without some sort of dedicated iSCSI or NFS storage? Having just one node export the Quorum disk seems to defeat the purpose of HA!

What I had in mind was to run a 4-node cluster using Sheepdog and IPMI fencing, but I currently have no separate shared storage. While I'm sure I could export a tiny Quorum disk off a Raspberry Pi or something equally trivial (and easily replaceable), that seems inelegant.

In the Wiki page http://pve.proxmox.com/wiki/Two-Nodes_High_Availability_Cluster, I see near the top

"Although in the case of two-node clusters it is recommended to use a third, shared quorum disk partition, Proxmox allows to build the cluster without it. Let's see how."

, nowhere in the text of that page do I see any way to build without a shared quorum disk! The "workaround" appears to be using a VM inside PVE to export the shared quorum disk - am I understanding this correctly?
Does the quorum disk need to be high-performance?

On a related note, the Sheepdog page mentions that the metadata should be kept on a highly-available RAID volume; why? If I put sheepdog metadata on the OS boot volume and that disk dies, I have to reinstall PVE on a new disk and that sheepdog member is permanently dead anyway... if both the OS and a data disk die simultaneously, same result, and I rebuild. Am I missing something?

Thanks,
-Adam Thompson
athompso@athompso.net

 

[e100]

Quorum disk is only needed if you have fewer than three nodes.
Yes you can cluster with no shared storage.
The 'cluster' in Proxmox simply provides a way to allow any node to configure all nodes. Other solutions have a 'master' server where you have to configure things from, in Proxmox all nodes are a master.

Someone else will need to answer about sheepdog since I know little about it.

 

[athompso]

Quorum disk is only needed if you have fewer than three nodes.
Yes you can cluster with no shared storage.
The 'cluster' in Proxmox simply provides a way to allow any node to configure all nodes.

Sorry, I should have been more precise. I want an HA cluster, which according to High Availability Cluster requires "Shared storage (SAN or NAS/NFS for Virtual Disk Image Store for HA KVM)". I believe Sheepdog qualifies as "shared storage", as would ceph, gluster, etc. (of which sheepdog appears to be the only one with comprehensible installation instructions).

However, I just noticed that the HA page links to Proxmox VE 2.0 Cluster, which says "If you are interested in High Availability also, for reliable quorum you must have at least 3 active nodes at all times, or use a qdisk as seen in Two-Node_High_Availability_Cluster". Too bad the two-node wiki page isn't a bit clearer. (In fact, it's too bad the whole wiki is so... scattered. OTOH, I'm sure a good chunk of VMware's pricing goes to technical documentation, so I'm getting what I [am about to] pay for.)

I guess I've now answered my own question, thanks!
-Adam

 

[MimCom]

Based on what I learned from the videos I linked to in this thread, I believe ceph has the architecture to pull this off. The FS turned out to be far more difficult to fully distribute than anticipated, but the other bits are working quite nicely. Keep an eye on this one.

 

[spirit]

Hi,
quorum disk is only need if you a 2 nodes cluster.

So for sheepdog, simply build a 3 nodes cluster. (sheepdog use corosync and need quorum too)