[TUTORIAL] Condition VM start on GlusterFS reachability

[esi_y]

The translation in my language is bad on what you are telling me.Can you formulate your question differently? :
"Why didn't you let the VMs simply start at the end of the heal script being oneshot?"

My English was not any better. I basically wonder, if you have now 2 services, one is basically restarted every 10 seconds until it can actually do what it needs to do (start vms) and the other is there to basically facilitate that it the first can start (gluster heal), then why not simply start the vms at the end of the gluster heal?

Arguably, it would also start immediately when it can (not saying that the 10 seconds matter) and it would not let systemd keep starting up every 10 secondes something that for 10 minutes cannot run.

 

[zeroun]

My English was not any better. I basically wonder, if you have now 2 services, one is basically restarted every 10 seconds until it can actually do what it needs to do (start vms) and the other is there to basically facilitate that it the first can start (gluster heal), then why not simply start the vms at the end of the gluster heal?

Arguably, it would also start immediately when it can (not saying that the 10 seconds matter) and it would not let systemd keep starting up every 10 secondes something that for 10 minutes cannot run.

Thanks for explaining.

But it's still difficult. I'm French and you?

Nevertheless, I think I guessed the meaning of your question:
" and the other is there to basically facilitate that it the first can start (gluster heal)"

When a service fails at time t. However, it was active for a short time at time t-1.

If you imagine 2 services with one that tests the gluster and the other that starts the VMs. The second will be called as soon as the first is active.

And since we have the behavior described above. This cannot work.

That's why I use only one service. It returns false (1) if the cluster is not self-healing and does not move on to the next: exit 1

To start the VMs with the tag "gluster"

If I understood your thinking correctly, this is an excellent question.
Thanks for asking it @esi_y . :-D

 

[zeroun]

I was wondering if we could wait for the HA service to start, or when migrating a VM from one node to another with the condition that the self-repair of the GlusterFS cluster is finished...

In short, big and vast development topics...

Or as for having wonderfully integrated Ceph into Proxmox-ve, make available a proxmox-ve flavor with the same quality of integration but for glusterFS this time. For very modest hardware configurations like mine...

 

[esi_y]

I was wondering if we could wait for the HA service to start, or when migrating a VM from one node to another with the condition that the self-repair of the GlusterFS cluster is finished...

This is the issue (also I took with my linked bugreport above) that the CRS does not e.g. allow for some conditional rescheduling, i.e. why even attempt to migrate a service to a node where it cannot start, this is not strictly one specific filesystem topic, it's not even filesystem topic per se. Imagine your migration network is up, but your outside world access to that particular node got severed. Naturally, you would not want such service to HA migrate to such node, where it would provide service to no one exactly.

I really hoped people would go to the Bugzilla report and +1 themselves add these scenarios, so that next time HA stack is updated - and the CRS is in need of refreshment - it will take these situations into consideration. They will still be user-dependent, i.e. one would have to create a script to evaluate the conditions, but should be modular.

PS On my inquiries above, I still scratch my head why not put this into one service that loops and waits, but I have not touched gluster for a while and do not want to ask silly questions, so if I find some extra time to try this myself, I will update it to here later. It might be this (mis)understanding of mine regarding gluster rather than a language issue that's causing the confusion. (And trust me, my French would not have helped it, at all. )

 

[zeroun]

Additional notes :

In Proxmox VE 8.2.4, the service responsible for high availability (HA) is the pve-ha-lrm (Local Resource Manager) and the pve-ha-crm (Cluster Resource Manager). These services work together to manage and monitor HA resources in the Proxmox cluster.

• pve-ha-lrm: This service runs on each node in the cluster and manages local resources.
• pve-ha-crm: This service runs on the master node of the cluster and coordinates HA actions between the different nodes.

 

[gfngfn256]

pve-ha-crm: This service runs on the master node of the cluster

Just to get technical (I love it): Proxmox does not use a master/slave node configuration, but rather a manager-lock system, so the CRM in fact runs on all nodes as appears in the docs:

The cluster resource manager (pve-ha-crm) starts on each node and waits there for the manager lock, which can only be held by one node at a time. The node which successfully acquires the manager lock gets promoted to the CRM master.

 

[esi_y]

Just to get technical (I love it): Proxmox does not use a master/slave node configuration, but rather a manager-lock system, so the CRM in fact runs on all nodes as appears in the docs:

I think this is twisting the OP's words (and possibly understanding), he never claimed there to be a "master/slave" concept, he used the "master" designation that one gets from literally CLI output and there is indeed only one master at any given time (the "slaves" are, if you will, the LRMs), all the other instances of the CRM service (while admittedly running) are dormant, i.e. doing nothing, waiting to be stand-in, if necessary.

And for total completeness, there are also bugs in this approach that no one even acknowledges 6 months in :
https://bugzilla.proxmox.com/show_bug.cgi?id=5243

If it sounds like a nitpick, consider this is only an issue because the CRM needs to be migratory in this "manager lock" system.

 

[gfngfn256]

I think this is twisting the OP's words

I mostly commented, because I have seen many who do not realize that Proxmox does not use a master-slave/node system (as opposed to other Computer cluster system that do). All my comments on this forum (& others) are directed to ALL users browsing the posts, so whether or not that was the OP's intention is not relevant, it is a question of what reading his post may imply or be understood to others.

________________________________________________________



Now to nitpick, (you started & I just know you love it!):

he never claimed there to be a "master/slave" concept

Oxford dictionary definition of "master";

a man who has people working for him, especially servants or slaves.

There are other definitions there, but all imply to be a master over others or something.

So as far as I see it, if the OP states:

This service runs on the master node of the cluster

the definite implication is that this master node has slave node/s within the cluster.
Thus I conclude that my understanding of the OP's post is far from either "twisting the OP's words" or their "understanding".

while admittedly running

Dormant or not, this contradicts the OP's statement definite implication:

This service runs on the master node of the cluster

This understanding can make a huge difference to correct system management.

Anyway enjoy your day!

 

[esi_y]

I mostly commented, because I have seen many who do not realize that Proxmox does not use a master-slave/node system (as opposed to other Computer cluster system that do). All my comments on this forum (& others) are directed to ALL users browsing the posts, so whether or not that was the OP's intention is not relevant, it is a question of what reading his post may imply or be understood to others.

Fair enough, I just felt like we ran into some possible translation hiccups in this thread before and just wanted to express there's nothing fundamentally wrong with his note, it covers the point that only one node actually is CRM at any given time.

the definite implication is that this master node has slave node/s within the cluster.
Thus I conclude that my understanding of the OP's post is far from either "twisting the OP's words" or their "understanding".

I don't want to speculate where the OP took his note from, but ha-manager status literally designates "master" as the node the CRM is currently running on. It actually, at that point, is the master of the HA stack, so I understand why they use the term.

Dormant or not, this contradicts the OP's statement definite implication:

Alright, I take this one, the "master node of the cluster" was the trigger, it's the HA master.

When I look at the docs [1] now:

And you can view the actual HA manager and resource state with:

# ha-manager status
quorum OK
master node1 (active, Wed Nov 23 11:07:23 2016)
lrm elsa (active, Wed Nov 23 11:07:19 2016)
service vm:100 (node1, started)

So, well, not the best way to convey what you tried (by the docs, not you) either.

This understanding can make a huge difference to correct system management.

Anyway enjoy your day!

Likewise, no hard feelings!

[1] https://pve.proxmox.com/pve-docs-6/ha-manager.1.html

 

[zeroun]

Mystical proverb : "No master here below."