Virtual Proxmox VE Node

[Astraea]

I know asking or implementing a Proxmox VE node as a VM either within or outside of a Proxmox VE cluster can be a danger zone and also a disputed topic. I am considering deploying a single Proxmox VE node into my 7 node cluster for the following reasons:

1. It would provide a "Floating" or "virtual" IP address that I could use to access the cluster using a reverse proxy as there is not a cluster IP and i do not want to install something such as keepalived on the nodes. Also while i can use backends in Nginx, I have had issues in the past with timeouts when accessing VM consoles over NoVNC.
2. It would provide a location to plate VMs that are not being used and are in an "OFF" state to be stored in a location that would be separate from the nodes that are running active workload. While this is purely a cosmetic and personal preference piece I think it would help with managing my some 50 VMs.
3. Moving from 7 to 8 nodes would not effect my clusters ability to mainain quorum as if the node hosting the VM goes down it would remove 2 of 8 votes and once that VM was migrated to another node via HA it would regain that vote back providing the VMs vote back. Though i could also make it a non voting member i believe as well. Either way so long as I keep enough nodes active it should not create an issue.

Though question on point 3, would the required number of nodes to maintain quorum move from 4 to 5 with 8 nodes or does it not change until a cluster has 9 nodes.

 

[UdoB]

quorum move from 4 to 5 with 8 nodes

Yes. Quorum needs to be more than 50 percent, so 4 out of 8 is not ok.

 

[bbgeek17]

There is also a mathematically significant chance that the physical node that is hosting the virtual node is the one that will go down, so you will loose two nodes in your cluster at the same time. So take that into consideration of whether such complication is worth it over a proxy.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Astraea]

I'm just walking through the logic of not messing with the votes within Proxmox VE to see if this works....

• Currently the cluster can loose any 3 nodes and everything keeps trucking along.
• Moving to an 8 node cluster would mean needing 5 nodes online for quorum, still allows for 3 nodes to fail.
• However if the node that fails is the node with the VM node then 2 votes of 3 allowed are lost.
• If the cluster is at 8 online and looses 2 then it is still in quorum as there is still 1 more vote than needed, and once the VM node migrates it would be back up to 7 of 8 votes.
• Should the cluster loose another node and of course worst case is the node with the VM node, then the cluster is down to 5 votes which is still quorum and the VM node would migrate and then bring the votes back to 6.
• Which at this point I have 2 nodes offline and 6 nodes online though 1 vote is the VM node.
• Worse failure case would be while in this 6 online node state another node fails and this node is also the one with the VM node. This would bring the cluster to 4 nodes online and would loose quorum and as such the VM node would not migrate and would not come back online. This leaves the cluster unable to start, stop, move etc any resources though the remaining nodes would keep their resources running. This would however mean any other resources that were not recovered already from the 2 previous failures or they were on the newly failed node would not be recovered.

The real difference with this state vs my current 7 node cluster is that at 4 nodes I would be in quorum currently and out of quorum if there is this VM node. The other thought would be to go to 9 nodes but the only way that works is to add another physical node.

The other option would be to edit the corosync file and keep the system at 7 voting nodes and expecting 4 for quorum and have the VM node not have a vote....this seems like it would solve the quorum problem as when the third node fails on the 7 node cluster everything is recovered on the 4 nodes, this would include a the VM node which would be more or less a silent partner (no vote).

 

[Astraea]

While i fully agree that using a VM as a node either within the cluster or hosted in or on another machine or cluster of any kind is not following any sort of "best practices" and should not be blindly deployed in any production environment, i think I have come to a solution that I can work with.

I created a test a test environment and kept things very simple on purpose.

1. Created 4 nodes (pve0, pve1, pve2, pve3) as Proxmox VMs within my lab environment and assigned 4 GB RAM (was 2 GB ram but got out of memory error while updating after install) and 8 CPU cores and a virtual OS disk of 20 GB and a single network interface.
2. Downloaded the latest Proxmox VE installer ISO.
3. Installed Proxmox VE using the ZFS file system (as this is what is used in my production environment) using the downloaded ISO.
4. Updated all 4 nodes to the latest version contained in the no-subscription repository and rebooted all the nodes.
5. Next was to cluster nodes 1, 2 and 3 into a cluster. Starting from node 1 to create the cluster and then adding 2 and then 3 respectively into the cluster.
6. Once quorum was established I then followed the directions for editing the corosync file and updated the node IDs as follows: (this allowed for the no-vote node to be assigned ID 1)
   1. pve1 now has a Node ID of 2 instead of 1
   2. pve2 now has a Node ID of 3 instead of 2
   3. pve3 now has a Node ID of 4 instead of 3
7. After restarting the corosync on each node and waiting a minute or two, quorum was established and everything was stable and as it was before the edits with only the IDs being changed.
8. I then added pve0 into the cluster and waited for that to complete. I thought I was going to have to manually move pve0 to have the ID of 1 but the cluster seemed to selected automatically as that was its ID when it was finished being added.
9. I then followed the same steps to edit the corosync file and this time changed the quorum_votes for pve0 to 0 and waited for the changes to be picked up.
10. I then checked pvecm status and it showed that there are 4 nodes in the cluster, that it is expecting 3 votes with the highest being 3 and that it needs 2 for quorum.

I believe this will allow me to have a virtual node within my cluster while not creating an even number of node votes and thus removing the clusters ability to maintain quorum at 4 nodes.

This will also give me a sorta floating WebUI and IP that I can use for the various administration tasks that are preformed and allow for a way to reverse proxy my cluster that can be accessed without referencing a single physical node and or running the issues around using a reverse proxy that references multiples nodes as the backend.

 

[UdoB]

The other thought would be to go to 9 nodes

This additional 9th node may be a very small device like a Raspberry Pi or a VM on another cluster-external hardware. See https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_corosync_external_vote_support

 

[Maximiliano]

Moving from 7 to 8 nodes would not effect my clusters ability to mainain quorum as if the node hosting the VM goes down it would remove 2 of 8 votes and once that VM was migrated to another node via HA it would regain that vote back providing the VMs vote back. Though i could also make it a non voting member i believe as well. Either way so long as I keep enough nodes active it should not create an issue.

Note that with 7 nodes you need 4 votes for quorum and with 8 you need 5. 9 would require 5 votes again.

- Running Promox VE inside a VM is not recommended in production
- Having a node host a VM that cast an extra vote for the same cluster the node belongs to (or any way to have a single machine produce more than a single vote) is not recommended
- Running a cluster with an even number of nodes it not recommended

 

[Astraea]

While i totally agree this is not in anyway a support or a production setup, and to never have any single node produce more than 1 vote. I am not sure i understand this part of your reply:

@Maximiliano > - Running a cluster with an odd number of nodes it not recommended

My understanding is that you need a minimum of 3 nodes for a stable cluster and that you should always work to have clusters of an odd number of nodes.


My final implementation of this idea was to deploy a small VM on the cluster and install Proxmox into it and add it to the cluster but change its vote to 0, this keeping only the physical nodes with a vote and maintaining a 7 node quorum structure (needing at least 4 nodes up for quorum).

 

[Maximiliano]

Sorry I made a mistake in my reply and edited it. Even number of votes are problematic.

My final implementation of this idea was to deploy a small VM on the cluster and install Proxmox into it and add it to the cluster but change its vote to 0, this keeping only the physical nodes with a vote and maintaining a 7 node quorum structure (needing at least 4 nodes up for quorum).

In general the only setup we recommend is

- Odd number of votes
- no more than 1 vote per physical machine (if you add a qdevice or a Proxmox VE node as a VM it counts as an additional vote)
- all cluster members provide exactly 1 node

 

[Astraea]

- Odd number of votes: I have kept only 7 votes in the cluster and only physical nodes are given a vote.
- no more than 1 vote per physical machine: Each physical node has 1 vote, the virtual node does not have a vote.
- all cluster members provide exactly 1 node: This is the only thing that is not now done by my cluster as the virtual node does not have a vote.

I will monitor this setup and if there are any problems or issue here I will post back so that future readers can know the outcome, though if it does work without any issues I think it is best to keep with the message that this is a non supported configuration. I am hopeful that the upcoming multi-cluster or centralized management system that is coming to Proxmox VE would allow me to remove this virtual node.