So.. who wants to hear a really really bad idea that I have currently implemented. :P

[effgee]

2 x Proxmox Physical Machines (Clustered, no HA,1 public ip each, vm firewall owning the public ip)

• They are not in the same physical location
• They each have one NIC
• They are routed via public network to each other. (They have private nat'd ips and vpns to each other)
• I have them talking (Corosync) over several VPNs.
• I have put them in a cluster for management and offline migrations (No HA wanted/needed as well as impossible)
• Each of these machines have a single public IP (each) that I give to a virtual firewall (VM running on each) which then gives internet back to each of the physical machines.
• Small problem with cluster though, on reboot they wont start any virtual machines unless they have quorum, and since in order to have quorum they need their firewall running (teehee) which is a VM.
• One machine I have given 2 votes to, so it can reboot itself and start its firewall vm.
• The other machine has 1 vote, which means it wont start its firewall vm to get access to the quorom.
• Qdevice doesn't work either cause of all network access through virtual firewall no starting issue. (Might run one in a docker container...although I have worked around it for now)
• I force the second one to start its firewall ( and net connectivity) via a very silly bash script (/usr/bin/pvecm expected 1; sleep 5; repeat until network is up) on startup that sets the pvecm expected to 1 until it starts the firewall and gets quorate.
• Oh, also the vpn (Wireguard) for the quorum network is also dumb and doesn't restart when it doesn't have connectivity due to firewall vm being down, even when the fw/network comes up. Had to script that to also be smarter.
• Tada!

A way to exclude a specific vm (to start on boot) from quorum requirements would have made this a lot easier.

Or the ability to unify Proxmox management interfaces without full "clustering" would have also made this easier.


Anyways, I am not complaining, just sharing my insanity and hopefully someone can get a laugh about it or suggest something more clever...And yes, I know if I used the internal Proxmox firewall this could be a lot easier..

 

[guletz]

Hi,

In your case I would use a external hardware firewall(under 100 euro/pcs) who will also create a vpn beethen this 2 servers. On boot time I would set pvecm vote =1, and after x min I would check the remote server, and if the remote is up I woud change the votes as "normal"(2 on one hoste and 1 on second host)

Good luck / Bafta

 

[effgee]

I actually have no physical access to these machines. They are hosted/rented servers.
I updated my script just to be a bit smarter and so far so good. I just run it on boot via cron.

#!/bin/bash


GrepStatus=`pvecm status | grep "Quorate:          Yes"`
TryCounter=0
LogFile=/quorumhax.log
echo "The counter is starting" > $LogFile

while [  "$GrepStatus" = "" ]; do
             echo "The counter is " $TryCounter >> $LogFile
             let TryCounter=TryCounter+1

             echo "Changin pvecm expected 1"
             /usr/bin/pvecm expected 1

             echo "sleeping for 5 seconds"
             sleep 5

             echo "Checking for quorate again..."
             GrepStatus=`pvecm status | grep "Quorate:          Yes"`
         done

echo "Quorate Haxd." >> $LogFile
echo "Starting firewall just in case"
/usr/sbin/qm start 2001

Hi,

In your case I would use a external hardware firewall(under 100 euro/pcs) who will also create a vpn beethen this 2 servers. On boot time I would set pvecm vote =1, and after x min I would check the remote server, and if the remote is up I woud change the votes as "normal"(2 on one hoste and 1 on second host)

Good luck / Bafta

 

[guletz]

Hi again,

And you can do more: you can change your priority for your vpn+corosync trafic with iptables( DSCP = 48 as a start point) ! It could help ....!