Setting up 3 Node Cluster - One Node Over Wireguard

[theprez1980]

Hey All -

I decided to start from scratch and seem to be running into an issue -

Here's my setup:

2 Nodes - Same hardware, local to each other and connected to a 5x5 fiber internet connection. - 192.168.0.X/24
1 Node - Same hardware, remote on a 5x5 gig fiber connection in the same city - 192.168.3.X/24

All 3 nodes can see, ping, and access each other via SSH or via IP:8006 - the remote node is connected via wireguard via OpnSense. latency is <5ms. No dropped/blocked UDP packets in the firewall on either side.

--
Adding the local nodes goes as expected - no issue.

Adding the third node, which is remote - never completes successfully - it freezes at the command line with Wating for Quorum.....Ok and never proceeds. I end up with the node appearing on the other two's GUI, but it's grayed out with a question mark.

The remote machine returns the following after pvecm node:

Membership information
----------------------
Nodeid Votes Name
1 1 pve01
2 1 pve02
3 1 pve03 (local)


Looking at journalctl -u corosync I see a lot of this kind of activity but perhaps its normal?

Dec 05 15:58:25 pve03 corosync[4342]: [TOTEM ] Retransmit List: 20 21 22

I'm not sure where to begin troubleshooting and have already rebooted all three nodes, any ideas?

 

[Johannes S]

You are sure Wireguard keeps the connection open? From the wireguard website:

By default, WireGuard tries to be as silent as possible when not being used; it is not a chatty protocol. For the most part, it only transmits data when a peer wishes to send packets. When it's not being asked to send packets, it stops sending packets until it is asked again. In the majority of configurations, this works well. However, when a peer is behind NAT or a firewall, it might wish to be able to receive incoming packets even when it is not sending any packets. Because NAT and stateful firewalls keep track of "connections", if a peer behind NAT or a firewall wishes to receive incoming packets, he must keep the NAT/firewall mapping valid, by periodically sending keepalive packets. This is called persistent keepalives. When this option is enabled, a keepalive packet is sent to the server endpoint once every interval seconds. A sensible interval that works with a wide variety of firewalls is 25 seconds. Setting it to 0 turns the feature off, which is the default, since most users will not need this, and it makes WireGuard slightly more chatty. This feature may be specified by adding the PersistentKeepalive = field to a peer in the configuration file, or setting persistent-keepalive at the command line. If you don't need this feature, don't enable it. But if you're behind NAT or a firewall and you want to receive incoming connections long after network traffic has gone silent, this option will keep the "connection" open in the eyes of NAT.
(https://www.wireguard.com/quickstart/ )

Why do you need to have one node offsite? For my taste this setup looks way to error-prone for production use.