2-Node Cluster Setup

[joegoldman]

G'day forum.

Just wanted to get the latest standard practice with proxmox regarding a 2-node cluster setup. I know that the clustering requires 3 for quorum, and I intend to run a Q-Device (from my house) to the 2 servers to act as the extra vote, but I have other questions regarding storage and resource usage.

I intend to have 4x 2TB SSD's and 4x XTB Spin Drives (not sure how big yet) in each server
They are identical CPU's and both with 512GB RAM.
I want to ultimately load balance between them but also have them able to fail over to each other.

My intent was to RAID10 the disks for performance, and use something like Ceph or GlusterFS to do essentially a 1:1 copy of all data between the servers. I know I am ultimately using 16TB of SSD for 4TB usable space but the redundancy is most important to me for this setup, and performance is pretty key as I'll be running some disk heavy operations (databases)

This is for my personal startup projects, so I don't have really any budget to go add a 3rd server, I ultimately just want the case of if server a dies, server b re-boots important servers from server a and everything keeps running.


I am also looking at ultimately just having 2 VMs for each service in some kind of load balance/failover, and run each proxmox independantly - but can I 'half' cluster them, i.e. add them both to the same management portal? I would prefer HA as some VMs i might not be able to have proper redundancy, but I can live with it if theres no good options for the HA clustering with 2 nodes.

Thanks in advance for your advice!

 

[ucholak]

hi, these notes can help:

# 2 nodes cluster
#   https://rickelobe.com/2019/12/04/two-node-proxmox-cluster-with-high-availability/
#   https://gitlab.rickelobe.com/virtualiztion/proxmox-2n-ha-monitor

cd /usr/local/src
git clone https://gitlab.rickelobe.com/virtualiztion/proxmox-2n-ha-monitor.git

sed -i 's@--down_threshold=3@--down_threshold=12@g' proxmox-2n-ha-monitor/monitor_cluster.service
cp  proxmox-2n-ha-monitor/monitor_cluster.service /lib/systemd/system/


# modified systemctl service
echo '
[Unit]
Description=Monitor Cluster Service
#After=multi-user.target
After=network.target
Conflicts=getty@tty1.service

[Service]
Type=simple
ExecStart=/usr/bin/python3 /usr/local/src/proxmox-2n-ha-monitor/monitor_cluster.py --down_threshold=12
StandardInput=tty-force

Restart=always
RestartSec=60


[Install]
WantedBy=multi-user.target
' > /lib/systemd/system/monitor_cluster.service




systemctl daemon-reload
systemctl enable monitor_cluster.service
systemctl start monitor_cluster.service
systemctl status monitor_cluster.service


# 2 VOTES
# vim /etc/corosync/corosync.conf
#  quorum_votes: 2
#  config_version: ++

 

[Maximiliano]

Hello, Please do not run random scripts you find in the internet without knowing what they do

 

[aaron]

Have you considered using ZFS + Replication? The interval for the replication can be a short as each minute. It can be combined with HA and should a node actually fail, the potential data loss will be minimal if even present, depending on your workload. Live migrations between the nodes will also be a lot faster.

 

[joegoldman]

Have you considered using ZFS + Replication? The interval for the replication can be a short as each minute. It can be combined with HA and should a node actually fail, the potential data loss will be minimal if even present, depending on your workload. Live migrations between the nodes will also be a lot faster.

I had not! Is the replication built into ZFS? Is it a block level replication? i.e. when a VM Hard Disk changes minute to minute, will it sync only the changes or does it have to re-sync the entire image file?

Is this naturally supported by proxmox in a cluster, so both nodes where to look for the hard disk for the same VMs or is it just data replication and in a failure state there's some manual re-mounting to get the VM up and running?

 

[aaron]

The guest replication feature currently only works with ZFS and yes, it utilizes the ZFS send/recv functionality -> Incremental.

Requirements are that the pool and storage is called the same across the nodes.

In case of a VM being moved/recovered to another node, the replication job's direction is switched. So yeah, (live) migration and HA recovery are handled by Proxmox VE.

You can also have multiple replication jobs. But honestly, it gets a bit unwieldy with more than three nodes. But then other storage options like Ceph become interesting.