Basic Cluster setup

[RaccoonNL]

Hi guys,


Never have used proxmox before, but I'd like to play around with it. I got the option recently to buy some HP Prodesk 400 G5 SF (i5-8500) from the office cheap. I bought 3 of them and some RAM sticks to upgrade the RAM from 8 GB to 32 GB (per unit). But for fun and tinkering, I was looking at automating fail over, so naturally I started looking at clustering. My end goal would be automatic fail over (even when 2 out of 3 are down) and optionally automatic load balancing. Stuff I'm going to run on it is home assistant, jellyfin (movies are currently on an almost full 1TB SSD on a Fujitsu futro s920; movies can stay on there, but that box doesn't like encoding or decoding, so I want to move jellyfin onto this cluster).


But I was wondering if 1 GB of Ethernet will be enough and what to do storage-wise. Read on the forum somewhere that Ceph doesn't like when 2/3 servers in the cluster are down, so I'm looking at ZFS replication or raid 1.


But before spending more money on it, I'd like some advice and tips or tricks.


Thanks in advance.

 

[Daniel_Dog]

Personally I do not use clustering as I run just one server but here is my input:

If you plan on using ZFS/local storage then you are responsible for getting all the VM data to each host you may want to run the VM on. (Before a host goes offline as everything not synced before hand will be lost until the original host is back online.)
Many people use CEPH for making sure the VM data is on all hosts and is accessible to all host. (Even if one of the hosts goes offline.)

CEPH does not like to when it has less then 3 nodes but it “can” do it temporality until the 3th node comes back online. Just not with 1 as at that point it does not know what host has the correct data and weather or not its host has the correct data or not.
And also Proxmox VE needs at least two online nodes/hosts for HA at all times to work. (3 node minimum since if a node goes down then the other nodes need to be able to figure out that a node is down. This is done by voting and it needs a majority and 1/3 would not be a majority.)
https://pve.proxmox.com/wiki/Cluster_Manager#_quorum

 

[RaccoonNL]

Personally I do not use clustering as I run just one server but here is my input:

If you plan on using ZFS/local storage then you are responsible for getting all the VM data to each host you may want to run the VM on. (Before a host goes offline as everything not synced before hand will be lost until the original host is back online.)
Many people use CEPH for making sure the VM data is on all hosts and is accessible to all host. (Even if one of the hosts goes offline.)

CEPH does not like to when it has less then 3 nodes but it “can” do it temporality until the 3th node comes back online. Just not with 1 as at that point it does not know what host has the correct data and weather or not its host has the correct data or not.
And also Proxmox VE needs at least two online nodes/hosts for HA at all times to work. (3 node minimum since if a node goes down then the other nodes need to be able to figure out that a node is down. This is done by voting and it needs a majority and 1/3 would not be a majority.)
https://pve.proxmox.com/wiki/Cluster_Manager#_quorum

I'm planning on running them all the time ofcourse, but keeping worst case in mind I don't want to lose data when 2 nodes go offline before I can fix it.