Hello forum,
I am considering building a new production (not lab) Proxmox cluster with HA for both running VMs and the underlying storage. Given the current DRAM shortage, I have to consider how to optimize the design in terms of total DRAM purchase for the entire cluster.
I am currently considering two options:
A. Two Proxmox nodes and a third non-proxmox machine acting as tie-breaker to help the cluster choose the machine that will be responsible for production execution if the network links between the two nodes fail. In terms of DRAM, this means that each Proxmox node will need enough physical memory to run all the VMs while the other node is down or isolated, thus the DRAM per node should be (sum of all VMs + clustering overheads such as ZFS overhead), total cost is thus 2 x (sum of all VMs) + 2 x (cluster overheads)
B. Three Proxmox nodes with no tiebreaker outside the cluster. In terms of DRAM, this means that if one node fails, the two remaining machines could split the VMs between them, thus the DRAM per node should be (sum of half the VMs + clustering overheads such as ZFS overhead), total cost is thus 1.5 x (sum of all VMs) + 3 x (cluster overheads).
Option B, thus theoratically saves 25% of the VM HA memory cost, but adds 50% to the cluster overhead memory cost, and also adds the cost of a third physical machine.
Official Proxmox "system requirements" were clearly written when DRAM was cheap, suggesting, without stating reasons, that an additional 1GB RAM/TB disk be added to the clustering overheads. Question is how much this can be safely squeezed for cost, perhaps to 0.5GB/TB or 0.25GB/TB corresponding to 2 bytes/disk block or 1 byte/disk block. Fundamental issue is how much of the stated overhead must be in memory, versus how much is just cached data that can be reloaded from disk/regenerated on the fly versus how much is somehow forced to be kept in physical node RAM at all times.
Another question affecting the purchase calculation is if Proxmox HA requires complete copies of all running VM memory on the node that would take over if the running node crashes, or if Proxmox uses a mechanism that just keeps the memory snapshots on redundant disks until the moment of failover. Obviously, if Proxmox reboots VMs after their active physical node fails, then no DRAM is needed on the node that will potentially run the VM after failover. My calculations for scenario B above assume near zero physical DRAM reservation for potential failover of VMs running on other nodes, thus if each of 3 nodes use y/3 GB for VM memory each, each node needs y/2 GB memory for VMs, of which y/6 GB will just idle waiting for the arrival of HA reloaded VMs from other nodes, whereas keeping alive VM memory clones would need 2/3 * y GB, of which y/3 GB is idle VM memory clones (y/6 GB from each of the other nodes).
I am considering building a new production (not lab) Proxmox cluster with HA for both running VMs and the underlying storage. Given the current DRAM shortage, I have to consider how to optimize the design in terms of total DRAM purchase for the entire cluster.
I am currently considering two options:
A. Two Proxmox nodes and a third non-proxmox machine acting as tie-breaker to help the cluster choose the machine that will be responsible for production execution if the network links between the two nodes fail. In terms of DRAM, this means that each Proxmox node will need enough physical memory to run all the VMs while the other node is down or isolated, thus the DRAM per node should be (sum of all VMs + clustering overheads such as ZFS overhead), total cost is thus 2 x (sum of all VMs) + 2 x (cluster overheads)
B. Three Proxmox nodes with no tiebreaker outside the cluster. In terms of DRAM, this means that if one node fails, the two remaining machines could split the VMs between them, thus the DRAM per node should be (sum of half the VMs + clustering overheads such as ZFS overhead), total cost is thus 1.5 x (sum of all VMs) + 3 x (cluster overheads).
Option B, thus theoratically saves 25% of the VM HA memory cost, but adds 50% to the cluster overhead memory cost, and also adds the cost of a third physical machine.
Official Proxmox "system requirements" were clearly written when DRAM was cheap, suggesting, without stating reasons, that an additional 1GB RAM/TB disk be added to the clustering overheads. Question is how much this can be safely squeezed for cost, perhaps to 0.5GB/TB or 0.25GB/TB corresponding to 2 bytes/disk block or 1 byte/disk block. Fundamental issue is how much of the stated overhead must be in memory, versus how much is just cached data that can be reloaded from disk/regenerated on the fly versus how much is somehow forced to be kept in physical node RAM at all times.
Another question affecting the purchase calculation is if Proxmox HA requires complete copies of all running VM memory on the node that would take over if the running node crashes, or if Proxmox uses a mechanism that just keeps the memory snapshots on redundant disks until the moment of failover. Obviously, if Proxmox reboots VMs after their active physical node fails, then no DRAM is needed on the node that will potentially run the VM after failover. My calculations for scenario B above assume near zero physical DRAM reservation for potential failover of VMs running on other nodes, thus if each of 3 nodes use y/3 GB for VM memory each, each node needs y/2 GB memory for VMs, of which y/6 GB will just idle waiting for the arrival of HA reloaded VMs from other nodes, whereas keeping alive VM memory clones would need 2/3 * y GB, of which y/3 GB is idle VM memory clones (y/6 GB from each of the other nodes).
