HA Best Practice

T

Turbine

New Member
Nov 11, 2024
11
1
3
Hello guys
Will finish to setup my 2 nodes proxmox cluster. ( AMD 4500U 16Gb 500gb Nvme )
I also have 1gbe NAS.

Ceph is doable ( performance wise ) with 2 nodes for shared storage to do HA and live migration?
Or should i stick to local storage and do the backups to the NAS

What are my options ?
 
For a high-availability (HA) setup, a three-node configuration is required. The third node can be a low-spec device, such as a laptop or an Atom mini-PC. For testing or non-production purposes, you could set up Proxmox within a virtual machine (VM) on the second node, while the primary load runs on the first node. Alternatively, if you are using VMware or Hyper-V, Proxmox can be configured as a VM on these platforms for production use. In this setup, the third node would act as a "dummy" member, where minimal resources—such as a single core and 2 GB of RAM—will suffice.
 
Deepen Dhulla said:
For a high-availability (HA) setup, a three-node configuration is required. The third node can be a low-spec device, such as a laptop or an Atom mini-PC. For testing or non-production purposes, you could set up Proxmox within a virtual machine (VM) on the second node, while the primary load runs on the first node. Alternatively, if you are using VMware or Hyper-V, Proxmox can be configured as a VM on these platforms for production use. In this setup, the third node would act as a "dummy" member, where minimal resources—such as a single core and 2 GB of RAM—will suffice.
Click to expand...

This is ceph or proxmox HA required ?
This is for my smart home setup.
I want to simplify things as much as possible.
it’s not possible to do it with 2 nodes ?
what about performance wise with the ceph configuration ?
 
To achieve high availability (HA), a quorum typically requires three nodes. Therefore, an HA setup, with or without Ceph, would necessitate a three-node configuration.

For a homelab, you may want to avoid HA and instead use a manual switchover, which can be simpler and more manageable.

Alternatively, you could set up Proxmox VE as a virtual machine within the second PVE server, as previously suggested.

Another way ...but complex .. https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support
 
Last edited:
Deepen Dhulla said:
To achieve high availability (HA), a quorum typically requires three nodes. Therefore, an HA setup, with or without Ceph, would necessitate a three-node configuration.

For a homelab, you may want to avoid HA and instead use a manual switchover, which can be simpler and more manageable.

Alternatively, you could set up Proxmox VE as a virtual machine within the second PVE server, as previously suggested.
Click to expand...
I can still return one of the mini pc’s.
For my home use, you think it is better to setup only 1 proxmox or should I still use two of them and install what is necessary for redundancy ( such as my dns server on 2 different nodes ) and use the NAS for backups, and if the pc failed , restore all the vm’s from backup?
 
Last edited:
You can use a two node cluster for HA setup, but then you need a qdevice. This has to be installed on a third computer.
Ceph needs 3 nodes in the cluster. Ceph isn't required for HA, but you must have some form of shared storage.
 
For my home use, you think it is better to setup only 1 proxmox and use the NAS for backups, and if the pc failed , restore all the vm’s from backup? ... this is bare minimum away for an IT setup. it's your choice
 
Deepen Dhulla said:
For a high-availability (HA) setup, a three-node configuration is required. The third node can be a low-spec device, such as a laptop or an Atom mini-PC. For testing or non-production purposes, you could set up Proxmox within a virtual machine (VM) on the second node
Click to expand...
Do not do this! At least not with the argument "I need three Nodes for a cluster". This is what a Quorum Device is for!

https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_qdevice_technical_overview
 
  • Like
Reactions: Johannes S, Turbine, Azunai333 and 1 other person
Deepen Dhulla said:
For my home use, you think it is better to setup only 1 proxmox and use the NAS for backups, and if the pc failed , restore all the vm’s from backup? ... this is bare minimum away for an IT setup. it's your choice
Click to expand...
Please explain ?
UdoB said:
Do not do this! At least not with the argument "I need three Nodes for a cluster". This is what a Quorum Device is for!

https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_qdevice_technical_overview
Click to expand...
So what is your recommendation?
 
Turbine said:
So what is your recommendation?
Click to expand...
Build a cluster. It is fun and useful. If you want to stay as small as possible:
  • check if your NAS can serve Containers or VMs of any flavor. Install a small as possible Distribution like Debian - or reuse one that is already running - and install the Quorum Device there
  • If that is not possible think about using an old Raspberry Pi for the QDev. Low power consumption is king
Of course the more better way is to have an actual third node. This increases the possibilities - and the price of the whole setup, both initial buying costs and continuous energy costs.
 
  • Like
Reactions: Johannes S and Turbine
UdoB said:
Build a cluster. It is fun and useful. If you want to stay as small as possible:
  • check if your NAS can serve Containers or VMs of any flavor. Install a small as possible Distribution like Debian - or reuse one that is already running - and install the Quorum Device there
  • If that is not possible think about using an old Raspberry Pi for the QDev. Low power consumption is king
Of course the more better way is to have an actual third node. This increases the possibilities - and the price of the whole setup, both initial buying costs and continuous energy costs.
Click to expand...
I have an Orange Pi - 2gb ram and 8gb of emmc storage.
It can be enough for 3rd proxmox node just for voting and Quorum or I should use it as qdevice ?

Thanks
 
Turbine said:
Ceph is doable ( performance wise ) with 2 nodes for shared storage to do HA and live migration?
Click to expand...
No, no Ceph with two nodes!

No, it is not a good idea to build a Ceph cluster from three full(!) nodes.

Actually for Ceph I personally recommend at least five nodes with three or more OSD per node.


With two nodes you can do ZFS replication. This works very well. This will give you HA and live migration. As long as you can tolerate to lose the data since the last replication point in time.

Of course you may use the NAS via NFS. But then this is a single-point-of-failure!


Choose your poison ;-)
 
  • Like
Reactions: Johannes S and Turbine
Turbine said:
I have an Orange Pi - 2gb ram and 8gb of emmc storage.
It can be enough for 3rd proxmox node just for voting and Quorum or I should use it as qdevice ?
Click to expand...
There is no official support for other architectures than X86/Amd64.

Using it as a Quorum Device will probably work.
 
  • Like
Reactions: Turbine
UdoB said:
No, no Ceph with two nodes!

No, it is not a good idea to build a Ceph cluster from three full(!) nodes.

Actually for Ceph I personally recommend at least five nodes with three or more OSD per node.


With two nodes you can do ZFS replication. This works very well. This will give you HA and live migration. As long as you can tolerate to lose the data since the last replication point in time.

Of course you may use the NAS via NFS. But then this is a single-point-of-failure!


Choose your poison ;-)
Click to expand...
ZFS replication needs extra disks ?

Can i do replication without zfs to the existing local storage of each node ? ( if i got it right, its only “good” option to achieve HA without proper shared storage )
 
Last edited:
Turbine said:
ZFS replication needs extra disks ?
Click to expand...
As so often - it depends:

If you've chosen ZFS during the installation process you have alreada a "rpool". This is sufficient. To verify/confirm the presence:
Code: 
~# zpool status rpool
  pool: rpool
 state: ONLINE
 ...


If you've chosen to use another filesystem then... yes.
 
  • Like
Reactions: Turbine
UdoB said:
As so often - it depends:

If you've chosen ZFS during the installation process you have alreada a "rpool". This is sufficient. To verify/confirm the presence:
Code: 
~# zpool status rpool
  pool: rpool
 state: ONLINE
 ...


If you've chosen to use another filesystem then... yes.
Click to expand...
I am before installing everything so this is a good time point..
I only have 1 local NVMe on each node.
What should I do to enable the replication between local storage ?
 
  • Like
Reactions: Turbine
Turbine said:
But i need choose ZFS file system in the installation to achieve that ?
Click to expand...
You need to have a ZFS Pool. And your VMs must utilize it.

Either you create it during installation - which is of course recommended - or you add new disks later and create that pool on them. This is usually done to have independent pools for the OS and for the VM storage. But this needs several disks...

Do not try to create a ZFS pool on a spare partition or an logical volume!
 
UdoB said:
You need to have a ZFS Pool. And your VMs must utilize it.

Either you create it during installation - which is of course recommended - or you add new disks later and create that pool on them. This is usually done to have independent pools for the OS and for the VM storage. But this needs several disks...

Do not try to create a ZFS pool on a spare partition or an logical volume!
Click to expand...
Got it. I think
Just to make sure - during installation - I can create ZFS pool with 1 nvme disk ( total ) on each node ?
Any key points while I do that ?
 
You must log in or register to reply here.
Top Bottom