Validate configuration for multiple Proxmox servers

[Maconly34]

Hello Team Proxmox,

I have 4 Proxmox servers:
A - main with 20 cores, 64 GB Ram, 36 TB RaidZ2
- multimedia, documents, Docker services (password, Nginx, Homepage etc.)

B - secondary with 12 cores, 32 GB Ram, 24 GB RaidZ2
- its backup role of replacing the main one for multimedia and document access in the event of failure of A

C and D - miniPCs
- They are there to ensure the Quorum if I create a Cluster
- Provide Docker services in the event of failure of the main server (A)

- An external drive with 8 disks, i.e. 24TB RaidZ2. It can only be connected via Thunderbolt (USB-C).
Its role is to provide multimedia/document backup (one-off updates every xx days).

- I want to automatically replicate the content of A to B once a day and when it's finished I turn it off and on automatically again the next day before the next replication (ZFS replication).
If A fails, I manually turn B on. No need multimedia/documents for high availability
However, I want the Docker services to continue running

So my first idea, which may be aberrant , I'm counting on your help to guide me:
- Create a cluster with main and minis PC : A, C and D.
- If A fails, C or D takes over the Docker services (HA, replication of the Docker VM).
- Next I turn B back on manually to ensure multimedia and access to documents

What do you think?

Thanks for your help

 

[Maconly34]

This post is already on its 10th page and still no response...
Is the analysis of my strategy too complicated?
Thanks for your help

 

[_gabriel]

Is the analysis of my strategy too complicated?

too complicated isn't the word, too things sure

Create a cluster with main and minis PC : A, C and D.

Doesn't require help : https://pve.proxmox.com/wiki/Cluster_Manager#pvecm_create_cluster

With Replication (ZFS), you get HA (not fast as Shared Storage of course)

 

[Maconly34]

too complicated isn't the word, too things sure


Doesn't require help : https://pve.proxmox.com/wiki/Cluster_Manager#pvecm_create_cluster

With Replication (ZFS), you get HA (not fast as Shared Storage of course)

How can you ask a question about the effectiveness of a strategy without explaining the context?

 

[UdoB]

"Keep it simple, stupid." If that's the correct term for a clustered environment ;-)

This is what I would do: create a cluster of all four. Add an additional QDevice to get five votes.

Now "B" can be shut down at any time and as long as you want without problems. And C+D will continue to do its job wenn A fails while B is already down.

Just my 2€¢...

 

[Maconly34]

Thank you for your answer.
So total cluster of 4 servers + Qdevice for the quorum if B is switched off (to save energy) and A is down.
Do I need to give B an extra vote?
THANK YOU

 

[Maconly34]

too complicated isn't the word, too things sure


Doesn't require help : https://pve.proxmox.com/wiki/Cluster_Manager#pvecm_create_cluster

With Replication (ZFS), you get HA (not fast as Shared Storage of course)


P.S. I didn't ask for help in creating a Cluster, but to validate this strategy. Thanks all the same

Regarding ZFS replication, does the receiving ZFS have to be the same total size as the source or just the amount needed to accommodate all the data?

Sorry if I'm asking obvious or stupid questions but I'm just starting out.

 

[UdoB]

Do I need to give B an extra vote?

No. What should it be good for?

In my approach each A+B+C+D+Q = 1+1+1+1+1 = 5 in total. Two parts may fail while keeping Quroum at 3. Actually I am not sure what the QDevice will do if you change the balance to be 1+2+1+1+1 = 6. This would not make sense!

Probably (not sure, not tested!) the QDevice would disable itself, so you land on 1+2+1+1 = 5. But this is not what you want. The example was to shut down B often, so it would be 1+0+1+1=3 which is still fine. But now when A fails the remaining 0+0+1+1 will NOT quorate. Administration of the cluster becomes impossible. (Without applying additional tricks.) And no, I do not expect the QDevice to kick in to help in this situation.

 

[Maconly34]

Let me be more specific. As you say, an extra vote would be pointless.
Currently my most powerful server (A) runs all the services (Vaultwarden, Unbound, Adguard, Nginx, Homepage, Home-Assistant and Plex).
In the event of a breakdown, I would like a mini PC (C or D) to take over these services, except for Plex, while I start up the 2nd powerful server (B) as a backup.
So HA for : Vaultwarden, Unbound, Adguard, Nginx, Homepage, Home-Assistant.
Knowing that B will be started at fixed times for data replication.
THANK YOU

P.S. Qdevice = Raspberry ?

 

[UdoB]

P.S. Qdevice = Raspberry ?

Yes, that's a valid possibility.

Another is a piece of software installed on any other available Linux system, running 24*7. Or a VM: if I had the need for a Qdev I would utilize my Synology Diskstation. The Qdevice does not need much CPU power; it does not need to be reachable with really low latency (cluster members do!); it does not need to be on the same network...

 

[Maconly34]

Thank's for this information!
Regarding ZFS replication, does the receiving ZFS have to be the same total size as the source or just the amount needed to accommodate all the data?

 

[UdoB]

Regarding ZFS replication

The receiving ZPool may have any size. Of course it needs to be large enough to store the data. (Not tested, I try hard to avoid "Pool full" conditions! Weird things may happen in that situation.)

Another limitation when using the integrated ZFS-replication mechanism of PVE is that the specific pool must be present on all nodes with the same name - to qualify as "shared".

This seems trivial, but if you grow your cluster without correct planing you may end up with for example, just for demonstration: one machine with a small "rpool" + large "datastore", another node with small "rpool" and large "storage" and a third system with only a large "rpool".

This is worst case: "datastore" and "storage" can not replicate and "rpool" is too small to receive VMs from the third node.

That's bad design; but note that this does not affect manual live migration - just replication is not possible.

Some things are a not as streamlined as one wishes ;-)

 

[Maconly34]

Fortunately, you told me about this and I thank you for that! I was certainly heading for a big problem...
Explanation:
- I want to replicate data from A to B (the 2 big servers). The pool to be replicated is on Truenas, and I was planning to replicate from Truenas.
- Is this possible given your previous explanations, since Truenas is going to manage the replication?
- Problem: the 2 mini PCs have very little space and one has no Rpool!

If that's not possible, I'll use Rsync.

Thanks again for your help

 

[UdoB]

- Is this possible given your previous explanations, since Truenas is going to manage the replication?

You can only replicate PVE <--> PVE or TrueNAS <--> TrueNAS.

But you can create your own "ZFS send/receive"-scripts or look for toolsets doing so. ZFS in itself is compatible across that border. But then your are leaving the well established PVE software stack and are on your own. I have no idea of some other guy may have prepared such tools already. I know TrueNAS "a lttile bit", but currently I do not use it.

- Problem: the 2 mini PCs have very little space and one has no Rpool!

As already said: the official "Replication"-feature requires the same pool on both sides.

What you can do is to utilize manual migration with a different target storage. And on the CLI there is "qm remote-migrate".

The last ressort is the way via a backup server (PBS recommended!). Just run a backup frequently and do a restore (to a selectably datastore) when required. No automatism though for this approach...

Combining heterogene hardware into a cluster is usual for a homelab - and it always has some aspects that do not fit together without "hickups"...

If that's not possible, I'll use Rsync.

Look for "ZFS send/receive" if you want to script yourself. Or look for some "ZFS replication scripts" on Github for full-feature scripts otherwise.

First look here:

~# apt show pve-zsync

Maintainer: Proxmox Support Team <support@proxmox.com>

Description: Proxmox VE ZFS syncing tool
 Tool for automated syncing of ZFS subvolumes and pools.

 

[Maconly34]

Hello UdoB,
Thank you very much for these precise answers
I'm going to look at the scripts whose code you gave me.

 

[Maconly34]

OK, I've just set up the cluster with the 4 servers as you recommended.
-> If A failed, I could start up B, which was already configured to run all the multimedia services (Plex, Truenas etc.) with its own data.

In this current cluster case, can I have a duplicate of all these services on the backup server B without this causing any problems (VM off with manual startup and backup server B also off)?

 

[UdoB]

In this current cluster case, can I have a duplicate of all these services on the backup server B without this causing any problems (VM off with manual startup and backup server B also off)?

I do not see any additional problems. The problem with the differently named Datastores still exist though.

The "slow" workaround was already mentioned: just make Backups frequently enough and pull a new instance from that Backup when required...

 

[Maconly34]

Ok, sorry for the beginner's questions