Addind a v6 node to a v5 cluster

[greg]

Greetings

I'm trying to add a new node to my cluster. I followed the documentation to update all the current nodes to the latest v5 and corosync v3.
It's mostly working:
- current nodes see each other
- on the GUI of one node I can control the other nodes
- in Datacenter/Cluster, I see the node list

However:
- in Datacenter/Cluster, it's written "Standalone node - no cluster defined", and the "join information" button is disabled
- if I try to add a node with CLI, I have a weird message:

Login succeeded.
check cluster join API version
Request addition of this node
An error occured on the cluster node: Parameter verification failed.
! property is not defined in schema and the schema does not allow additional properties
Cluster join aborted!

What am I doing wrong?

Thanks in advance
Regards

 

[t.lamprecht]

This is not supported, you cannot really join older nodes (as in major release older, intra-major-release works fine) to newer cluster or vice versa. You could finish the join manually but, IMO it's much better to either:
* get the PVE 5.4 nodes up to PVE 6.x, then join
* install a PVE 5.4 on the new node, join, then upgrade to PVE 6.x as soon as possible

 

[greg]

Oh I see, I mis-understood that it was possible.
I'll try to update the cluster then, it's more work but make more sense than creating a new v5 node.
thanks.

 

[t.lamprecht]

Oh I see, I mis-understood that it was possible.

We only really support PVE 5 and 6 nodes coexisting during the upgrade of a whole cluster, and then the cluster need to be formed before starting the upgrade. So existing are possible, but new ones not during that.

 

[greg]

So would it be safer to first add the new node as v5 and then upgrade everything? I have a 3 nodes cluster in production and I can't afford a mirror testbed

 

[t.lamprecht]

So would it be safer to first add the new node as v5 and then upgrade everything?

So you have the three node cluster now and would add a fourth node? If so then I'd add the node after the upgrade, as 6.x node.
A fourth node won't make the cluster "safer" for the "how many nodes can be offline" question.
As quorum required for three node cluster are two online nodes, for a four node cluster it would need three online nodes to get the majority.

Also note that you could run a nested PVE setup (in three VMs) on your production cluster to test through the upgrade, just as a suggestion.

 

[greg]

Thanks for your answers!

Also note that you could run a nested PVE setup (in three VMs) on your production cluster to test through the upgrade, just as a suggestion.

Hum I've never tested that sounds fun! Would it help me preparing my cluster update to v6? So I would copy the current hosts as "guest hypervisors" on one of the current actual nodes?

 

[t.lamprecht]

Would it help me preparing my cluster update to v6?

Yes, you can even snapshot the hosts before doing an upgrade, then rollback and retry. Makes experimenting and getting a feeling for the upgrade easy. Although it's not a 100% replacement for a physical test lab (para-virtualized HW isn't 1:1 real one), it can be made quite close to it.

So I would copy the current hosts as "guest hypervisors" on one of the current actual nodes?

I'd not copy anything 1:1, risks network, etc. duplication and other weird things which could maybe even influence the physical host production setup. I mean, can be done, but needs taking more care.
Normally I install three VMs with PVE from scratch, or one and clone that + nodename changes afterwards, both pretty quick to do.

 

[greg]

I'd not copy anything 1:1, risks network, etc. duplication and other weird things which could maybe even influence the physical host production setup. I mean, can be done, but needs taking more care.
Normally I install three VMs with PVE from scratch, or one and clone that + nodename changes afterwards, both pretty quick to do.

Do you have a procedure document to help me understand how I can do that?

 

[t.lamprecht]

Do you have a procedure document to help me understand how I can do that?

There isn't too much complexity to it, it's mostly like you'd install it normally but in VMs:

1. Create a VM with the Proxmox VE 5.4 ISO selected, use 2 GB memory (1 GB can work, but then nested guest tests may not) and at least 2 vCPU cores. A disk with >8 GB should do for the test, if you have enough space use >=32G
2. Start the VM and install Proxmox VE, be sure to use an IP address and node name which you do not confuse with your real Proxmox VE setup
3. Repeat that until you have three virtual Proxmox VE nodes
4. then you can cluster them
5. Then I'd make a snapshot of all three VMs from the real PVE host, so you can easily rollback and test again

Instead of step 3 you could also clone the first created VM for number two and three and change the hostname afterwards, may be quicker and possibly saving some space but more prone to make errors when changing the hostname.