Hello!
I am in the situation where I have to (rather: want to) upgrade and then extend my Proxmox cluster, which currently consists of two nodes (pve01 and pve02) with the same hardware (server model, CPU, RAM, network cards, disks) which are running several important (of course!) VMs and containers, the vast majority of which run on pve01. Both nodes are currently on PVE 7.0-11, so a pretty old version but still "pretty compatible" to the latest version 8.1, I think.
Now I am thinking what would be the best way to upgrade the entire cluster and afterwards add two new nodes (which already run on 8.1.4 but have different hardware than pve01 and pve02, including a different disk setup). Due to this, I am a little reluctant to add them to the cluster, since the documentation (https://pve.proxmox.com/wiki/Cluster_Manager#pvecm_join_node_to_cluster) states that "the node will inherit the cluster’s storage configuration" and I'm not sure what that means exactly and what consequences it will have if the joining nodes have completely different disks (sizes, models, number) than the existing nodes in the cluster? Will the join operation re-format everything on the joining nodes to match the disk layout of the existing nodes? What if that's not possible because different disks exist between the nodes?
An alternative would be to leave the two new nodes outside of the cluster (or join those two in a separate cluster, since the two new nodes DO have the same hardware & disk layout among themselves) and just upgrade the existing cluster from 7 to 8 somehow. For this upgrade, I see two methods:
1. Migrate every VM/container from pve02 to pve01, then upgrade pve02 "in place" according to this (https://4sysops.com/archives/upgrade-proxmox-ve-7-to-8/) guide (is it legit?), then migrate everything from pve01 to the updated pve02 and follow the guide on pve01. Afterwards, split the VMs/containers between the two nodes again.
2. Migrate every VM/container from pve02 to pve01, then remove pve02 from the cluster like described here (https://pve.proxmox.com/wiki/Cluster_Manager#_remove_a_cluster_node), install Proxmox 8.1 fresh from a USB stick and re-join the cluster afterwards (without configuring any storage beforehand, since it will inherit the storage-config from the cluster??). Then migrate everything from pve01 to pve02, remove pve01 from the cluster, install fresh from USB there and re-join.
For method #2, I am unsure what is meant in the documentation by "make sure that it will not power on again(in the existing cluster network) with its current configuration" after removing a node from a cluster... do they really mean "power on" or rather "boot into the installed Proxmox"? Because, if I want to freshly install Proxmox from a USB stick, I would have to power on the node... it just wouldn't boot into the installed Proxmox but rather from USB into the installer which then overwrites/deletes the installed version. I assume they mean "make sure it doesn't boot"?? What if I miss the "press F10 to boot from USB" prompt, though? Will it wreck my cluster immediately upon booting? There is also nothing written in the linked documentation on what to do with the storage that is configured on the removed node in the "Datacenter -> Storage" view.. will this storage automatically get removed when removing the node with the
All in all, method #1 probably sounds the safest? If the upgrade fails for some reason on pve02, I would still be left with one functioning node which has all the VMs/containers running. And I could then still switch to method #2 and remove the offline node from the cluster, then install 8.1 fresh on it, right?
While writing this, I get the feeling that method #1 is likely the right way to go. Just still want to make sure since these are production systems and I'm relatively new to Proxmox.
Thanks in advance for any advice!
I am in the situation where I have to (rather: want to) upgrade and then extend my Proxmox cluster, which currently consists of two nodes (pve01 and pve02) with the same hardware (server model, CPU, RAM, network cards, disks) which are running several important (of course!) VMs and containers, the vast majority of which run on pve01. Both nodes are currently on PVE 7.0-11, so a pretty old version but still "pretty compatible" to the latest version 8.1, I think.
Now I am thinking what would be the best way to upgrade the entire cluster and afterwards add two new nodes (which already run on 8.1.4 but have different hardware than pve01 and pve02, including a different disk setup). Due to this, I am a little reluctant to add them to the cluster, since the documentation (https://pve.proxmox.com/wiki/Cluster_Manager#pvecm_join_node_to_cluster) states that "the node will inherit the cluster’s storage configuration" and I'm not sure what that means exactly and what consequences it will have if the joining nodes have completely different disks (sizes, models, number) than the existing nodes in the cluster? Will the join operation re-format everything on the joining nodes to match the disk layout of the existing nodes? What if that's not possible because different disks exist between the nodes?
An alternative would be to leave the two new nodes outside of the cluster (or join those two in a separate cluster, since the two new nodes DO have the same hardware & disk layout among themselves) and just upgrade the existing cluster from 7 to 8 somehow. For this upgrade, I see two methods:
1. Migrate every VM/container from pve02 to pve01, then upgrade pve02 "in place" according to this (https://4sysops.com/archives/upgrade-proxmox-ve-7-to-8/) guide (is it legit?), then migrate everything from pve01 to the updated pve02 and follow the guide on pve01. Afterwards, split the VMs/containers between the two nodes again.
2. Migrate every VM/container from pve02 to pve01, then remove pve02 from the cluster like described here (https://pve.proxmox.com/wiki/Cluster_Manager#_remove_a_cluster_node), install Proxmox 8.1 fresh from a USB stick and re-join the cluster afterwards (without configuring any storage beforehand, since it will inherit the storage-config from the cluster??). Then migrate everything from pve01 to pve02, remove pve01 from the cluster, install fresh from USB there and re-join.
For method #2, I am unsure what is meant in the documentation by "make sure that it will not power on again(in the existing cluster network) with its current configuration" after removing a node from a cluster... do they really mean "power on" or rather "boot into the installed Proxmox"? Because, if I want to freshly install Proxmox from a USB stick, I would have to power on the node... it just wouldn't boot into the installed Proxmox but rather from USB into the installer which then overwrites/deletes the installed version. I assume they mean "make sure it doesn't boot"?? What if I miss the "press F10 to boot from USB" prompt, though? Will it wreck my cluster immediately upon booting? There is also nothing written in the linked documentation on what to do with the storage that is configured on the removed node in the "Datacenter -> Storage" view.. will this storage automatically get removed when removing the node with the
pvecm delnode command?All in all, method #1 probably sounds the safest? If the upgrade fails for some reason on pve02, I would still be left with one functioning node which has all the VMs/containers running. And I could then still switch to method #2 and remove the offline node from the cluster, then install 8.1 fresh on it, right?
While writing this, I get the feeling that method #1 is likely the right way to go. Just still want to make sure since these are production systems and I'm relatively new to Proxmox.
Thanks in advance for any advice!
