PVE CLuster upgrading/updating guide?

m.ardito

Active Member
Feb 17, 2010
1,473
16
38
Torino, Italy
Hi all,
anybody knows about a forum post/wiki page/web page where i can find a "pve cluster safe upgrading" guide or something? I upgraded a single server in the past, recently have built my first production cluster but i'm quite unsure on how to do this kinda critical step with a production pve cluster

Thanks, Marco
 
Just update all nodes, nothing special.

And yes, if you got a critical production system you should test upgrades in your test lab.
 
Well, i have a few questions - as i said i am VERY novice with clusters:

* if i remember well, all cluster nodes should (must?) have same pve version, but the fact that i update one node at a time WILL force a situation when at least two cluster members WILL have different version. How to approach this? May vms be migrated between members having different pve versions? I think not...

* So, should one
- migrate all vms to the "old pve version" (OPV) members, so they are online while some servers upgrade
- update enough OPV members to the "new pve version" (NPV)
- backup all vms from remaining OPVs and restore them to NPVs
- then update all remaining OPV to become NPV
- migrate back all vms on their cluster members (all NPVs now)
?

* Is there any difference in starting the update/upgrade from the master, or leave the master as the last one, or not?

* Is there a best approach strategy to test existing vms/storage/network with new pve versions, in a test cluster environment?

* If one has not a test cluster environment, except buying one, what should he/she do to try to keep a mostly working production cluster, minimizing troubles?

thanks, Marco
 
Last edited:
Hi all,
Marco posed important questions! (One year ago)
Why the team don't respond?
Many people use proxmox in a production environment, where they can find answers to questions?

Thanks,
Jan
 
Use the search function on this forum.

The best and interesting results are:
http://forum.proxmox.com/threads/6729-Cluster-upgrade-Best-practice?highlight=upgrading+cluster

But the questions that Marco posed are very interesting and with search I can't find nothing.

You can also get commercial support from our team if you prefer that:

[url]http://pve.proxmox.com/wiki/Get_support
[/URL]

Nothing to say, the best way is support this project
 
Hi all,
Marco posed important questions! (One year ago)
Why the team don't respond?
Many people use proxmox in a production environment, where they can find answers to questions?

Thanks,
Jan

You are just wrong, I answered the question the same day.
 
I did not answer again as its all said in the first answer.
 
...probably someone here is too busy answering more interesting and less obivious questions, like: "when will be 2.0 ready? we're already in Q3!"...
but i agree there is paid support, and nothing more can be answered free on the forums than "just upgrade all nodes, but if you do not test upgrades in a test lab, you're on your own", by the pve team.
i suppose my company should think about open tickets even just to ask questions like mines...
i'm not asking them anything else here.

but i just don't understand why nobody else (regular users) commented, adding their own experience... i expected at least something like relax, "don't ask yourself too many questions, do as they said: just upgrade all nodes, everything will go fine... i did it yesterday!". but... nothing. this really leave me... wordless.. literally-) btw, i'm still with the 1.7 cluster here (no test lab, unfortunately...),
but <joking> probably 2.0 will solve this kind of problem, so "when will be 2.0 ready? we're already in Q3!" :-D </joking>

thanks, anyway. when i'll upgrade my cluster, i'll report here anything that will happen, my experience...

marco
 
...probably someone here is too busy answering more interesting and less obivious questions, like: "when will be 2.0 ready? we're already in Q3!"...
but i agree there is paid support, and nothing more can be answered free on the forums than "just upgrade all nodes, but if you do not test upgrades in a test lab, you're on your own", by the pve team.
i suppose my company should think about open tickets even just to ask questions like mines...
i'm not asking them anything else here.

but i just don't understand why nobody else (regular users) commented, adding their own experience... i expected at least something like relax, "don't ask yourself too many questions, do as they said: just upgrade all nodes, everything will go fine... i did it yesterday!". but... nothing. this really leave me... wordless.. literally-) btw, i'm still with the 1.7 cluster here (no test lab, unfortunately...),
but <joking> probably 2.0 will solve this kind of problem, so "when will be 2.0 ready? we're already in Q3!" :-D </joking>

thanks, anyway. when i'll upgrade my cluster, i'll report here anything that will happen, my experience...

marco
Hi,
perhaps you get not so much answers because:
1. the cluster-configurations are different
2. most things are in the wiki/forum.

to 1.: Depends on your storage - do you use local-storage, shared-storage esp. DRBD?
to 2.: Like wrote in the wiki - live migration work only with the same PVE-Version (and the same kernel). EG you can use live-migration to node b, upgrade node a, reboot node a and use offline migration from node b. After that upgrade node b (after that online-migration will work again).
This is perhaps a little bit different with DRBD (split-brain, resyncing...).

Udo
 
Thank you, udo, as always your posts and replies are really appreciated, insightful.
i understand there are so many different setups, and all, but i couldn't find anything anywhere. even in the wiki cluster page, where you could find haow to create, troubleshoot clusters, but nothing on how to upgrade. so today i added the (above) forum tom's response, as a starting point. Hopefully more info could be added there (maybe also linked from the "upgrade pve" page).

btw, i have shared iscsi storage. but i wasn't sure.

i still have doubts:
* you say ''you can use live-migration to node b, upgrade node a, reboot node a and use offline migration from node b. After that upgrade node b (after that online-migration will work again).''
* tom says ''Just update all nodes, nothing special.''

i don't know if it's me, but i can't understand why so different paths.

at my eyes, tom's path minimizes downtime. i read ''Just update all nodes, nothing special.'' as:
* stop all vms on the cluster.
* upgrade all nodes, verify packages, kernels, all.
* check cluster consistency after upgrade
* restart vms

so, no backups/restore? good.

i agree that it's probably my absolute low experience on both pve and pve clusters which is confusing me, but i think i am not alone.
clustering is a more complicated matter than single servers, and i think clear infos should be put on those wiki pages...

Thanks, Marco.
 
Last edited:
It is always a good idea to have backups, things can go wrong.

I use DRBD with only two nodes.

I usually update like this if I want to keep the disruption to each vm as short as possible:
1. Live Migrate all VMs to the master node
2. Update the slave node.
3. Reboot Slave node.
4. One by One turn off a VM, offline migrate to updated node, turn it on on new node.
5. After all vms are moved I updated the master node and reboot it.
6. Then I live migrate some of the vms back to the master node where they belong.

I have also done this backwards where I updated the master first and never had an issue.
But remember reading somewhere it is advised to update the slaves first then the master.

I have also updated like this:
1. turn off all the vms on a node.
2. install updates
3. reboot
4. let VMs autostart or manually start them
5. repeat process till all nodes are updated.
 
Thank you too e100, very much. Your contribute is much appreciated.
i kept thinking of both ways, but having past experience from other users is always good...

In your words, i can see no particular instruction related to your 2-nodes DRBD setup, isn't it?
is not my case, but could you give advices for DRBD users in the need of a cluster upgrade?

I'm feeding the wiki cluster page with infos here, so other users can benefit without digging through forum posts...

Thanks, Marco
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!