PVE hosts package upgrade - Best practice and tips?

[monotroska]

Hello everyone.

Have a handful of PVE hosts running in a cluster. I was wondering how everyone approached everyday upgrades?

Do you wait for kernel or major PVE upgrades to be available, migrate all the VM's, upgrade and full restart?
Or do you upgrade smaller packages for the hosts as they become available? In that case, do you migrate the VM's or simply let them run?
Got one host you test upgrades on first?
Are there any packages you wont update while VM's are running? For example qemu, pve-manager or libc or something else?
What about if your running containers? More focus on upgrades in that case?

Look forward to hearing some inputs on how everyone else manages their upgrades. Any tips on making the task fast and secure?

Best regards,
Mono

 

[Dunuin]

There is a script from tuxis that will migrate the VMs to other nodes and later back. Maybe this could be helpful in an upgrade script? Migrate all VMs, do upgrade, reboot, migrate VMs back.

And yes, a test server is useful, but would be best with a homogenious cluster. Wouldn't help much if a kernel upgrade for example screws up some drivers but the test server is using different hardware so you don't encounter a problem.

And a subscription of cause would help, so you aren't betatesting for enterprise repo users

 

[LnxBil]

Or do you upgrade smaller packages for the hosts as they become available?

Depends on the severity of the security implications.

In that case, do you migrate the VM's or simply let them run?

Again, depends on the update. If this is a PVE QEMU update, it is different from e.g. recent OpenSSH patches.

Are there any packages you wont update while VM's are running? For example qemu, pve-manager or libc or something else?

Yes and it is very hard to find out which.

 

[monotroska]

There is a script from tuxis that will migrate the VMs to other nodes and later back. Maybe this could be helpful in an upgrade script? Migrate all VMs, do upgrade, reboot, migrate VMs back.

And yes, a test server is useful, but would be best with a homogenious cluster. Wouldn't help much if a kernel upgrade for example screws up some drivers but the test server is using different hardware so you don't encounter a problem.

And a subscription of cause would help, so you aren't betatesting for enterprise repo users

Thank you for the link. Looks like a great script. We will certainly look into that in order to make our tasks easier.
Valid point about being betatesters

Depends on the severity of the security implications.


Again, depends on the update. If this is a PVE QEMU update, it is different from e.g. recent OpenSSH patches.


Yes and it is very hard to find out which.

Thank you for the advice. I agree there are no black and white. Its always on a case by case basis in the end.

Best regards,