> Is it worth upgrading?
Well, PVE 5.x has reached the end of its support
lifecycle since 2020-07, you don't get security updates for PVE 5.x
If you are actually still on PVE 5.1, it could indicate that you haven't applied any updates for for > 2 years
(considering that 5.2 has been released on 2018-05).
A lot of (mostly Intel) CPU related bugs have been plugged since then as well, but also (to me) even small things such as the capability
of reloading the network changes without a reboot using ifupdown2 is very welcome, but the delta of improvements
on the
Roadmap is long between 5.1 and the current 6.3.
It even little fixes I've come to like since 6.0 so yes, it usually is worth upgrading, 6.0 has been released in 2019 and been given incremental additions leading to the current 6.3 release.
> I am super concerned about upgrading as I have looked at "what could go wrong" and it just feels like a whole heap of pain.
Well, at some point you will "have to bite the bullet", be it that you have to replace your current servers by newer hardware
not supported by the 4.15.x kernel shipped with PVE 5. The bigger the jump, the bigger the changes, the bigger the risk for
issues. ;-)
At least consider updating from PVE 5.1 to the latest 5.4 then check out the (actually good) Upgrade documentation for the upgrade from
PVE 5 to 6 - even then run 'pve5to6' which checks for a couple of pitfalls on the road to PVE6.
Since you are a Proxmox subscriber as per your badge, you have access to the PVE subscription repository which is being updated in a more conservative fashion, those packages are more heavily tested. Thus updates are usually safer to apply.