Hi,
We run a 20-node production cluster, with just over 1100 VMs. Hosts are typically Intel XEON Scalable processors, with between 768GB-1TB RAM, and between 1-2 RAIDZ3 SSD pools per host - with roughly 10TB usable per pool.
We do not have separately installed OS pools i.e. the primary/first zpools are installed and bootable with PVE and have running VMs.
We are struggling to develop a sustainable strategy and practice around keeping PVE hosts suitably updated, and are erring on the side of caution by doing live VM migrates to either updated or newly installed hosts whenever we choose to run through update cycles.
This clearly isn't sustainable and the more we grow, the more difficult it becomes to run through these cycles.
Some questions (and we're just looking for some 'comfort' - mostly..
)
1. Is it worthwhile/beneficial - especially in the context of doing updates on hosts running live VMs - to have a separate OS/PVE zpool installed apart from the zpools that have running VMs?;
2. How much risk is there in updating hosts without shutting down VMs - and is any risk lessened by having a separate OS pool?;
3. Is there any value in using something like Kernelcare to deal with issues around having to 'minimise' the requirement for a host reboot?;
4. If Kernelcare is an option, is it safe and stable? And (importantly) doesn't the use of Kernelcare potentially cause a race condition/conflict between Kernelcare updates and kernel updates using the PVE updating process?
5. How is everyone else doing updates in reasonably dense and busy production environments? We can't be the only ones facing this dilemma?
Really looking forward to some insights here..
Angelo.
We run a 20-node production cluster, with just over 1100 VMs. Hosts are typically Intel XEON Scalable processors, with between 768GB-1TB RAM, and between 1-2 RAIDZ3 SSD pools per host - with roughly 10TB usable per pool.
We do not have separately installed OS pools i.e. the primary/first zpools are installed and bootable with PVE and have running VMs.
We are struggling to develop a sustainable strategy and practice around keeping PVE hosts suitably updated, and are erring on the side of caution by doing live VM migrates to either updated or newly installed hosts whenever we choose to run through update cycles.
This clearly isn't sustainable and the more we grow, the more difficult it becomes to run through these cycles.
Some questions (and we're just looking for some 'comfort' - mostly..
)1. Is it worthwhile/beneficial - especially in the context of doing updates on hosts running live VMs - to have a separate OS/PVE zpool installed apart from the zpools that have running VMs?;
2. How much risk is there in updating hosts without shutting down VMs - and is any risk lessened by having a separate OS pool?;
3. Is there any value in using something like Kernelcare to deal with issues around having to 'minimise' the requirement for a host reboot?;
4. If Kernelcare is an option, is it safe and stable? And (importantly) doesn't the use of Kernelcare potentially cause a race condition/conflict between Kernelcare updates and kernel updates using the PVE updating process?
5. How is everyone else doing updates in reasonably dense and busy production environments? We can't be the only ones facing this dilemma?
Really looking forward to some insights here..
Angelo.