Hi,
having the OS on a fast enterprise SSD has advantages among:
* a lot more IOPS than on a spinning disk - or any other slow medium. Nice, among others, for our configuration filesystem's database
* all things go faster - package updates (also a lot of fsyncs)
* under heavy load, IO bottlenecks may become bad having the OS and possibly it's SWAP on a fast (bandwidth, latency and IOPS, wise) medium may help quite a bit here
And yeah, often mainboards have now something like 1 M.2 slot and 6 or even 12 SATA 6Gbps ones. For a Ceph hosting server it could make sense to just add the 6 OSDs and use a smaller (<256GB) M.2 NVME as OS, that's simply convenient - IMO. Maybe using the remaining fast space also as temporary backup store to speed up backups, etc.
That said, it clearly isn't a must. Most executables are in memory quite quick after booting, the pmxcfs may profit from faster IOPS but it makes only problems on really really slow stuff (e.g., USB keys) or on big setups (> ~12-16 nodes) with resource usage on the limit of maxing everything out.
PVE definitively works fine from a spinning disk. Try at least to use server-class enterprise hardware, it improves reliability and reduces headaches.
We just try to avoid the case where people use slow hardware and wonder why things are slow, which pops up more often than one would think.
We also recommend fast hardware because Admins can then tell their manager that it's an requirement with a "proof", and make their life easier to get fast enough hardware instead of barely fast enough ones which, when adding maintenance cost and increased wait times for various operations, costs even more in the long run.
