That is an option if you don't need high availability but keep in mind that your VMs aren't protected against data corruption like bit rot if you are not using ZFS/CEPH. Your backups won't help you to restore corrupted files, because without the ability to scrub you will never know if a file is already damaged before you back it up. So its not unlikely that all your backups will only contain corrupted versions of your VMs.
Yes, I guess so, but since the actual data (files, media and whatnot) used by the VMs/LXCs would be stored on the HDD ZFS pool would it matter if you got a corrupted file on a VM or LXC? In that case it would most likely be a "system" related file which should not be that critical anyway as you could always rebuild the VM or LXC container as long as the data is intact on the storage pool.
The question then is, where do you put the DBs? On the SSD where data can get corrupted or on the secure HDD ZFS pool what will be hit hard by all the sync IOPS. Even if you don't run a webserver with MySQL, most programs will use some kind of databases in the background to store their stuff.
The DBs would then be on the SSDs as part of the VM data, which could get corrupted yes, but probably nothing that could not get rebuilt quite fast I think. As much complexity using ZFS on the VM drive seems to introduce to me it feels like a bigger risk as long as one cannot afford very expensive SSD / NVMe drives and have extensive knowledge on how to configure the Proxmox/ZFS/VMs correctly for that type of setup.
You are not the first!
Although I must say, since I have no experience with this myself yet, I am getting really confused based on the feedback I am getting. There are equally many people who states that this is not an issue as there are people preaching doomsday scenarios if using consumer SSDs. I like to be at the cautious side, but those enterprise drives can set you back quite a bit depending on your setup. I've got replies from several running with consumer Samsung SSDs with ZFS on heavy load servers for several years and stil have a bunch of lifetime left on the SSDs afterwards, so it is difficult to know what to believe without first hand experience.With regards to enterprise SSD I actually thought it was the opposite when it comes to speed. I.e. that consumer drives are typically faster, but with lower lifetime than enterprise SSDs, but maybe I have misunderstood. Maybe someone can elaborate?
That all really depends on your workload and setup. My homeserver is running 20 VMs and these are writing 900GB per day while idleing where most of the writes are just logs/metrics created by the VMs themself. Sum that up and a 1TB consumer SSDs TBW will be exeeded within a year.You are not the first!
Again, its all about the workload. Consumer SSDs are great if you need small bursts of sequential async reads/writes but the performance will drop massively as soon as the cache is filled up because of long sustained loads. In such a case a enterprise SSD will be much faster because the performance won't drop that hard. And most consumer SSDs won't be able to use caching at all for sync writes, so here the performance is always horrible.
