Full ack, this is what I wanted to write. In addition to rrdcache, the /etc/pve filesystem, or more precisely the sqlite database behind it also writes a lot.
It is totally useless to use SSD for the PVE system alone (excluding swap). The system is normally once boots and runs for weeks maybe month if you apply all kernel updates directly (and reboot) and everything you need will be in the cache. So the benefit of having SSD is negligeable.
If you have on the other hand swap and use it a lot, you will have even higher wearout and not a fast system, because of the low-end SSD devices.
If it has to be two SSDs for the OS, just buy two used 120 GB Samsung Enterprise SSD from ebay, which are cheaper and more reliable than the EVOs.
Main problem is the write amplification that is usually worse on consumer SSD. Especially if there is no powerloss protection so sync writes can't be cached/optimized. PVE actually isn't wirting that much. Lets assume its some hundret MBs per day of real data. But its doing alot of small writes (metrics, logs, writing the cluster configs each minute, ...) and if a 4KB IO causes for example a 128k write to the NAND these hundrets of MB per day easily gets amplified to something like dozens of GBs per day. But even then that shouldn't be such a big problem. The SSDs are rated for 150TB TBW over 5 years. So (150*1000 GB TBW)/(5*365 days)=82GB per day. So as long as there isn't more than 82GB written to the SSDs NAND per day your SSD should survive its 5 year warranty.
And by default there is no swap when using ZFS. But ofcause, if you add swap to that OS disks that will highly increase the wear.
Just monitor your SSDs SMART stats and look that your writes are less than 82GB per day.
Well these SSDs are consumer grade, so no powerloss protection indeed, and the prospect of corrupting the OS disks in case of a power failure would be a real pain. Maybe I should go for a pair of SAS HGST 600Gb for the job? I know ... 600Gb would be overkill, but in this case I could put an ISO storage in there.
I'm just looking for the OS part, since the VMs will run from mechanical disks for the time being. When time and budget arrives, I will create a new ZFS on enterprise SSDs and move the VMs there. If it were today, I would be thinking about some new PM897 480Gb disks.
I'm also taking the chance to look into SSDs on ebay. I don't know about your experience, but I have none buying SSDs online, and I was keeping out just because we never know what usage they went through.
Anyway, at the moment I found 2 products that seem a possibility, can anyone give me an opinion? I'll learn something from it, that's for sure
- Samsung PM883 MZ-7LH2400 240GB
Buying 2x plus import and postage charges, it would set me at around 80€
- Samsung SM863 MZ-7KM1200 120GB
Buying 2x plus import and postage charges, it would set me at around 115€
From what I looked for, they both have power loss protection. The SM863 is a generation older than the PM883, but also have higher endurance:
- SM863 - DWPD 3.5 / 5 years
- PM883 - DWPD 1.3 / 3 years
Even though the SM863 should be better on paper than the PM883, the SM863 is older and thus worked for more time (potentially) and already reached the 5 year warranty.
No more information, besides some specs about the disks, which can be found around the web. I could ask the seller, but don't know if anyone would be willing to run some tests or provide some smart data.
Thank you.
