[SOLVED] Right approach w/2 enterprise ssd's?

[tweaks]

Read a *ton* of posts and analysis by @Dunuin & others about proxmox wearing out consumer ssd's and the implications of excessive writes due to zfs. I've got 2 initial setup questions based on this.

For a new promox svr setup... 32gb ram, managed to get 2 micron 7450 max 800g (4300 TBW). I’m now paranoid about TWB usage so figured best approach might be to have all the vm’s/ct’s on 1 ssd, and the promox server, iso’s, backups and all syslogging on the other ssd.

1. Would this be a smart approach to balance TBW on the drives?
2. Oh yeah, considering keeping it ext4 (as these are enteprise ssds) instead of zfs to keep wear down (dumb idea?). Am I obsessing about TBW too much?

Thanks for all your help.
PS - yes, I'm also doing the standard commands if I don't have HA clusters since its just 1 server :
systemctl disable --now pve-ha-crm.service systemctl disable --now pve-ha-lrm.service systemctl disable --now corosync.service systemctl disable --now pvesr.timer
and WRITE_TIMEOUT=3600 in /etc/default/rrdcached

 

[ness1602]

Since you only have 32gb ram, what will this server be used for? If you are using it for home VM/CT, than you could go with pretty much any disks.

 

[Dunuin]

You didn't specified the workload, but those two SSDs usually should last for years with a single ZFS mirror for everything.
I would just try it and have a look at the SMART attributes after a month. You can then calculate how much data got written to those SSDs in that month and you can extrapolate how long it should take until exeeding the TBW.

 

[tweaks]

homelab use (pihole, grafana, influxdb, prometheus, docker, ntop for pfsense, homebridge/homeassistant, assorted other minor services) but because of pihole, uptime is an issue - bottom line no customer production use. Bear in mind all of these services log a crapton (syslog). So if I go ZFS mirror, then I prolly need to aim my OCD against starting to think about tuning recordsize and here (aka noatime, ashift,etc) and account for a certain amount of ram gb for zfs alone?
bit rot aside, @Dunuin , do you still reccomend zfs (mirror) vs ext4 w/enterprise ssd's?

 

[Dunuin]

I would always recommend a mirror if you care about uptime or your data. And there aren't that much options with proxmox.
- HW raid in case you got raid card with ext4 or LVM-Thin (might be an option if you got the raid card and you don't care that much about data integrity)
- onboard pseudo-HW raid with ext4/LVM-Thin (not recommended because it combines the downsides of HW raid with the downsides of SW raid)
- ZFS (great data integrity but not best performance and high wear)
- btrfs (not recommended because experimental)
- mdraid (more hacky to install and not officially supported)

So I personally would stick with ZFS, especially when you already got the SSDs that can handle it.
Just keep in mind that raid is no backup and you shouldn't store your backups on the same pool your production data is on. So I would get another SSD as a PBS datastore (or even better put that in another machine for a dedicated PBS server if you got a spare machine).
And yes, disabling atime or at least enabling relatime will reduce some writes. Tuning recordsize will only help with LXCs as VMs will use the volblocksize instead. But the default 128K recordsize is dynamic and the default 8K volblocksize should be fine for a mirror.

 

[tweaks]

Ok now that makes sense. Thanks so much for the thorough response as well.

 

[tweaks]

@Dunuin I'm going to push you a bit on the ZFS mirror suggestion simply to clear something up in my head. Why use zfs mirror if I'm using enterprise ssd's, that are well cooled, and by using 2 separate ssd's allows me to separate the io's (and write usage) from vm/ct from the server itself (ie:logging, backups, administrative tasks)? I understand the risk of losing one ssd, but if I follow your recommendation to backup to some other pbs server, what would I gain from using a zfs mirror on enterprise ssds?
Asking just to clarify for discussion.

 

[Dunuin]

1.) less data loss, as you won't lose the data since your last backup
2.) a single disk got no parity data, so ZFS can't auto-heal itself (it will only tell you that data got corrupted, but can't fix it)
3.) server will continue running like nothing happened, so no downtime (might be offline for days if you need to order a new SSD first)
4.) less work, as you don't need to install and set up everything again
5.) A mirror also gives additional read performance (but not write performance). You PVE system disk will idle most of the time, as probably 99% of the IO will come from the VMs/LXCs. So its useful when the VMs/LXCs can make use the the read performance of both disks.

 

[tweaks]

ahhhh so
#2 is the kicker there - didn't realize it needed separate drive (redundant config) for parity to auto-heal
#4 good pt - anything to make life easier
awesome points thank you

 

[Dunuin]

#2 is the kicker there - didn't realize it needed separate drive (redundant config) for parity to auto-heal

You could tell ZFS to write two or more copies of everything then you would get bit rot protection with just one disk. But that would also mean double the wear and half the capacity. So not really a point unless your host can only fit one disk. Otherwise, it's always better to use a mirror because of better performance and redundancy.

 

[hassoon]

You could tell ZFS to write two or more copies of everything then you would get bit rot protection with just one disk. But that would also mean double the wear and half the capacity. So not really a point unless your host can only fit one disk. Otherwise, it's always better to use a mirror because of better performance and redundancy.

Thanks for the suggestion. can I ask you whats the case for CEPH?
Im thinking to use zfs raidz on 4x mx500 crucial ssd which seemed to work on our test server. but Im thinking to switch to gigastone enterprise ssdas their advertise themselves as virtualisation ready and nas ready.
Ive also use noticed kingston throwing a strange error when I use them for raidz.
so whats the best optimal case for ceph with ssd?

 

[UdoB]

Ive also use noticed kingston throwing a strange error when I use them for raidz.

You could have added that error message...

so whats the best optimal case for ceph with ssd?

What do you want to hear? Cheap is fine? Of course Enterprise SSDs are recommended. For each and every use case and for each and every Filesystem. Yes, that's purely an (my) opinion

If your Ceph has enough redundancy then you can choose cheaper devices. But this statement is fundamentally true for ZFS also!

And with "enough" I mean something like size=4/min_size=3 for a replicated_rule or k=3,m=2 for EC. (Just as an example.)

The first variant uses 4 Nodes, so would recommend to have at least five - to allow one to fail and see auto-repair in action. The later uses 5 Nodes so would recommend to have six...

Disclaimer: I am NOT a Ceph specialist...

 

[hassoon]

You could have added that error message...


What do you want to hear? Cheap is fine? Of course Enterprise SSDs are recommended. For each and every use case and for each and every Filesystem. Yes, that's purely an (my) opinion

If your Ceph has enough redundancy then you can choose cheaper devices. But this statement is fundamentally true for ZFS also!

And with "enough" I mean something like size=4/min_size=3 for a replicated_rule or k=3,m=2 for EC. (Just as an example.)

The first variant uses 4 Nodes, so would recommend to have at least five - to allow one to fail and see auto-repair in action. The later uses 5 Nodes so would recommend to have six...

Disclaimer: I am NOT a Ceph specialist...

Thx for the advice.
Its not about cheap, just thought that zfs was the most intense in terms of read and write while ceph is less intense and might tolerate.
my final conclusion is that Ive selected a generic brand with enterprise grade Cache 24/7 Durable TLC High Endurance which I think might be good for the purpose.