Setup help (easy on SSD health)

[Blockie]

Hey,

I looked around a lot before but I'm still quite confused. I have an HP EliteDesk I wanna use for a bit of data backup (probably TrueNAS or OpenMediaVault), photo storage, self-hosted music and maybe Jellyfin. For that I bought two 2TB NVMe drives. Now, I'm guessing I want to run this in a raid configuration and I also guess I should do this in the Proxmox installation rather than later on - if that even would be possible. I also have an additional SSD I could plug in. (There are 2 NVMe Slots and one Sata for SSD)

I don't really want to wreck my drives though (as they are relatively expensive and I've read that Proxmox does a lot of write operations when mirroring(?) the installation), so any advice which config I should pick? Or should I put the Proxmox installation and the VMs on the single, other one? I'm not 100% sure what I'd have to do for that in the installation then though. I feel like SSDs should last at least 10 years - especially when it's such a simple setup, no? My NVMe drives are good for 1200TBW, so they shouldn't fail at all actually. Still, I've read that somehow people still have a lot of issues in that regard. Of course other people also say they have no issues at all, I'm so confused. Do I have to keep something in mind with the configuration then?

Ah, and yes, I feel like 2TB (if Raid) is enough for the type of data I want to have safe, unless I'm missing something.

I'm a total beginner with this stuff, so I hope you can help me!
Also, I'm sorry if I wrote something really wrong or dumb, I'm learning!

 

[leesteken]

I have an HP EliteDesk I wanna use for a bit of data backup (probably TrueNAS or OpenMediaVault), photo storage, self-hosted music and maybe Jellyfin. For that I bought two 2TB NVMe drives. Now, I'm guessing I want to run this in a raid configuration ...

Why won't you simply share the make and model of the drives? Maybe they are perfect for ZFS (PLP) or maybe they are terrible (QLC). Do you need the most space available (and no redundancy) or did you buy two to use in a mirror (RAID1) to prevent the service from going down when one drive fails (sooner or later)?
I assume that your "bit of data backup" is a secondary copy (w.r.t the source like mobile phone etc.) but will you also have multiple and off-site backups of those drives (using PBS for example) to some kind of other storage? Please note that RAID is not a backup and they serve different purposes.

 

[Blockie]

Why won't you simply share the make and model of the drives? Maybe they are perfect for ZFS (PLP) or maybe they are terrible (QLC). Do you need the most space available (and no redundancy) or did you buy two to use in a mirror (RAID1) to prevent the service from going down when one drive fails (sooner or later)?
I assume that your "bit of data backup" is a secondary copy (w.r.t the source like mobile phone etc.) but will you also have multiple and off-site backups of those drives (using PBS for example) to some kind of other storage? Please note that RAID is not a backup and they serve different purposes.

Sorry, I didn't know that the model may have a big impact. I have the Exceria Plus G3 which I guess is not PLP as they do not have DRAM.

To be honest, I just kind of read that everybody just goes for a raid configuration. I figured, it's because it would be way more of a hassle to replace a failed drive otherwise. If restoring proxmox from a backup is not that bad, I could also go with that. Although I've read it could take a significant amount of time.
I did buy the two expecting to use them in a mirror, correct. But for me RAID is a little bit like a "backup", as in: When one drive fails, all is well. For an actual backup I'd use an external drive or something like that. I currently don't care enough to also have an off-site backup - I wouldn't even know how to start with that. And yeah, data backup is from my phone and PC. I think I just use the word backup a little bit wrong. ^^

 

[leesteken]

Sorry, I didn't know that the model may have a big impact. I have the Exceria Plus G3 which I guess is not PLP as they do not have DRAM.

At least they are TLC and compared to my Micron 7450, they are not too bad. I think ZFS will be fine (since you are not running a production database), but I am just a random strange on the internet (with no guarantees). Since you don't have PLP, a UPS would be recommended since a power loss during trimming could be a disaster causing corruption on both drives. A cheap SSD (fast reads) combined with a HDD (dependable writes) stands a better chance than two identical SSDs (without PLP), ironically.

To be honest, I just kind of read that everybody just goes for a raid configuration. I figured, it's because it would be way more of a hassle to replace a failed drive otherwise. If restoring proxmox from a backup is not that bad, I could also go with that. Although I've read it could take a significant amount of time.

Yes, keeping things running even when parts are failing does save a lot of time.

I did buy the two expecting to use them in a mirror, correct. But for me RAID is a little bit like a "backup", as in: When one drive fails, all is well. For an actual backup I'd use an external drive or something like that. I currently don't care enough to also have an off-site backup - I wouldn't even know how to start with that. And yeah, data backup is from my phone and PC. I think I just use the word backup a little bit wrong. ^^

At least you will have a second copy,but if you accidentally remove a file, it could easily be lost forever (on all copies).

 

[Kingneutron]

> But for me RAID is a little bit like a "backup", as in: When one drive fails, all is well

Nope, RAID is not considered a backup. It's redundancy, and helpful for uptime. I just helped another guy where both mirror disks died/got corrupted.

You need to postulate for worst-case, not hope for the best.

Nothing is a replacement for a proper backup. You cannot be too paranoid, this is why we keep drumming the 3-2-1 backup march.

https://serverfault.com/questions/2888/why-is-raid-not-a-backup

 

[Blockie]

At least they are TLC and compared to my Micron 7450, they are not too bad. I think ZFS will be fine (since you are not running a production database), but I am just a random strange on the internet (with no guarantees).

Ok, now I got to know: What to people usually do with Proxmox? Like, with me having just a photo server/NAS/Spotify Alternative Self-Hosted/Jellyfin, there wouldn't be constantly write operations except that the OS itself would do, right? Unlike having some kind of databases.
I've also read there is some stuff you can disable to lower the wear even more. I found cryptic stuff like "swap disabled and vm.swappiness set to 0" for example. (I'll figure it out when I get there)

Since you don't have PLP, a UPS would be recommended since a power loss during trimming could be a disaster causing corruption on both drives.

I don't know how I missed this, but I've never read about a UPS so far. Also gotta be honest: I can't really afford one. The price of the unit itself seems fine, but the power consumption? 10-20W is a lot, especially considering the system itself will use less than that. That's 25-50€ per year. I probably could almost pay just for cloud storage with that money.

I guess (or rather hope), this scenario only is that likely when using RAID 1 (or any RAID)? Meaning, I probably just want to go with Raid 0 (if RAID at all) and just back up everything externally. Restoring from backup can't be that bad and I can live with the downtime.
Sorry if I'm being difficult.

Nope, RAID is not considered a backup. It's redundancy, and helpful for uptime. I just helped another guy where both mirror disks died/got corrupted.

You need to postulate for worst-case, not hope for the best.

Nothing is a replacement for a proper backup. You cannot be too paranoid, this is why we keep drumming the 3-2-1 backup march.

https://serverfault.com/questions/2888/why-is-raid-not-a-backup

You are right, I didn't think about how much the drives "work together" then and how much more likely (or generally how likely) it is that they both die/corrupt. I did intend to have a backup for those as well though.

 

[Kingneutron]

> I don't know how I missed this, but I've never read about a UPS so far. Also gotta be honest: I can't really afford one. The price of the unit itself seems fine, but the power consumption? 10-20W is a lot, especially considering the system itself will use less than that. That's 25-50€ per year. I probably could almost pay just for cloud storage with that money

Dude, seriously? 50Euro per year is like 4Euro and change per month, for better odds on your data surviving a power event. Ever tried restoring a few TB of data from the cloud instead of local storage? Do you have any idea how long it would take, and the fact that it would basically Bogart your Internet connection the whole time?

Don't nickel-and-dime yourself when it comes to protecting your computer equipment.

 

[leesteken]

Ok, now I got to know: What to people usually do with Proxmox? Like, with me having just a photo server/NAS/Spotify Alternative Self-Hosted/Jellyfin, there wouldn't be constantly write operations except that the OS itself would do, right? Unlike having some kind of databases.
I've also read there is some stuff you can disable to lower the wear even more. I found cryptic stuff like "swap disabled and vm.swappiness set to 0" for example. (I'll figure it out when I get there)

Proxmox logs a lot or little messages and that works better on a old HDD than a consumer SSD. Proxmox is designed as a enterprise cluster hypervisor, not for a small low-power consumer mini PC. And you'll be putting multiple OSes worth of writes on one drive. Search the forum a bit on how to reduce or what good drives are.
That said, I believe your drives will probable be fine.

I don't know how I missed this, but I've never read about a UPS so far. Also gotta be honest: I can't really afford one. The price of the unit itself seems fine, but the power consumption? 10-20W is a lot, especially considering the system itself will use less than that. That's 25-50€ per year. I probably could almost pay just for cloud storage with that money.

I guess (or rather hope), this scenario only is that likely when using RAID 1 (or any RAID)? Meaning, I probably just want to go with Raid 0 (if RAID at all) and just back up everything externally. Restoring from backup can't be that bad and I can live with the downtime.

SSDs (or RAID controllers) without a power backup are more sensitive than HDD about losing writes and metadata. This is independent of RAID configuration (or no RAID). I just want to warn you that a mirror/RAID1 does not prevent this scenario because the drives are writing and trimming at the same time. And that's why a RAID is no backup.

Using an separate (off-site/cloud) backup is indeed also a good strategy. Just run the hardware as hard and dangerous as possible as restore from backup if anything breaks.

You are right, I didn't think about how much the drives "work together" then and how much more likely (or generally how likely) it is that they both die/corrupt. I did intend to have a backup for those as well though.

If you run a stripe/RAID0, either on ZFS or LVM, then you might lose everything when one drive fails. It was not clear to me before that you intend to have backups for your Proxmox and VM/CTs. Make sure to have to backups from the start and not later when it's more convenient, and test them before you desparatly need them.

 

[Blockie]

Dude, seriously? 50Euro per year is like 4Euro and change per month, for better odds on your data surviving a power event. Ever tried restoring a few TB of data from the cloud instead of local storage? Do you have any idea how long it would take, and the fact that it would basically Bogart your Internet connection the whole time?

As I've written in my first post, this is just for my home use, hosting some music streaming, photo storage and some data storage. It's not the end of the world if at some point it will be down for a few hours. Backup is just on an external drive, I can live without an offsite storage for now. For me, 50€ is a considerable amount. I'd rather spend a few hours restoring the server from a backup - in the unlikely event of the SSD dying, than paying that much per year. Power outages are an extraordinary rare thing - maybe once a year at most, and for my PC for example (even when it regularly happened because of my dad), it was never an issue. I'm aware a Proxmox server is something different, but I can't imagine that it will always corrupt every time there is a power loss. There is just a (for most people not ignorable) chance, that it may happen, no?

Proxmox logs a lot or little messages and that works better on a old HDD than a consumer SSD. Proxmox is designed as a enterprise cluster hypervisor, not for a small low-power consumer mini PC. And you'll be putting multiple OSes worth of writes on one drive. Search the forum a bit on how to reduce or what good drives are.
That said, I believe your drives will probable be fine.

I'll look around for that, yeah. I've just seen a ton of people just use this for what I want to use it as: Just a little NAS and Hosting of some Media.
I've just seen that there is this high write amplification and I'm wondering how bad it really would be in my case. Like, one guy said he has around 45GB written per day, isn't that a lot? Sadly, people never say what they actually do on their system so it's hard for a beginner like me to find out how that relates to my setup. Let's say, I have my system setup and there's like 5GB of photos, 5GB of documents, 100GB of music, and like 1TB of movies. This would equate to 3 VMs I suppose: Navidrome, Jellyfin, TrueNAS/OMV.
Of course there are also a bunch of people who dont have such issues, just hard to figure out what their differences are,

All I'm doing as the end user now, is just listen to some music. How much GB would this setup write per day? I am aware you won't really be able to tell and give an exact guess, but why would it be that high? It's not doing anything that requires writing except OS stuff.
And one important question: Backups are smart, right? They don't always just take everything that is there and back it up, it actually is checked what is changed and then it's backed up right? I can't imagine it being any other way, otherwise the offsite backup would take a looong time lol.


Still, what I've taken from all that, is that I will:
- stick with EXT4 and not bother with RAID, as the advantages of it are negligible for me and the additional stress on the drives is not good
- use a normal SATA SSD for the installation to avoid putting the load on the NVMe drives. (or does this not matter?)
- search for configurations that further lower the strain on the drives
- use an external drive for data backup (at some point add off-site backup)

Thanks for your help and clarifications! If I still got something wrong, please tell me.

Short TL;DR if I wrote too much:
- Power losses don't always corrupt/destroy SSDs, right?
- How much GB does a Setup with Navidrome (music), Jellyfin and TrueNAS/OMV write per day without adding new media? (broad estimate)
- Do backups check what's changed (+ parity check of course) and only back that up? Or is everything backed up?

 

[leesteken]

- stick with EXT4 and not bother with RAID, as the advantages of it are negligible for me and the additional stress on the drives is not good

It will probably be fine. Don't worry too much and just try and learn.

- use a normal SATA SSD for the installation to avoid putting the load on the NVMe drives. (or does this not matter?)

It's both solid state storage, so it does not matter much. As I said a SATA HDD would be good enough but your want SSD (preferably NVMe) because of the many random-read IOPS for the VMs. But your usage is probably quite low.

- search for configurations that further lower the strain on the drives

Use a old fashioned rotating HDD (but not SMR) but you'll probably be fine with your NVMe drives.

- Power losses don't always corrupt/destroy SSDs, right?

It's not uncommon to corrupt the filesystem(s) on a consumer SSD when power fails but not always. Use an old fashioned rotating HDD (but not SMR) if you have power failures often.

- How much GB does a Setup with Navidrome (music), Jellyfin and TrueNAS/OMV write per day without adding new media? (broad estimate)

I really don't know. You'll probably be fine with your chosen NVMe drives.

- Do backups check what's changed (+ parity check of course) and only back that up? Or is everything backed up?

Proxmox always backups everything each time. They have a nice open-source product named Proxmox Backup Server (PBS) that automatically deduplicates backups and can very quicky backup VMs (from Proxmox VE).

 

[Dunuin]

My NVMe drives are good for 1200TBW, so they shouldn't fail at all actually.

Here at home that would last for 3-4 years until those 1,200TB TBW are exceeded.
1,200 TBW isn't much. There are smaller SSDs that are rated for 282,600 TB TBW.

Ok, now I got to know: What to people usually do with Proxmox? Like, with me having just a photo server/NAS/Spotify Alternative Self-Hosted/Jellyfin, there wouldn't be constantly write operations except that the OS itself would do, right?

Usually everything is doing contant writes. Maybe not that much, but this easily adds up when running lots of VMs/LXCs. Lets say you got 1200 TB TBW and you want it to last for 10 years.
(1200 * 1000 * 1000 * 1000 * 1000 bytes TBW) / (10 * 365 * 24 * 60 * 60 seconds) = 1,200,000,000,000,000 bytes TBW / 315,360,000 seconds = 3.8 MB/s. So you are only allowed to write continuously with 3.8 MB/s to those drives if you don't want to exceed that TBW within 10 years. And that is without any write amplification of virtualization, nested filesystems and so on. Lets say your write amplification is factor 3 so 3.8 MB/s / 3 = 1.267 MB/s. Now you are only allowed to write with 1.267 MB/s as this gets amplified to 3.8 MB/s of writes to the SSDs. And then lets say you are running 10 VM: 1.267 BM/s / 10 VMs = 0.1267 MB/s. So each of those 10 VMs is only allowed to constantly write with an average of 126 KB/s. And writing 126KB/s is easily done.

As I've written in my first post, this is just for my home use, hosting some music streaming, photo storage and some data storage. It's not the end of the world if at some point it will be down for a few hours. Backup is just on an external drive, I can live without an offsite storage for now. For me, 50€ is a considerable amount.

If you take your data seriously, you also use a UPS at home. And yes, here at home it adds another 15W.

in the unlikely event of the SSD dying

Not unlikely. My homeservers are killing a few each year. So yes, as already recommended, raid1 + local backups + offsite backups would be a good idea.

Power outages are an extraordinary rare thing - maybe once a year at most, and for my PC for example (even when it regularly happened because of my dad), it was never an issue.

Search this forum for all the "HELP! Got a power outage and now PVE won't boot anymore. Have no backups. How to rescue my data..." threads. There are tons and I stopped counting them. Point of the UPS is not so you could use it in case of an power outage. You should install some service like NUT that will monitor your UPS and once a power outage is detected it will automatically initiate a proper shutdown of the server to prevent data loss while the server is running on battery for a few minutes.

I'm aware a Proxmox server is something different, but I can't imagine that it will always corrupt every time there is a power loss. There is just a (for most people not ignorable) chance, that it may happen, no?

There will always be some data loss. A computer stores data in its volatile write cache that will be wiped once there is no power. But no, not always problematic if you for example only lose some temporary data. It's gambling without an UPS and you will have to decide how important those pictures and so on are to you and if you want to risk losing them. Also keep in mind that data recovery companies charge you many hundreds or even thousands of dollars to rescue data. With that in mind such an UPS + raid1 + proper onsite+offsite backups are not that expensive.

And one important question: Backups are smart, right? They don't always just take everything that is there and back it up, it actually is checked what is changed and then it's backed up right? I can't imagine it being any other way, otherwise the offsite backup would take a looong time lol.

It depends. If you don't setup another server as a PBS you will have to use the built-in VZDump backup archives and that one will backup everying each time. So if you backup a 1TB VM daily for two weeks you got 14TB of backup archives.

- How much GB does a Setup with Navidrome (music), Jellyfin and TrueNAS/OMV write per day without adding new media? (broad estimate)

My homelab writes over 1TB per day while no one is using it (so idling doing nothing). Really depends what you do. Databases are especially bad when it comes to write amplification and basically everything is using some kind of database (OMV, PVE, Jellyfin, .. too).
It helps to shut down the server or at least stopping the VMs when not using them. Saves electricity and hardware wear. But yes, not very convenient.