Questions about a new Proxmox setup

[newpain01]

I got a Dell Precision T3610 with an Intel Xeon E5-1620 v2 @ 3.70GHz and 96GB of ECC RAM that I'm planning to use for setting up my Proxmox server to run things like TrueNAS, Frigate, Plex, PiHole, HomeAssistant and various other containers and maybe 5-6 virtual machines. I'm planning to add 6-8 hard drives for the NAS later. I picked up a 1TB Crucial MX500 SSD that I was planning to use as the main drive to install Proxmox and the VMs and I'm trying to get some tips on the best way to set this up. I am thinking maybe 1TB ssd just for Proxmox is overkill?

• I read that Proxmox doesn't let you use the local-lvm partition to store data, so I am wondering what would be the best way to partition the SSD considering my use case? (I'm planning to store some container templates and maybe 5-6 ISO images, so probably 50GB tops) The default Proxmox installation allocated 100GB to root and the rest is for VMs. I guess that should be enough.
• Can I make backups of the whole Proxmox system to the NAS once I set it up or do I have to back it up to the SSD?
• Is it a good practice to use a single SSD for all of this, or is it better if I get 2 x 500GB SSDs in RAID 1 to use for the system? (I think 500GB should be enough for my needs and 1TB might be too much)
• I am wondering how to get additional 6-8 SATA power connectors from the Dell Precision T3610 power supply? The power supply is definitely capable of running that configuration, but not enough sata power cables. I am trying to avoid those Y splitters because I heard they can overwhelm the main wires and cause a fire, wondering if that is even true, considering the low voltage and current that hard drives use.

 

[noel.]

HI,
Here are my thoughts, I am, however, sure, that other people will have other opinions and give you other good layout ideas.

• local-lvm (which is lvm-thin) can't contain isos, yes. However, you can store container templates (and vms), see the storage docs. Also, besides local-lvm, there is a local, which is a directory at /var/lib/vz. This directory can contain ISO images.
• Backups: depending on how critical your data are, the more thought you should put into this. We offer a service called Proxmox Backup Server. If I were you, I would use a few of your drives (perhaps 2 in RAIDZ) exclusively for a Proxmox backup server, run that backup server as a vm in Proxmox VE. You can set up different backup frequencies and retention times for different vms on you Proxmox VE (e.g. not back up the plex server so often because the data on it are not as important). The best practice would be to have local backups as well as backups on a different machine (and ideally on a 3 machine outside your house and network, in case of, say, a house fire or a ransomware attack)
• Backing up Proxmox itself: Depending on how secure you want everything to be, running Proxmox on the single ssd you mentioned is fine, although RAID is obviously more secure. All you really need to back up are the /etc and /etc/pve directories (and the /var/lib/vz directory, if you keep container templates there). You can also back up the entire Proxmox VE. I use timeshift to back up my OS at home. So you could, for example, back up your PVE install once a week to an external drive.
• I don't know much about power connectors, I know there are also Molex to SATA cables, in case your powersupply has molex. I don't, however, know if people use these to power HDDs or just LEDs or fans.

Your best option is probably to play around with Proxmox VE a little bit and find out which configuration you are comfortable with/is safe enough for you.

I hope this was helpful

 

[Dunuin]

I looked once at the ratings of those SATA and molex power connectors. If I rememebr right, a SATA connector can handle the power for 2 disks and a molex conector the power of up to 4 disks. So good crimped Y-cables (the cheap non-crimped ones might melt) are a valid choice as long as you don't attach more than 2 disks to a SATA connector or 4 disks to a molex connector of the PSU.

And for the raid of the system disk. When your single system SSD dies (and it will sooner or later) your whole server will become nonfunctional. It might take days to buy a replacement disk and set up everything again. You will have to decide if it is ok for you when your server won't be able to run for some days. For me personally, this wouldn't be an option as I really need to access the data on that NAS as I don't store important data locally (Microsoft thinks win11 needs no bit rot protection...with ReFS not available for consumers...). Install PVE as a raid1 and you will save your time setting up PVE again, you will lose no data and everything will continue working with a failed disk. But keep in mind that PVE only supports ZFS raid and it's not recommended to use consumer SSDs with ZFS like your MX500. See for example here where people complain about the horrible performance with MX500:
https://forum.proxmox.com/threads/disk-ssd-performance-question.114727/
https://forum.proxmox.com/threads/bug-iowait-on-proxmox-7-2-11.116057/#post-502027
https://forum.proxmox.com/threads/backups-in-proxmox7-slow-restore-high-io-on-ssd.116011/post-501809
https://forum.proxmox.com/threads/ssd-setup-for-vm-host.109703/

 

[newpain01]

Thank you both for the information, it's very helpful indeed!

Regarding Molex connectors, this power supply doesn't have any. There are 2 extra 6 pin wires (typically used for additional power on GPUs), but I believe those are 12V and cannot be used for SATA. I think my only option here is to use SATA to SATA Y-cables. I currently have 3 SATA power cables, so I'll have to split them a couple of times to get 8 SATA power connectors (I don't like the idea - too many splitters)

@Dunuin Thanks for the info on the SSD. I wasn't aware the MX500 has those issues, I had Lexar NS100 previously and those are pretty bad and do not have DRAM, so I thought maybe I should "upgrade" to MX500 for the Proxmox setup. The MX500 has a 5 year warranty, so I thought I will be "safe", but you are right, I don't want to be with a nonfunctional server for days. I still can't decide on space though.

Is there any list of decent SSDs for PVE? Any recommendations on what to look for?

 

[Dunuin]

Thank you both for the information, it's very helpful indeed!

Regarding Molex connectors, this power supply doesn't have any. There are 2 extra 6 pin wires (typically used for additional power on GPUs), but I believe those are 12V and cannot be used for SATA. I think my only option here is to use SATA to SATA Y-cables. I currently have 3 SATA power cables, so I'll have to split them a couple of times to get 8 SATA power connectors (I don't like the idea - too many splitters)

Yes, a SATA connector can only handle the power of two disks. So max of 6 disks with your 3 connectors if you don't want to go over the specs.

@Dunuin Thanks for the info on the SSD. I wasn't aware the MX500 has those issues, I had Lexar NS100 previously and those are pretty bad and do not have DRAM, so I thought maybe I should "upgrade" to MX500 for the Proxmox setup. The MX500 has a 5 year warranty, so I thought I will be "safe", but you are right, I don't want to be with a nonfunctional server for days. I still can't decide on space though.

Is there any list of decent SSDs for PVE? Any recommendations on what to look for?

I would buy something like two Intel S4510/S4520 or Samsung PM883/PM893 or Micron 5200 Pro/5300 Pro/5400 Pro for small workloads. Or two Intel S4610 or Samsung PM897 or Micron 5300 Max/5400 Max for heavier workloads.

And the MX500 warranty is 5 years OR reaching the TBW of 360TB. If you write more than 360TB before the 5 years are over the warranty is still lost. And 360TB sounds like much, but it really isn't. Let's say you got a write amplification of factor 20, like me. Then writing only 18TB inside a VM will result in 360TB written to the SSDs NAND. And 18TB isn't much over 5 years. 18TB divided by 5 years is 114 kb/s. So with a factor 20 write amplification, you can't write with more than an average of 114kb/s or you will lose the warranty. And 114kb/s isn't much. Let's say you got 10 VMs, then each VM might only write with 11kb/s. That's easily achieved by just writing logs.

 

[newpain01]

Perfect, thanks for the info. Again, I wasn't aware the warranty has that TBW part, never had a drive fail on me, but I get your point, it doesn't require much to get there. What is the amplification factor of 20?

Any experience with the Samsung SM863 or the Dell OCZ Intrepid 3800? I can get those for a pretty good price locally and they are an enterprise grade hardware, even though lower grade than the one you listed.

Another stupid question maybe, I see that the SAMSUNG SSD 860 PRO has a pretty hight TBW of 4800, even though it's a consumer SSD. Does that mean it's a good choice for this use case or it's only only the TBW that we care about?

What would be the downside of using ext4 with a single consumer grade ssd and backing up the whole Proxmox system daily to an external device? That way, if the ssd fails, I swap a drive, do a fresh Proxmox install and restore the backup from the external device, am I right or are there any other gotchas that I’m overseeing?

Funny thing I watched the other day, this guy is using the cheapest dram-less $20 Kingston SSDs in RAID1 with Proxmox. I've had bad experience with those particular SSDs in the past, so I figured the MX500 must be significantly better.

 

[Dunuin]

Another stupid question maybe, I see that the SAMSUNG SSD 860 PRO has a pretty hight TBW of 4800, even though it's a consumer SSD. Does that mean it's a good choice for this use case or it's only only the TBW that we care about?

TBW is higher because it uses higher quality NAND chips (MLC) but sync write performance should still be worse as it is missing the powerloss protection enterprise SSDs got.

What is the amplification factor of 20?

https://www.tuxera.com/blog/what-is-write-amplification-why-is-it-bad-what-causes-it/

What would be the downside of using ext4 with a single consumer grade ssd and backing up the whole Proxmox system daily to an external device? That way, if the ssd fails, I swap a drive, do a fresh Proxmox install and restore the backup from the external device, am I right or are there any other gotchas that I’m overseeing?

You can't back up the whole system daily (or at least that would be very annoying), as a block-level backup requires you to shut down PVE, reboot into some live system, then do a block-level backup of the whole drive, and then boot into PVE again. What could be done are automatic backups of your PVE config files and guests. But then you still lose a day of data if the SSD dies with probaly some days of downtime. And with something like ext4 you got no bit rot protection, so your data may silently corrupt over time. Without this bit rot protection/detection, when you store a file on an HDD/SSD and access it 1 year later, it isn't guaranteed that the file is still the same. So yes, you can have backups, but you never know if you are backing up already corrupted data.

 

[newpain01]

I knew I must be missing something there, thanks for explaining.

Any insight on the drives I linked above?

 

[newpain01]

@tom I read in another thread that you run a lot of SM863 with good results, is that still the case? Any insight on those drives?