New(bie) storage configuration (RaidZ1 with 3HDDs + SSD)

[ProLog]

Hello Forum,

I am planning to use Proxmox for a new server/workstation in a lab/research environment, though I am still undecided about the right storage config, as I am still quite new to ZFS and VMs in general. Pretty much set are 3x 6TB HDDs (Seagate Enterprise / Ironwolf or eqv.) for general storage and VM backups.

This machine will be used to host some (not to demanding) business software + database and also for some simulation/data crunching research.

my current plan would be something like this:
2x 240GB SSD Micron 5100 PRO

• RAID 1 - mirrored
• Proxmox
• 2x "high availabilty" VMs / containers
  
• (optional: - ZIL/L2ARC )

1x 1TB SSD Samsung SM961

• single (no mirror / raid)
• 3-4x "development/research" VMs where high speed is in demand (not simultaneously)
  
• (optional L2ARC)

3x 6TB HDD Seagate Enterprise / Ironwolf

• RAIDZ1
• general purpose storage
• backup VMs

This server/workstation shall not be used as only backup for any data, but the core systems + 2 VMs should run in a more or less "production environment".

------

Is there anything wrong with this setup?
Should I go with cheaper/more expensive hardware (e.g. Samsung 850 instead of Micron 5100)?


No dedicated HBA/RAID Controller planned (SuperMicro server motherboard wil be used)

 

[Andreas Piening]

I guess you might get more opinions on the perfect storage setup than the number of peoples you ask, here are my thoughts:
- You use no hardware RAID controller in your setup and I think this is perfectly fine. Using JBOD is the correct way to deploy ZFS pools.
- You have your SSDs mirrored so you have redundancy while you get better read speed. You can always use enterprise hardware if you can afford it, but as long as you have redundancy (RAID1) this is fine. At least I head nothing bad about Micron SSDs.
- RAIDZ is good for backups and general data storage where you don't need that fast IO.
- I wouldn't go for L2ARC on your setup since the amount of storage is pretty small and nothing should hold you back from putting enough RAM in your system. It depends on your exact usage scenario but even 8 GBs of RAM dedicated for ZFS should be suffizient. Then add a fair amount for the host system and for your VMS and that's your minimal RAM requirement.

So it looks good to me, except for the only "bad" thing I can see is the 1 TB SSD which does not have redundancy: You should avoid that in any kind of productive setup. Add a second one and go for RAID1 or use the 3 x 6 TB RAID instead if the speed is not really needed.

You have not mentioned the amount of RAM your system would have. Having fast SSDs is not a substitution for enough RAM. Not only enough so that the VMs don't swap but also enough so that the OS can do caching.
This also depends on if you are using KVM or LXC for your VMs. You need more RAM for LVM since your options to overprovision your physical RAM are very limited.

 

[ProLog]

I guess you might get more opinions on the perfect storage setup than the number of peoples you ask, here are my thoughts:

well thanks a lot for your response

- I wouldn't go for L2ARC on your setup since the amount of storage is pretty small and nothing should hold you back from putting enough RAM in your system.

The plan right now is to go with 128GB (and AMDs EPYCs - though probably 64GB would be enough for the start - but the price difference between is to low to justify going with 8GB DIMMs instead of 16GB DIMMs)

So it looks good to me, except for the only "bad" thing I can see is the 1 TB SSD which does not have redundancy: You should avoid that in any kind of productive setup. Add a second one and go for RAID1 or use the 3 x 6 TB RAID instead if the speed is not really needed.

Those VMs running on the large/fast SSD would be "development" machines. I tought it would be a good idea to separate them from the "production" machines also so there is no bottle neck in data r/w. It probably wouldn't be a problem if they are down for some days. From what I read overnight backups to the "storage" array should be possible ?!

Do you think instead of the 2xMicron and 1xSamsung drive I should rather go with
2x1TB SSDs in RAID 1
3x500GB SSDs in RAID Z1
4x500GB SSDs in RAID 10
????

This also depends on if you are using KVM or LXC for your VMs. You need more RAM for LVM since your options to overprovision your physical RAM are very limited.

both KVM & LXC

 

[Digitaldaz]

Unless your dev machines are all going to be very demanding, if you are going down the all SSD route probably better with either a single RAIDZ2 or a couple of RAIDZ. If you can a small faster slog device like S3700 or maybe Samsung SM863a.