HDD/SDD RAID questions before installation

[Renato77]

I am about to buy a HPE Microserver that only has 4 internal bays for SDD/HDD. I need to run two VMs on this machine. Both VMs need separate partitions for data (one is a SQL database). I do have a NAS for backup storage and 1Gb Ethernet.

So what would be a wise setup? I would love to run a RAID-Z2 on 4 HDD. But I am not so sure if any RAID is a good idea with this limited number of disks.

What would you do?

 

[Dunuin]

Any raidz isn't recommended if you need performance (so not recommended for a VM storage or to run DBs). And HDDs aren't great as VM or DB storage too because they easily get overwhelmed by all the small random operations or sync writes because of the low IOPS they can handle. Especially if you don't by 10K/15K SAS HDDs.

Best would be a mirror (raid1) or stripped mirror (raid10) of enterprise SSDs. If you really need the integrety of raidz2 you also could install proxmox ontop of a Debian Buster and use a mirror of 3 drives. It that case up to 2 drives may fail too without loosing data.

 

[Renato77]

Thank you. My initial plan was to use 3 Enterprise SSDs for Proxmox, for the VMs, and for the data partitions. A fourth HDD for internal backups. Will Proxmox be writing less to the SSDs if no RAID is being used?

 

[Dunuin]

Thank you. My initial plan was to use 3 Enterprise SSDs for Proxmox, for the VMs, and for the data partitions. A fourth HDD for internal backups. Will Proxmox be writing less to the SSDs if no RAID is being used?

Yes, any kind of parity (raid1, raid10, raidz) will double the writes because everything needs to be written to two drives instead of one. And sync writes (like DBs do) will double that again (data will first be written to the ZIL and later to the journal and both are on the same drive if you don't use a SLOG). Also keep in mind that you loose the self-healing feature of ZFS if you don't have parity. Without raid ZFS can detect bit rot but not repair it.

 

[leesteken]

I would also advise 3-way-mirrrors because the read IOPS and speeds increase and your data is not immediately at risk when one drive fails.

If you really need the integrety of raidz2 you also could install proxmox ontop of a Debian Buster and use a mirror of 3 drives. It that case up to 2 drives may fail too without loosing data.

Why the need to install on top of Debian? Can't you select a three-way mirror in the Proxmox installer? Wouldn't it be simpler to install to a two-way mirror and add the third afterwards? Or did I understand your remark incorrectly?

 

[Dunuin]

Why the need to install on top of Debian? Can't you select a three-way mirror in the Proxmox installer? Wouldn't it be simpler to install to a two-way mirror and add the third afterwards? Or did I understand your remark incorrectly?

I thought I haven't seen 3 mirror configurations there. But yeah, just installing a 2 drive mirror and adding another disk later should be the better way. I didn't think of that.

 

[leesteken]

I thought I haven't seen 3 mirror configurations there. But yeah, just installing a 2 drive mirror and adding another disk later should be the better way. I didn't think of that.

I just checked: you can select zfs (RAID1) in the installer and select as many drives as you like to be mirrors of each other. 3 works fine, 4 probably also.

 

[Renato77]

Well, this surely is good advice, but as I said, unfortunately there is space for 4 disks. As a novice, I think I must have played every possible configuration in my mind through and through. Right now I am stuck on 2 mirrors. One for Proxmox, one for the VMs including the data partitions. Snapshots send to my NAS. Tomorrow will be different, I guess.

 

[Dunuin]

Well, this surely is good advice, but as I said, unfortunately there is space for 4 disks. As a novice, I think I must have played every possible configuration in my mind through and through. Right now I am stuck on 2 mirrors. One for Proxmox, one for the VMs including the data partitions. Snapshots send to my NAS. Tomorrow will be different, I guess.

If you are low on ports or drive slots its fine to use the same pool for proxmox and VMs/data. ZFS doesn't need full drives, it will work fine with partitions. So you could leave some space on the drives unallocated while installing and add additional partitions later to create another pool on the same disks or just just the complete disks and store the VMs/data on the same pool that proxmox is installed on (rpool).

 

[mauriflorentin]

Hi @Renato77 I'm curious how you finally installed it? What raid/file system did you choose?

I'm about to buy a HPE Microserver too, can you please share your experience and setup?

 

[Renato77]

I will answer later. Our company got hit by a flood... so the server is unplugged and at a different location. I also need to double check the settings before I tell you something that isn't true.

But I can truly say that I like the Microserver so far and I do not regret having bought it.

 

[mauriflorentin]

I will answer later. Our company got hit by a flood... so the server is unplugged and at a different location. I also need to double check the settings before I tell you something that isn't true.

But I can truly say that I like the Microserver so far and I do not regret having bought it.

Thanks @Renato77 wait for your reply.

 

[Renato77]

I got so much going on right now... sorry for keeping you waiting.

I bought four Seagate 2 TB Enterprise HDDs and set them up as a ZFS Raid 10 which means a stripe of mirrors. First, I wanted to go with SSDs, but the higher price and the fact that I did not know how much data Proxmox is going to write on them made the decision a bit easier. 2 TB HDDS in RAID 10 means round about 3.7 TB of free storage space.

I have one Windows VM and two small Linux containers (one for storing mails, on for AdGuard) running on the Microserver. For that I installed 16 GB of RAM and reserved 8 GB and 2 cores for Windows. I do not run demanding software, so these settinsg work fine for me. Reading speed was ridiculously fast. Windows via RDP operates pretty fast and I did not have any issues at all.

But you got to be aware that a RAID 10 means doing snapshots and backups regularly. One drive can fail, though.

 

[Dunuin]

2 TB HDDS in RAID 10 means round about 3.7 TB of free storage space.

Keep in mind that ZFS gets slow if you use more then 80% and switches into panic mode if you fill it up above 90%. So its best to set a 80-90% quota for the entire pool so it is impossible to fill it completely up by mistake. So your 3.7 TB is more like 3.0 TB.

I have one Windows VM and two small Linux containers (one for storing mails, on for AdGuard) running on the Microserver. For that I installed 16 GB of RAM and reserved 8 GB and 2 cores for Windows. I do not run demanding software, so these settinsg work fine for me. Reading speed was ridiculously fast. Windows via RDP operates pretty fast and I did not have any issues at all.

Also keep in mind that ZFSs ARC needs some RAM to work properly. Rule of thumb would be 12GB RAM for the ARC if you got 4x 2TB drives. Should work with less RAM but the more you reduce the ARC size, the slower your pool will get. You could lower the ARC size step by step and use arc_summary to check your hit rates and free metadata/dnode cache and stop if the hitrates drop too much or metadata/dnode cache is capping.

 

[Renato77]

Thank you.
Different question: I was wondering if in theory I could add the remaining cores to that one Windows VM.? Is this somewhat common?

 

[Dunuin]

Thank you.
Different question: I was wondering if in theory I could add the remaining cores to that one Windows VM.? Is this somewhat common?

If you don't need super fast latencies you can easily overcommit your CPU. Its for example no problem to allocate 32 vCPUs to VMs if you only got a quadcore as long as most of the vCPUs are idleing most of the time. But its not good to set the same amount of vCPUs to a VM as your host got threads. So if you for example got a quadcore with 8 threads I wound't give a single VM more than 6 vCPUs.