Storage recommendations

[bcmike]

Hi,

I've got a great new 4 node 5.3 cluster set up with lots of RAM, ZFS, and everything is working great. The only thing is that before I rolled into production I noticed some IO delay. Not a big deal but nothings really running under load yet, so a bit troubling. Long story short I figure out that its due to one of my hardware trade offs, I have 2, 2TB 7.2k sata drives in a ZFS RAID 1 mirror (everything's local storage). In my use case i'm not going to have a lot of reads and writes but I guess this is still an under powered solution for production. I have a prejudice for RAID 1 as I like the idea of just popping hard drives out of a smoking system and popping them into new hardware and booting.

Anyway I've come to the conclusion I'm going to have to go with faster drives and RAIDZ with three drives

Have I mentioned I'm on a tight budget.... (lol who isn't) Here are my proposed options from most expensive to least.

Option1:
Three 1 TB SSD's in each machine. This is probably cost prohibitive

Option 2:
Three 600 GB 15k SAS drives in each machine

Option 3:
Three 600 GB 10k SAS drives in each machine.

Also thinking about options 2 & 3 but with 1 256GB SSD for ZFS caching (although I'm not entirely sure how to do that)

Keep in mind that even the least expensive option is probably going to hurt.

Any recommendations or ideas would be most appreciated. Also would simply adding an SSD to do caching solve the issue??

Thanks in advance

 

[aaron]

First, why get a great new 4 node cluster with PVE 5.3? Why not the latest 6.0? Or at least 5.4?

Some thoughts as only you can decide what to buy

With fast NVME SSDs you might get enough performance with two in a mirrored setup.

Regarding option 2 and 3, depending on the financial situation I personally would avoid option 2 and use the slightly higher budget and try to get 4 10K drives and use them in a RAID 10 setup.

Regarding caching: a dedicated fast device for the ZIL can beneficial. ZFS got the ability with 0.8 to assign "special allocation classes" to very fast VDEVs. Since this is quite new there is not a lot of experience yet but you can research the topic and it might be useful.

A L2ARC can be useful but it can also backfire. If you got plenty of RAM use that.

My 2 cents on the topic, hopefully there was something useful for you.

 

[bcmike]

Great info thank you so much!

I'm actually on 5.4 and I've been seeing some problems still with 6 on the forum so I'm going to wait for a little bit.

I actually found some 15k 600Gb drives that fit the budget so I'm buying four for each server. I'm going to research RAID 10

I have 49 GB of ram so not loads but I'm not really planning on running lots of VMS so I might have some room to use RAM. ARC already seems to consume a lot.