Hardware decision making help...

[flipswitchingmonkey]

Hi all,

I'm considering setting up a Proxmox server, but I'm lost in the sea of possibilities

What I'm looking to build is primarily a file server, with some containers or VMs (internal webservers, databases with light use etc) on top. The file server should ideally offer around 20TB of space and should be able to saturate at least an 10G connection when reading and writing small-ish files (image sequences mostly).
15 users will be accessing with a mix of 1G and 10G connections

Here's my thinking so far:

Basic specs:
CPU: some dual Xeon with 8 or 10 cores each
RAM: 128GB DDR3 ECC

Money is an issue (when is it not)... so 20TB of enterprise flash is a no go, unfortunately.

I have a HP D2700 SAS enclosure here which does SAS 6Gb and SATA 3Gb with up to 25 2.5" drives. I don't have to use it, but it's available.
I also have an LSI RAID controller for said enclosure with battery backup.

Option 1:
Get a server case or enclosure with 12 3.5" drives, fill with 4TB SAS HDDs and run as RAID10 (mirrored zfs)
Get either three PCIe SSDs (Optane 900p?) or three 12G SAS SSDs (if I get a server with a 12G backplane) and use as L2ARC, ZIL etc.

Option 2:
Similar to above, but instead get an HBA with external SAS and connect the HP D2700, put 25 1TB SAS drives in in a RAID6 (raidz2) configuration.
Caching again as in Option 1

Option 3:
Fill the D2700 with 500GB or 1TB *consumer* SATA 3G SSDs (so they will peak at 300MB/s each) and live dangerously. Run as RAID5 with the RAID controller card, no ZFS and no caching, since the SSDs should be fast enough. I'm aware of the longevity issues with consumer SSDs. Write-back would be enabled through the RAID card.

So there I am. The 3.5" drives would certainly provide much more space for the money, but I'm not sure about the speed. Caching with the Optanes (or similar) would be very fast, but anything uncached... not sure.

The 2.5" SAS drives would in total be more expensive, but I would expect them to be faster (simply by virtue of being more drives, though again, not sure with the whole vdev thing...)

The consumer SSD option would probably the fastest and easiest to set up by far, but it would leave me worried - has anyone been running a RAID or zfs purely with consumer SSDs (Samsung EVO 860 for example) and can share some experiences?

Any help or opinions would be welcome!

 

[Marshalleq]

I too am grappling with this a bit. Seems like I really can't do consumer SSD. I have run them both with EXT and LVM-Thin and also ZFS. ZFS seems to be the slowest for some reason, even slower than running off the terrible green drive in there. I can't believe I'm saying this, but I think I need to RAID-0 some magnetic drives to get SSD like performance and take out my consumer grade Samsung SSD's altogether. I put three 3TB normal HDD's into some thunderbolt enclosures and RAID-0'd them for a performance test last week. They screamed. Faster than my 1TB Samsung SSD. So I'm thinking that just maybe the best option would really be RAID 10 magnetic. Suspect you'd get the performance, space and cost benefits, but could be wrong. I'll probably just go RAID-0 for now with backups. Uptime isn't really an issue for me.

 

[msalwe]

Running RAID 5 with 250GB consumer SSDs since last six months. Performance is as expected. Using ext4 FS which by the way gives better perf than zfs or anything else. Only worry is TBW of SSD. Will watch how for failure of ssd will be very unlikely that all ssds fail at once and I lose all my data. But then this possibility is same for mech hdds also.

 

[flipswitchingmonkey]

So after many, many rounds of wiping the server completely (and long hours...) I've settled on something different. First of all, for hardware I went with kind of Option 1. A Dell R720 filled with Micron 5200 SSDs. Those are not consumer SSDs. I run a different RAID on another server where I tested the SSD RAID with Samsung 860s and they perform very well - until the cache runs out. Then they turn into a dog. With few users you will probably never have an issue. With many users bombarding the RAID, I'm sure this will be a problem over time. So I went with the Microns which have some nice enterprise features and were priced okay as well, thanks to recent price improvements. TBWs are high enough (iirc one full write per day)

The SSDs are running in hardware raid-5 now, using XFS as the file system. This is because we are serving amongst other things lots of large video files.

Now initially I started out with Proxmox and some Debian LXC and VMs running underneath. It mostly worked fine, but the network performance was not what I hoped for. Tried various different distributions and some were better than others, but still...

Long story short, I've now settled on running Fedora bare-metal. It's giving me pretty much max performance, but I'm losing the virtualization of the main OS, which worries me a bit, but so be it. Obviously I can still run VMs and containers on top of that, so that's good.

This is not to take away from Proxmox. It's just that I got better performance through bare-metal (and also ESXi, tbh). But price/feature wise proxmox is still hard to beat.

Edit: also, Samba is such a PAIN in the arse with active directory!

 

[alexskysilk]

The SSDs are running in hardware raid-5 now, using XFS as the file system. This is because we are serving amongst other things lots of large video files.

You will get MUCH better performance if you separate your virtual disk storage and your large video files to separate luns. virtual disk storage REALLY wants as many queues as possible. While RAID5 may be fine for your data files, even a 2 disk RAID1 would probably outperform your raid5 for virtual disk use, and RAID10 certainly.

The other thing you may want to take into consideration is that RAID5 is dangerous. During degraded mode (when a disk has failed) you are wholly at the mercy of your system; bad ram, any read errors (disk or bus) can result in bad data written permanently and transparently to you. If you must run parity raid, have at least dual parity (RAID6.) ECC ram and SAS drives mitigate the risk SOMEWHAT, but disks are usually cheaper then your data.

 

[flipswitchingmonkey]

RAID 5 is definitely a no-no for HDD based RAIDs, but recovery is so much faster with SSDs that it's fine (imho).

As for separating the drives - I think you're coming at this from a ZFS perspective. This is XFS on a hardware raid, there are no LUNs (and putting fewer disks into a hardware raid 5 actually makes it slower). I'm getting 2.9gb/s for sequential and something like 400mb/s for 4k random writes, iirc. IOPS were around 90k.

 

[alexskysilk]

recovery isnt the only issue; its the length of time between being degraded and being whole. Unless you have a hot spare in the system rebuild time is the least of your concern, and if you have a hot spare you may as well have it be active.

A RAID controller still only has one queue per disk group. Your disk group may be fast but can only service one request at a time. having multiple requestors will necessarily be waiting in queue.

Of course you're the one who determines if its fine or not, its your system