Storage config for new Proxmox installation

[schneida]

Hi guys,

I need help with choosing the right storage configuration for a Proxmox VE setup. First of all - I'm considering buying the Basic Subscription - is this type of question something I can ask support about? Also does Proxmox offer discounts for Austrian non profit / civilian service organisations which the server belongs to?

Back to topic: Skip this introduction I you don't want to know about the background!

We recently purchased a new Lenovo TS150 Server (Xeon E3-1245 v5, 4x 3.50GHz, 16GB ECC RAM, 2 WD RED @ 4TB, 2 Intel SSD DC S3520 240GB) - the original plan was to rollout Hyper-V with Server 2016 Essentials and a Linux VM on it. However Hyper-V was a bust (me coming from a Linux background) and time was short so we went for installing Server Essentials 2016 as bare metal and have the Linux VM running as guest on Server 2016 Essentials. So far so good, everything was working for 2 weeks after which the server started to lock up completely without any indication in the Event Logs, Bluescreens, Dumps whatsoever. Neither Lenovo nor Microsoft could tell me whats going on so I went ahead tried some stuff (changing SATA ports, disabling all power saving mechanisms, etc..) and the problem went away for another 2 weeks. So right now the server again locks up around every 24 hours and has to be restarted (luckily the Intel AMT webinterface works, so I can restart remotely). Now I suspect that the issue is related to some Windows driver or something similar and that the hardware itself is ok (its new after all). This suspicion is based on the impression that Windows Server 2016 Essentials pretty much sucks (encountered many bugs) and our old server which was working flawlessly for the last 3 years with Linux and kept crashing (blue screens) with Windows Server which is the original reason we bought a new server.

To make a long story short, I hope that by virtualizing Windows Server 2016 Essentials I can avoid receiving calls at 5am that the server crashed again. Now I'm on a tight schedule as I'm living in another city and only get to see the server every other weekend - next time this weekend, and I'm planning on doing the migration then.

---- CONTINUE HERE IF YOU SKIPPED THE ABOVE ----

What would you suggest for this setup in tems of storage (remember Lenovo TS150 Server with Xeon E3-1245 v5, 4x 3.50GHz, 16GB ECC RAM, 2 WD RED @ 4TB, 2 Intel SSD DC S3520 240GB) - more RAM could be bought if required. Proxmox should host Windows Server 2016 Essentials as Active Directory + File host, and a Linux guest running a custom PHP based application + SQL database. Maybe in the future I will also install pfSense as a third guest.

I read up on RAID and that Software RAID (I guess this also includes Intel RSTe) is not supported by Proxmox and that I could either go the Debian + Proxmox manual installation way or use ZFS. However I have almost no knowledge about ZFS (have been using LVMs the last couple years) so I'm worried I won't be able to handle problems. Also if ZFS, should the SSDs be used as Cache for the WD REDs or should I create two pools and hand storage through to the Windows Server and manually distribute data based on usage? Currently I configured 2 RAID 1, one SSD RAID, one HDD RAID and most seldomly accessed "archive" data is on the HDD and the rest on the SSD.

Many thanks for your input. I know that these kind of questions are hard to answer without thoroughly testing the hardware first but maybe there are some general up-to-date best practices. So far many benchmarks I have stumbled upon were from 2011, 2013 and lots of things have changed since then...

Thanks!

 

[udo]

Hi,
in your case it's sound, thar zfs is the right decision for you (but spend more ram after testing).

Udo

 

[fabian]

Hi guys,

I need help with choosing the right storage configuration for a Proxmox VE setup. First of all - I'm considering buying the Basic Subscription - is this type of question something I can ask support about? Also does Proxmox offer discounts for Austrian non profit / civilian service organisations which the server belongs to?

just responding to this here - Proxmox Gmbh does not offer (commercial) pre-sales support or consulting (see http://www.proxmox.com/en/proxmox-ve/pricing and the Subscription Agreement for details on what enterprise support covers or not).

if you specific questions regarding support prices, subscription changes or status, billing or anything else that is not public/technical/.., just drop an email to office@proxmox.com (not yet a customer) or open a ticket in the shop system (already a customer).

 

[schneida]

@udo thanks for your input! I'm just worried that ZFS is going to be to complex and I would break things. But if I understand correctly, the Proxmox Installer supports setting up ZFS in some kind of RAID and also makes sensible defaults?

@fabian thanks for the link. Of course I understand, that you can't offer pre-sales support, I was just wondering if one of the three paid support ticket can be used to get help with the initial setup. After reading the Subscription Agreement I would presume this is part of the Installation and Configuration support that's included!

 

[maxprox]

@udo thanks for your input! I'm just worried that ZFS is going to be to complex and I would break things. But if I understand correctly, the Proxmox Installer supports setting up ZFS in some kind of RAID and also makes sensible defaults?

ZFS with the installer is very easy, and the entire installation takes only 15 min. or less, so you can try different types. Especially look ahead into the wiki. I find the decision for the right solution with your two SSDs not so easy, because for example a ZFS Raid 1 only for the system would be very lavish here (?)
If you want to try ZFS, read this post spudger and this Raid10 example , as well as the following (ssd trim and zfs tuning tips)
best regards,
maxprox

 

[maxprox]

I would suggest without the two WDs to generate a ZFS Raid1 on the SSDs via Installer
reboot - test - shutdown the system, only then I would connect the two WDs and create a large ZFS Raid1 pool.
On the first SSD ZFS Raid1 pool I would put the Windows system partition C: \, on the second WD pool the larger data store. (Proxmox + Windows on the SSD, and the same with Linux VMs)

 

[schneida]

Thanks so much for your input! I actually did a lot of reading on ZFS (@maxprox thanks also for those links!) and yes, I think its best to keep the SSDs and the WDs in different pools and manually manage where stuff is going. Because I read that ZFS is so RAM consuming (but also efficiently using it in terms of ARC) I already ordered another 16GB RAM which I will get tomorrow leaving me with 32GB.

I still have some questions which I couldn't answer though:

• Is putting ZIL on an SSD worth the trouble in a KVM setup? Eg. is KVM able to pass synchronous writes and asynchronous writes through or will everything be written sync/async all the time? The way I see this working would be to use the WD as initial installation disks (where GRUB etc live) and partition the SSD to use some space as ZIL for the WD RED and the rest as separate ZFS mirror volume.
• L2ARC is nice to have but only if you have loads of RAM already - so in my case it's better to get another 32GB or even 64GB (which I could easily fit) instead of eg. a third SSD as L2ARC?
• I actually have a third WD RED available (vendor shipped one too much but forgot another thing and then told me too keep) which I was keeping as cold standby as it wasn't useful in my initial setup. I'm however now considering creating a RAIDZ1 out of three WD RED but have read lot of criticism regarding RAID5 and RAIDZ1 (although being better than RAID5) because it is almost certain, that in case a drive fails you will have to restore from backups anyway. I guess its better to keep the WD RED as a hot standby then and stay with mirroring?
• Whats your opinion about ZFS compression?
• I'm screwed if /dev/sda crashes (in a mirror of /dev/sda and /dev/sdb) because I won't be able to reboot the system anymore (see https://pve.proxmox.com/wiki/ZFS:_Tips_and_Tricks#Replacing_a_failed_disk_in_the_root_pool)?

 

[maxprox]

I am not a ZFS expert. In short I would suggest to use min. 8 GB for Proxmox with ZFS and max. 16 GB,
(16 is better). I would do nothing with L2ARC or ZIL. And I would set vm.swappiness = 1 (or max 10) because you have enough RAM, and limited the ARC cache as described here