SAN / NAS plan out assistance needed here

[geekspaz]

The company I work with is looking to move off VMware, and we've landed on trying Proxmox. I've personally jumped into Proxmox head first to try and test what I can for our team, and have worked through much of the unknowns so we can be ready.

However, one item I don't have the setup to test at home is configuring a shared network storage for the cluster where the VM files will reside. Before my company commits half a million in hardware to this leap, I've convinced the team to do a test run with $13,000 of enterprise grade hardware off eBay so we can do some basic load testing and find out how live migrations might work, see how different issues might surface, determine a standard system for monitoring, etc.
The basic hardware setup we will be testing with:
3x Dell PE R740XD, each with 768GB ram, dual Xeon 6152 CPU's, Nvidia P40 GPU, dual SFP28 NIC, and BOSS card with a pair of 128gb m.2 sata's
10x Micron 9300 Pro NVMe 3.84TB to put in 1 of the servers as the cluster NAS
1x Dell 48 port SFP28 switch to connect it all together.

The planned setup for our testing -
2 of the servers will be configured in a Proxmox cluster, with the 3rd configured with the 10 NVMe drives as the storage for the cluster. I am planning to connect to it via iSCSI.

What I need help with... Rephrase - the help I am aware that I need, is deciding on an OS for the server that will act as the NAS. I want to use a RAID5/RAID6 disk setup so the 10 x 3.84TB drives can give me 30TB of storage or more for the VM hard disks. I've read a decent amount of posts centering on TrueNAS Scale and RAIDZ1/RAIDZ2 (not anywhere close to calculating the formulas yet), and I'm just not quite onboard with taking a IOPS performance hit or storage space hit that seems to come with it.

Has anyone used a NAS distro that can get a good performance, great efficiency of space utilization, and I don't lose the data if a drive fails? But I'm also not wasting drives to just mirroring?

 

[bbgeek17]

Hi @geekspaz ,
There are a few architectural things you have to work out for a successful proof of concept:

2 of the servers will be configured in a Proxmox cluster

Two servers are not a valid/supported cluster construct. Each cluster member has an equal vote. If you lose the connection between the two members then there will be no majority present. That means neither one knows for sure that it can take over services. The likely outcome is both will reboot. When the nodes come back up, if there is still no connection, the PVE services will stay down.
A valid cluster configuration will contain an odd number of nodes.

3rd configured with the 10 NVMe drives

That implies that the 3rd one will be a Single Point of Failure (SPoF). If the server dies or goes offline, your entire virtualization layer will be offline.

TrueNAS seems to be the leading "free" option for NAS/SAN services. Like many other Open Source options TrueNAS uses ZFS for storage organization. As such, the OS does not matter that much. If your NAS uses ZFS - the efficiency/utilization will be similar to TrueNAS.

Has anyone used a NAS distro that can get a good performance, great efficiency of space utilization, and I don't lose the data if a drive fails? But I'm also not wasting drives to just mirroring?

Data protection comes with "waste" built-in. If you want to survive disk failure, that data needs to be in more than one place.
Given that you have 3 servers, you can try spreading your data across all nodes and using Ceph. There are calculators online that can tell you how much usable space you will get in that case.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[geekspaz]

Hi @geekspaz ,
There are a few architectural things you have to work out for a successful proof of concept:

Two servers are not a valid/supported cluster construct. Each cluster member has an equal vote. If you lose the connection between the two members then there will be no majority present. That means neither one knows for sure that it can take over services. The likely outcome is both will reboot. When the nodes come back up, if there is still no connection, the PVE services will stay down.
A valid cluster configuration will contain an odd number of nodes.


That implies that the 3rd one will be a Single Point of Failure (SPoF). If the server dies or goes offline, your entire virtualization layer will be offline.

TrueNAS seems to be the leading "free" option for NAS/SAN services. Like many other Open Source options TrueNAS uses ZFS for storage organization. As such, the OS does not matter that much. If your NAS uses ZFS - the efficiency/utilization will be similar to TrueNAS.


Data protection comes with "waste" built-in. If you want to survive disk failure, that data needs to be in more than one place.
Given that you have 3 servers, you can try spreading your data across all nodes and using Ceph. There are calculators online that can tell you how much usable space you will get in that case.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Thank you for the response! I completely agree on all of the points you made, but those issues are expected and are not the focus of this stage of testing. Our final production setup will have 3 Dell PE R760's for the cluster with a Dell Powerstore for the SAN/NAS, and plans to add a 4th PE R760 to the cluster within a year. The focus for this round of testing (we'll call it stage 2) is connecting a shared storage over 25gb nic and dual bonded 25gb nic's to see the performance and latency. We absolutely don't want to configure mirrored storage of any kind between hosts as that will be a great waste of storage - our data center currently has 100TB of used space, and we have no intention of setting up 3 hosts with 100TB each. The final production setup will employ other redundancy for the storage as well to cut the SPoF potential.

I was actually looking at Blockbridge but didn't feel it was fair to you guys to run a trial purely for the purpose of testing this but then cut out after, if it would even run on the targeted server as a NAS. If you want to discuss that here from a functional perspective and then pricing in DM's or on a call, I am definitely up for it, but I don't want to waste your time.

This post is what has me concerned with using TrueNAS (or any RAIDzx solution for this setup) - https://forum.proxmox.com/threads/about-zfs-raidz1-and-disk-space.110478/post-475702

 

[Johannes S]

I wanted to answer to geekspaz last answer but it seems the post was to be deleted?

He was worried about using RAIDZ and TrueNAS. While it's true that one can use RAIDZ with TrueNAS it doesn't mean that you have to use it. You could also use ZFS on TrueNAS (or any other OS supporting ZFS for all that matters) in a mirrored setup which would be an equivalent to RAID1. I haven't done any tests in this regard for myself but if I recall corectly mirroed setups are best in terms of failure resistance and performance but also uses most space for obvious reasons.
Just my two cents

 

[floh8]

Truenas is a good choice. U can use an NFS share or zfs-over-iscsi. For the 3. Node u can create a VM on the truenas with a qdevice. See wiki. As Raid I would suggest to use zfs mirror stripes (Raid10), but with NVME your network connection would be the bottleneck.

 

[bbgeek17]

I was actually looking at Blockbridge but didn't feel it was fair to you guys to run a trial purely for the purpose of testing this but then cut out after, if it would even run on the targeted server as a NAS. If you want to discuss that here from a functional perspective and then pricing in DM's or on a call, I am definitely up for it, but I don't want to waste your time.

Hi @geekspaz,

I recommend configuring your proof of concept (POC) with the same storage solution you plan to use in production. Otherwise, you might encounter unexpected issues later on.

Keep in mind that the Proxmox interactions and integration with Blockbridge differ significantly from TrueNAS and PowerStore. It wouldn't be wise to POC with Blockbridge and then switch to a NAS for deployment.

Storage can be complex, so it's best to measure three times and cut once.

Good luck!

P.S.

The focus for this round of testing (we'll call it stage 2) is connecting a shared storage over 25gb nic and dual bonded 25gb nic's to see the performance and latency

You may be interested in this article: https://kb.blockbridge.com/technote/proxmox-tuning-low-latency-storage/index.html

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[LnxBil]

cluster with a Dell Powerstore for the SAN/NAS

Does Powerstore have actual PVE driver integration for iSCSI for snapshots etc? If not, I would not build a new system using this legacy technology. I concur with @bbgeek17 that you will be very disappointed with your Powerstore if you tested Blockbridge and Powerstore is sub-part on the feature front.