SAN / NAS plan out assistance needed here

[geekspaz]

The company I work with is looking to move off VMware, and we've landed on trying Proxmox. I've personally jumped into Proxmox head first to try and test what I can for our team, and have worked through much of the unknowns so we can be ready.

However, one item I don't have the setup to test at home is configuring a shared network storage for the cluster where the VM files will reside. Before my company commits half a million in hardware to this leap, I've convinced the team to do a test run with $13,000 of enterprise grade hardware off eBay so we can do some basic load testing and find out how live migrations might work, see how different issues might surface, determine a standard system for monitoring, etc.
The basic hardware setup we will be testing with:
3x Dell PE R740XD, each with 768GB ram, dual Xeon 6152 CPU's, Nvidia P40 GPU, dual SFP28 NIC, and BOSS card with a pair of 128gb m.2 sata's
10x Micron 9300 Pro NVMe 3.84TB to put in 1 of the servers as the cluster NAS
1x Dell 48 port SFP28 switch to connect it all together.

The planned setup for our testing -
2 of the servers will be configured in a Proxmox cluster, with the 3rd configured with the 10 NVMe drives as the storage for the cluster. I am planning to connect to it via iSCSI.

What I need help with... Rephrase - the help I am aware that I need, is deciding on an OS for the server that will act as the NAS. I want to use a RAID5/RAID6 disk setup so the 10 x 3.84TB drives can give me 30TB of storage or more for the VM hard disks. I've read a decent amount of posts centering on TrueNAS Scale and RAIDZ1/RAIDZ2 (not anywhere close to calculating the formulas yet), and I'm just not quite onboard with taking a IOPS performance hit or storage space hit that seems to come with it.

Has anyone used a NAS distro that can get a good performance, great efficiency of space utilization, and I don't lose the data if a drive fails? But I'm also not wasting drives to just mirroring?

 

[bbgeek17]

Hi @geekspaz ,
There are a few architectural things you have to work out for a successful proof of concept:

2 of the servers will be configured in a Proxmox cluster

Two servers are not a valid/supported cluster construct. Each cluster member has an equal vote. If you lose the connection between the two members then there will be no majority present. That means neither one knows for sure that it can take over services. The likely outcome is both will reboot. When the nodes come back up, if there is still no connection, the PVE services will stay down.
A valid cluster configuration will contain an odd number of nodes.

3rd configured with the 10 NVMe drives

That implies that the 3rd one will be a Single Point of Failure (SPoF). If the server dies or goes offline, your entire virtualization layer will be offline.

TrueNAS seems to be the leading "free" option for NAS/SAN services. Like many other Open Source options TrueNAS uses ZFS for storage organization. As such, the OS does not matter that much. If your NAS uses ZFS - the efficiency/utilization will be similar to TrueNAS.

Has anyone used a NAS distro that can get a good performance, great efficiency of space utilization, and I don't lose the data if a drive fails? But I'm also not wasting drives to just mirroring?

Data protection comes with "waste" built-in. If you want to survive disk failure, that data needs to be in more than one place.
Given that you have 3 servers, you can try spreading your data across all nodes and using Ceph. There are calculators online that can tell you how much usable space you will get in that case.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[geekspaz]

Hi @geekspaz ,
There are a few architectural things you have to work out for a successful proof of concept:

Two servers are not a valid/supported cluster construct. Each cluster member has an equal vote. If you lose the connection between the two members then there will be no majority present. That means neither one knows for sure that it can take over services. The likely outcome is both will reboot. When the nodes come back up, if there is still no connection, the PVE services will stay down.
A valid cluster configuration will contain an odd number of nodes.


That implies that the 3rd one will be a Single Point of Failure (SPoF). If the server dies or goes offline, your entire virtualization layer will be offline.

TrueNAS seems to be the leading "free" option for NAS/SAN services. Like many other Open Source options TrueNAS uses ZFS for storage organization. As such, the OS does not matter that much. If your NAS uses ZFS - the efficiency/utilization will be similar to TrueNAS.


Data protection comes with "waste" built-in. If you want to survive disk failure, that data needs to be in more than one place.
Given that you have 3 servers, you can try spreading your data across all nodes and using Ceph. There are calculators online that can tell you how much usable space you will get in that case.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Thank you for the response! I completely agree on all of the points you made, but those issues are expected and are not the focus of this stage of testing. Our final production setup will have 3 Dell PE R760's for the cluster with a Dell Powerstore for the SAN/NAS, and plans to add a 4th PE R760 to the cluster within a year. The focus for this round of testing (we'll call it stage 2) is connecting a shared storage over 25gb nic and dual bonded 25gb nic's to see the performance and latency. We absolutely don't want to configure mirrored storage of any kind between hosts as that will be a great waste of storage - our data center currently has 100TB of used space, and we have no intention of setting up 3 hosts with 100TB each. The final production setup will employ other redundancy for the storage as well to cut the SPoF potential.

I was actually looking at Blockbridge but didn't feel it was fair to you guys to run a trial purely for the purpose of testing this but then cut out after, if it would even run on the targeted server as a NAS. If you want to discuss that here from a functional perspective and then pricing in DM's or on a call, I am definitely up for it, but I don't want to waste your time.

This post is what has me concerned with using TrueNAS (or any RAIDzx solution for this setup) - https://forum.proxmox.com/threads/about-zfs-raidz1-and-disk-space.110478/post-475702

 

[Johannes S]

I wanted to answer to geekspaz last answer but it seems the post was to be deleted?

He was worried about using RAIDZ and TrueNAS. While it's true that one can use RAIDZ with TrueNAS it doesn't mean that you have to use it. You could also use ZFS on TrueNAS (or any other OS supporting ZFS for all that matters) in a mirrored setup which would be an equivalent to RAID1. I haven't done any tests in this regard for myself but if I recall corectly mirroed setups are best in terms of failure resistance and performance but also uses most space for obvious reasons.
Just my two cents

 

[floh8]

Truenas is a good choice. U can use an NFS share or zfs-over-iscsi. For the 3. Node u can create a VM on the truenas with a qdevice. See wiki. As Raid I would suggest to use zfs mirror stripes (Raid10), but with NVME your network connection would be the bottleneck.

 

[bbgeek17]

I was actually looking at Blockbridge but didn't feel it was fair to you guys to run a trial purely for the purpose of testing this but then cut out after, if it would even run on the targeted server as a NAS. If you want to discuss that here from a functional perspective and then pricing in DM's or on a call, I am definitely up for it, but I don't want to waste your time.

Hi @geekspaz,

I recommend configuring your proof of concept (POC) with the same storage solution you plan to use in production. Otherwise, you might encounter unexpected issues later on.

Keep in mind that the Proxmox interactions and integration with Blockbridge differ significantly from TrueNAS and PowerStore. It wouldn't be wise to POC with Blockbridge and then switch to a NAS for deployment.

Storage can be complex, so it's best to measure three times and cut once.

Good luck!

P.S.

The focus for this round of testing (we'll call it stage 2) is connecting a shared storage over 25gb nic and dual bonded 25gb nic's to see the performance and latency

You may be interested in this article: https://kb.blockbridge.com/technote/proxmox-tuning-low-latency-storage/index.html

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[LnxBil]

cluster with a Dell Powerstore for the SAN/NAS

Does Powerstore have actual PVE driver integration for iSCSI for snapshots etc? If not, I would not build a new system using this legacy technology. I concur with @bbgeek17 that you will be very disappointed with your Powerstore if you tested Blockbridge and Powerstore is sub-part on the feature front.

 

[geekspaz]

I wanted to answer to geekspaz last answer but it seems the post was to be deleted?

He was worried about using RAIDZ and TrueNAS. While it's true that one can use RAIDZ with TrueNAS it doesn't mean that you have to use it. You could also use ZFS on TrueNAS (or any other OS supporting ZFS for all that matters) in a mirrored setup which would be an equivalent to RAID1. I haven't done any tests in this regard for myself but if I recall corectly mirroed setups are best in terms of failure resistance and performance but also uses most space for obvious reasons.
Just my two cents

Thank you for the feedback. It seems like TrueNAS is popular here, and understandably so. I am looking for an option in the realm of RAID5/6 for efficient use of space with some protection against failover still, and not losing too much performance to the setup.

Has anyone played with other NAS platforms? If not really, I guess I can try a few and report back.

Does anyone know a good tool for testing performance?
And does anyone have any pointers on things to avoid?

 

[geekspaz]

@bbgeek17 and @LnxBil - you have good points... I will need to investigate more.
Thank you for the feedback!

 

[alexskysilk]

Thank you for the feedback. It seems like TrueNAS is popular here,

There are many homelabbers here. I'd be wary going by the advice/experience of users with a different usecase.

am looking for an option in the realm of RAID5/6 for efficient use of space with some protection against failover still, and not losing too much performance to the setup.

parity raid/EC coded storage pools are generally not well suited for virtualization workloads (wide stripe size/low queue count; virtualization wants small random access). I would humbly suggest you're going about this the wrong way; "not losing too much performance" is completely meaningless when you havent defined your minimum acceptance criteria.

Define your expectations before you start evaluating solutions, namely:
1. What is the minimum IOPs requirement per VM/total workload?
2. What is the minimum usable capacity required? what are the growth projections? do you want to size for now/later/ability to scale in real time?
3. What is an accepted outage period? SPOFs will necessarily increase this.
4. What features do you require to go into production? eg, using your dell SAN will lose you the abiliy to thin provision/snapshot at the hypervisor level. Its POSSIBLE to work around this using hardware level thin provisioning/snapshots but very hard to quiesce those since there is no integration.

 

[Johannes S]

4. What features do you require to go into production? eg, using your dell SAN will lose you the abiliy to thin provision/snapshot at the hypervisor level. Its POSSIBLE to work around this using hardware level thin provisioning/snapshots but very hard to quiesce those since there is no integration.

Another alternative to snapshots might be to use backups instead of snapshots. It's even suggested in the proxmox migration wiki page:
https://pve.proxmox.com/wiki/Migrate_to_Proxmox_VE#Alternatives_to_Snapshots

Whether this is sufficient of course depends on ops usecase.

 

[waltar]

4. What features do you require to go into production? eg, using your dell SAN will lose you the abiliy to thin provision/snapshot at the hypervisor level. Its POSSIBLE to work around this using hardware level thin provisioning/snapshots but very hard to quiesce those since there is no integration.

I'm still not shure to loose thin provi and snapshots if using the dell powerstore as when config iscsi you get a virtual disk at pve host and then you could do the same as a pve inst. itself with pvcreate, lvcreate lvm and lvm-thin vol and so it's same functionality there.

3. What is an accepted outage period? SPOFs will necessarily increase this.

The powerstore will do ha to pve clients as a ctrl. may go down and service runs on.

 

[Johannes S]

I'm still not shure to loose thin provi and snapshots if using the dell powerstore as when config iscsi you get a virtual disk at pve host and then you could do the same as a pve inst. itself with pvcreate, lvcreate lvm and lvm-thin vol and so it's same functionality there.

Plain ISCSI doesn't support snapshots though according to the Storage page in the Proxmox wiki. ZFS over ISCSI would work but I'm not sure whether ZFS will play nice since OP wants to use RAID5/6. ZFS and Hardware RAID shouldn't be mixed If I recall correctly.
I agree with LnxBil that using Ceph would propably the more straightforward approach but I understand the constraints of OP which are making this not feasible at the moment.

Since TrueNAS VM capability was already mentioned: OP could use this not only for the qdevice but also to setup the Proxmox Backup Server in the same VM (so no overhead from two vms). Of course the actual backup shouldn't be on production storage. But OP could use this to utilize this PBS vm for a snapshot-like functionality: Since backups are using KVMs vm snapshot they don't need snapshot support on storage backend level. And this also would allow to to file restores in case a update went wrong but to fix this one wouldn't need to restore everything but just some configuration files.

The real backup of course should be on a dedicated backup server, be it PBS, Veeam or whatever backup solution OP is planning to use.

 

[geekspaz]

Everyone, thank you for the continued feedback and discussion. I'm flying out today to rack all the hardware in the data center and get basic network config setup for remote access via idrac's. I plan to test options based on what everyone is pointing out here over the next couple weeks. Since snapshots and backups are being discussed, we've been using Veeam to backup the VM's in VMWare, and are hoping to continue using that as we migrate over. It does run on its own hardware with its own storage. Veeam has some nice features for restoring, and we will be testing that out using their recently released Proxmox integration to see how it functions as well.

Please keep the considerations and ideas coming... I probably won't check in again for a couple days, but I truly appreciate this community and the kindness everyone is showing here.

 

[waltar]

I cannot talk enough about "TEST-restore" your backups/snapshots as it's a complete other story as having and functioning are not forced the same ...

 

[Johannes S]

Everyone, thank you for the continued feedback and discussion. I'm flying out today to rack all the hardware in the data center and get basic network config setup for remote access via idrac's. I plan to test options based on what everyone is pointing out here over the next couple weeks. Since snapshots and backups are being discussed, we've been using Veeam to backup the VM's in VMWare, and are hoping to continue using that as we migrate over. It does run on its own hardware with its own storage. Veeam has some nice features for restoring, and we will be testing that out using their recently released Proxmox integration to see how it functions as well.

I guess that you will then want to stick with Veeam as your primary backup tool for VMs and application aware backups. PBS (for which you woudn't need to buy an subscription, like PVE it's free to use without support but with a nag screen) however could be used for snapshot-like functionality if you need to use a storage backend which doesn't support snapshots per se. If you prefer to use the enterprise repository and have support you would need to buy a subscription off course.
If you are open to switch your vm backup software PBS might be worth a look due do it's good integration in PVE. It doesn't support cloud backups and application aware backups though. However in the forum are some threads, where some companys use PBS for the VMs and Veeam for their database backups to safe costs (since Veeam only needs to be licenced for the database VMs). Might be worth a look for your environment, if such a mixed approach might safe some costs and work better.

Please keep the considerations and ideas coming... I probably won't check in again for a couple days, but I truly appreciate this community and the kindness everyone is showing here.

You are welcome

 

[bbgeek17]

I'm still not shure to loose thin provi and snapshots if using the dell powerstore as when config iscsi you get a virtual disk at pve host and then you could do the same as a pve inst. itself with pvcreate, lvcreate lvm and lvm-thin vol and so it's same functionality there.

Thin LVM is not concurrent/shared access safe. You will have data loss if you manage to defeat PVE bumpers and implement it as shared storage.

we've been using Veeam to backup the VM's in VMWare, and are hoping to continue using that as we migrate over.

We were not yet able to qualify Veeam as a safe backup option with PVE. There is a thread about the issue both here and on Veeam forum. As soon as the patch drops we will test it again.

ZFS over ISCSI would work but I'm not sure whether ZFS will play nice since OP wants to use RAID5/6

ZFS/iSCSI is not an option with PowerStore.

however could be used for snapshot-like functionality if you need to use a storage backend which doesn't support snapshots per se

Backup Snapshot mode is not the same as storage Snapshots. They are two different technologies sharing the name.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[geekspaz]

I didn't know there was a bug with veeam... I'll look for that.

ISCSI actually is available with PowerStore, and the quote we got from Dell incorporates it. It might be something newer... I have no idea. I just know it's there. (Unless you mean Proxmox <-> PowerStore iSCSI connection isn't supported, then I might be back at square 1)

 

[bbgeek17]

ISCSI actually is available with PowerStore, and the quote we got from Dell incorporates it.

You misunderstood. The comment from @Johannes S and my follow up on it were referring to a particular storage type scheme available in PVE: ZFS over iSCSI (ZFS/iSCSI). This approach has multiple requirements absent in Powerstore:
- ability to ssh into the system
- ssh as root
- using ZFS as storage management for internal organization in the storage appliance
- ability to manipulate such storage organization via ZFS native commands inside the appliance
- presenting iSCSI via one of the supported iSCSI daemons
etc

There is some level of ZFS/iSCSI support for TrueNAS, however its not official.

In summary: if you plan to use Powerstore, your options (depending on feature set you bought) are:
iSCSI
FC
NFS

Each one has its own advantages and drawbacks. You do not have native PVE option that is equivalent to VMware vmfs/datastore.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[geekspaz]

Yep... I misunderstood.