Storage for small clusters, any good solutions?

[RoxyProxy]

Hi there,
the Title may be a bit deceptive as I know there are good solutions that work for many but for me/my workplace we face a bit of a dilemma.

I know I'm opening this can of worms again and this is also partly me venting a bit of my frustration and I'm sorry about that.

We wanna use Proxmox VE more or rather be able to get more of our customers to be able to switch to it. Many are already interested but for most of them the lack of a true cluster-aware filesystem is a problem.


I know that you can have a shared-lvm on a SAN storage and now even make snapshots with it since Version 9.0 and TPM Snapshots since Version 9.1 (if still not live and snapshots as volume chains are in tech-preview still), but then the problem is that everything is thick provisioned. So 2 Snapshots of a 500GB VM is 1.5TB of space used for just a single VM.


I also know that there is Ceph but I have some gripes with that too:
1. I find it a bit complicated and Im afraid one thing wrongly configured can have huge consequences in the long run (but that's my problem and I do need to do more research)

2. It needs 3 Servers minimum which is a problem as many of our customers that want to migrate have 2 Node HA Clusters running Hyper-V or VMware ESXi + a San storage (often direct attached) , so it would be a big investment getting another server (or 3 to renew the entire environment) +100GBit Network hardware and enough internal Storage on every host to cover for the storage you can't actually use.

3. I also read a couple of times that a 3-Node Ceph cluster, while technically possible, is less than optimal for production use and can be fragile if a node or a couple of disks fail.

So all in all ceph would also not be a viable solution for these customers and would only really be a good option for bigger customers and 5 or more servers.
Please do correct me if I'm missing something or if I'm completely in the wrong.


Are there any NAS Systems or renowned Storage Manufacturers (Dell, IBM, Lenovo) that have Solutions that can be used for ZFS over iSCSI?
I researched into that as well because in the Storage Table its listed to also have full functionality for snapshots and thin-provisioning while shared and fully supported by the proxmox team. It seems to be a rather obscure thing though, I couldn't find a whole lot about ZFS over iSCSI.


I also know that Cluster-Aware File-Systems like GlusterFS and OCFS2 exist and can work, I tried myself with OCFS2, but it's important for us that the technology used is officially supported by the proxmox team, so if there should be any problems that we cannot fix, the proxmox support won't handle it on a best-effort basis.


I want to ask if anyone has experience with blockbridge? It looks interesting.
If so, how is the support and is there support in the EU/Germany? Which hardware is needed?
Would a blockbridge storage work as a sort-of direct replacement for a SAN-Shared storage that was directly attached to 2-Hyper-V Servers for example?
It'd be great if anyone could share their experience with them.


Lastly, are there any other solutions that I'm missing, that are stable and tick the boxes of being thin-provisioned and have snapshot supported while being a shared storage?

-note: It's not that the customer must keep all their old hardware or stay on a SAN storage necessarily (that would be great though), if e.g. a blockbridge storage can be used as a viable alternative to a e.g. IBM FlashSystem 5015 or something like it, then buying then new storage would be fine. We just need a working solution that ticks all the boxes and doesnt mean replacing everything or being too expensive.

Thank you in advance, any guidance would be greatly appreciated!

 

[Cold_Cloud_1]

I have thus far migrated 5 of our clusters from Vmware to proxmox using a variety of storage systems. (on top of all our new deployments being Proxmox)
We did not elect to go down the ceph route for some of the reasons you mentioned as well as the additional maintenance expertise/overhead.
For smaller sites with 2 hosts ZFS replication works great and is cheap to setup.
Other sites we had existing Netapp storage using NFS and FC. NFS is simple to setup and makes the migration from Vmware very quick and easy but the performance leaves much to be desired, even with dual 40gb nics on the servers dedicated to storage traffic. Proxmox could certianly improve its implementation of PNFS (multipath), have tried many custom configurations to try and speed performance with marginal returns.
FC setup in proxmox is possible but not fun.

Blockbridge has been our favorite thus far, very simple to setup, tight integration with proxmox, not vendor locked to proprietary hardware like Netapp, and the support is awesome (real person 1 email/ call away). Speeds are even faster than what we were seeing with an all NVME Netapp array to vmware with NVME/TCP transport. We are migrating 2-3 sites a year to Blockbridge from Netapp and exclusively utilizing NVME/TCP transport. Even if speeds/costs were equal to Netapp I would choose blcokbridge just for the support, an area Netapp is severely lacking as of late.
All of our blockbridge deployments have been fresh with new NVME storage but I know they can be a front end to expose your existing storage array to proxmox, this might be a good route if you have a large array that still has life left in it.
Happy to discuss what has worked for us if you have more specific questions feel free to ping me.

 

[nomercymayhem]

I have experience using Blockbridge storage. When Broadcom decided to basically stop doing business with my company, we decided to migrate our cluster over to Proxmox. I didn't think migrating would be a big deal since we had a robust, reliable NetApp array, but as you mentioned, we started running into conundrum after conundrum after conundrum when comparing how Proxmox handled shared storage vs VMware's VMFS setup. We wanted to enjoy the same highly available and redundant storage setup we had with VMware, but this didn't seem possible.

After a ton of research, I landed on Blockbridge. The draw was this: fully redundant, high performance, highly available, use of off-the-shelf hardware, 24-hour support, vendor-managed updates, array-controlled creation/destruction of volumes, multipathing, and more. To be honest, I was blown away by their demo. I talked to at least one of their customers (an MSP) before I took the plunge. The MSP had been using Blockbridge for a year or two and had nothing but good things to say about the product, inlcuding interactions with the team and the ease of the overall setup.

For our use-case, we have a small four-host cluster that houses a couple hundred Windows VMs. The bill-of-matierals from Blockbridge that matched our needs included two Dell 1U servers and 10 Micron NVMe drives for a total raw capacity of ~30TB. The two servers have multiple 100Gb links to a ToR switch. From there, I connected my hosts up which only have 25GbE ports, but is sufficient for our needs.

Setup included basic IP gathering and network/hardware architecting to make sure the design worked for the Blockbridge team. Once the physical networking was validated, their install team helped get the array (including a cluster witness we opted to just make a VM) connected up to their servers. From there, they finished the entire setup, including all validation testing of the hardware. This part of the process was completely hands-off for me and my team.

Once completed, we were left with a 30TB storage pool that was hooked up to our Proxmox cluster. Since then, I've rarely given the array much though; it just works as advertised.

As VMs are created/deleted, the array handles it all seemlessly. Snapshots happen in an instant and are all done on the storage-side. Performance is way better than our all-flash NetApp array. Deduplication/compression appears to be signifigantly better than NetApp as well, as we're around 2.1x on Blockbridge, whereas the same VMs on the NetApp array was around 1.2x. The support team reaches out if there are any issues with the array (only happened once for us when we had a problem with the storage system our witness VM resided on). All updates are performed by the support team, so there's no need to wory about that. It's basically like have a full-time storage admin on staff.

I have no experience using them for VMware or Hyper-V, which I'm pretty sure they support. But if I was using one of those products, with what I know now, I wouldn't hesitate to price it all out. They're a great team and there are few companies out there that understand enterprise storage like the folks over at Blockbridge.

 

[LnxBil]

Are there any NAS Systems or renowned Storage Manufacturers (Dell, IBM, Lenovo) that have Solutions that can be used for ZFS over iSCSI?
I researched into that as well because in the Storage Table its listed to also have full functionality for snapshots and thin-provisioning while shared and fully supported by the proxmox team. It seems to be a rather obscure thing though, I couldn't find a whole lot about ZFS over iSCSI.

I'm unfamiliar with any vendor that supports ZFS-over-iSCSI. ZFS is not HA-capable, but some claimed 3rd party HA exists from unknown companies.
If you don't want to limit yourself to ZFS-over-iSCSI but also want explore other viable options, go with Blockbridge. They have proper integration and everything you need - as other already said.

 

[Gilberto Ferreira]

I'm unfamiliar with any vendor that supports ZFS-over-iSCSI. ZFS is not HA-capable, but some claimed 3rd party HA exists from unknown companies.
If you don't want to limit yourself to ZFS-over-iSCSI but also want explore other viable options, go with Blockbridge. They have proper integration and everything you need - as other already said.

Just to reference, there is OmniOS/Illumos which is a former Solaris clone which has some zfs-over-iscsi with COMSTAR and there's some iscsi-ha.
I never tested it, but could be worth.

https://icicimov.github.io/blog/high-availability/ZFS-storage-with-OmniOS-and-iSCSI/

 

[speatzle_]

Hi,
i work at a small to medium size IT service company in Germany, and we have been using Proxmox for ourselves and for customers since ~2020.
I myself have been using Proxmox at home since ~2016.
I was asked by Blockbridge to respond here, but these are my own experiences and thoughts.

We use a mix of ZFS, Ceph and Blockbridge.
For customers with 2 nodes, I'd recommend ZFS with Replication. It works very well and is easy to manage.
For larger clusters, Ceph works pretty well, but due to issues with usage balancing we aren't using it for anything with less than 5 nodes, it also needs constant monitoring.
With Ceph, latency is an issue and in my experience makes Windows Server feel sluggish. Hoping Crimson will improve this.
If ZFS Replication is not enough or you need minimal latency, Blockbridge is the way to go.

Blockbridge is very fast and stable. They have great support. We actually use our internal Blockbridge Storage with one of our Proxmox and one ESXi cluster at the same time. We provided our own Supermicro hardware, which we already had for Blockbridge (it has to be compatible, though). If you want a SAN-like experience for Proxmox, Blockbridge is it.

 

[ejmerkel]

I want to second @nomercymayhem regarding BlockBridge as a fantastic storage solution for Proxmox. We've been a customer of BlockBridge for a little over 4 years and have their storage solutions deployed in two different data centers in an HA configuration.

The company has been great to deal with in helping us engineer a solution and getting it deployed. The solution was custom designed based on our needs and budget. We're a small company so being able find storage that is robust yet not out of this world expensive was awesome.

They are based out of Boston I think and their staff and support has been great to deal with though to be honest our need for support is pretty rare....maybe a time or two a year if that.

 

[IsThisThingOn]

Take this with a huge grain of salt. I don't know you or your customers

IMHO you probably don't need HA. Redudant PSU and local storage is more than enough. And your next part is a good explanation why.

2. It needs 3 Servers minimum which is a problem as many of our customers that want to migrate have 2 Node HA Clusters running Hyper-V or VMware ESXi + a San storage (often direct attached) , so it would be a big investment getting another server (or 3 to renew the entire environment) +100GBit Network hardware and enough internal Storage on every host to cover for the storage you can't actually use.

That is not automatically real HA. That is what I often see in SMBs and call "Yeah it is HA, put please don't unplug or pull X".
Just having two Hypervisors that connect to a SAN, does not make for a HA setup.

So my advice is to think really hard if you really need HA. Or if maybe a cluster with ZFS and storage replication is the better option.

If you really need HA, I hope this post did not offend you.

 

[iddig]

Just my 23 years of experience and another 3 cents. I have used ceph, synology, Nimble Storage (back in the day) as local replication. About 1.5 years ago I did some research and decided to go with BlockBridge (30TB) because of the support and direct integration with proxmox. At that time I was using synology all flash. I have a 5 yr old server cluster that has been migrated through all of these storage solutions. When I went through the setup with blockbridge it was easy as the previous user had explained. Fully redundant storage on 100g switches for iscsi. Migrations and Ha work great from proxmox. I have literally forgotten about the appliances and management as its all taken care of. They let me know when a software update is available for the storage cluster, apply it without any downtime, and my users notice nothing. Its been great. The plugin for proxmox is easy to update/install following their documentation. Their support is very responsive if needed.

Previously I was using vmware as my virtualization solution. I run windows and a lot of linux containers. Research and make your choice best for your environment. The best of luck!

 

[glarsen]

We are a blockbridge customer for our cloud (2yrs, 2 zen4-48 clusters) and are working to migrate some larger customers onto their own blockbridges. Can't say enough about their storage knowledge and support. They are one of my favorite vendors to work with as they know what they are doing and are easy to work with. They've caught 1 or 2 issues in our cloud before we were aware, which was appreciated. Performant arrays as well, we have not had any customers that have been able to push our setup iops wise to the limit.

I find their pricing on par with the TCO of pure or alletra over a 5 yr period. The caveat being that your alletra or pure is a brick if you don't pay for support. Having it be on commodity iron is a nice failsafe if for some reason there was a falling out. You can always rebuild them as zfs nodes.

You will still need a q device for a 2 node cluster anyways, so you're looking at 3 devices. For some customers, a 2 node cluster with a qdevice and zfs replication between the 2 nodes may be a good fit depending on their data loss requirements. Probably would be looking at a 5 min hit for data. We do migrate a lot of customers using their existing SANs and set up ISCSI w/ multipath or NFS. It depends on the customers business requirements to advise better.

IMO, if a customer is requiring a 3 node cluster + SAN w/ controller based failover, blockbridge is hard to beat for proxmox customers. Especially if they want to login to their proxmox cluster and do any administration, there is less for them to break in this scenario vs ceph.

 

[ebaggs]

My company has been using Proxmox for a while now and have deployed in a variety of uses and I can confirm that Blockbridge has definitely been the most straight forward way to address the problems you've mentioned. While Proxmox doesn't official support Blockbridge, Blockbridges is the only Enterprise SAN that I'm aware of that natively supports Proxmox via a custom storage plugin, allowing it to bypass a lot of the limitations that you run into using a SAN with the standard LVM over iSCSI storage layers. It behaves very similarly to ZFS over iSCSI in practice, just hitting the blockbridge API to provision/modify disks rather than PVE communicating via SSH to send ZFS commands.

That being said, Blockbridge is likely overkill for an environment that only has 2 servers. If you want an easy button to drop in replace your SANs and your budget allows for it, Blockbridge would be a good fit. But depending on your customers needs, it's worth figuring whether you truly need full HA or if something like ZFS replication at a 5 minute interval would be acceptable for your environments. As much as true HA is the gold standard to strive for, the jump in cost and/or expertise to setup a true HA solution (especially with PVE) can be the driving decision to settle for a slighly less redundant option like ZFS replication since a lot of environments can afford the risk of a few minutes worth of data loss due to an unexpected failure.