New cluster - storage solution

[dignus]

For a new cluster we are investigating storage solutions. Cluster(s) will be 20 nodes in size, so nothing local and had to do failover and such.
Current clusters running on ultra-low-latency nvme, so ceph, NFS and zfs over iscsi ruled out.

Been speaking to various suppliers this week. Some of them offer a solution not natively integrating with proxmox, some of them. Well.. one does: blockbridge.

Does anyone have experience running on blockbridge, since they seem to be the only guys to have a native integration with promox?

@bbgeek17 you are not allowed to comment, although I do respect your dozens of helpful comments on various storage related topics

 

[nightowl]

@dignus - we have been a Blockbridge customer since 2016 after having tried just about every solution imaginable with our various hypervisor platforms (VMware, KVM on OpenStack, and now KVM on Proxmox). Every solution had their issues, except for Blockbridge. We have continued using Blockbridge for all of these use cases in many production systems across many Blockbridge clusters, and continue to expand (I think we're up to 5 BB clusters now?).

Reliability is THE most important part of a storage system, with stability, support, maintenance, and performance/throughput/latency coming in close seconds. I can honestly say that Blockbridge checks the boxes on all of these, which is honestly a pretty tough ask, and something no other vendor has been able to do (we have tried at least 5 other vendors in our history, some with horrific outcomes).

We have been using Proxmox with the NVMe-oF protocol with their driver in Proxmox to one of our Blockbridge clusters (two "complexes) with 20 servers so far, and haven't had any issues at all. Extensive reliability and performance testing has been done since we are incredibly sensitive to any issues whatsoever. Blockbridge has tuned the system to work exactly the way we want it, down to the Mellanox NIC settings, while understanding the switch configuration we have.

Troubleshooting is always tough, and Blockbridge has always been willing to assist more than we expect. For example, we have seen the case where multiple switches cascaded between servers and the Blockbridge system produced a noticeable latency difference - compared to servers and Blockbridge systems connected to the same switch. Blockbridge actually performed some latency measures for us to understand just how important port-to-port latency on switches can be! For example, Mellanox versus FS switches have a pretty significant port-to-port latency difference (Mellanox being half that of the cheaper FS switches), causing QD1 IOPS to be halved.

I would be happy to answer any questions about our environment (or at least what I can say without violating our internal privacy policies).

 

[ejmerkel]

@dignus We have been a Blockbridge customer for just about a year now and are in the process of installing our second set of storage units in another datacenter. So far things have worked flawlessly. The folks at Blockbridge are great to work with in terms getting setup and long term support. Thankfully we've only had one instance to reach out to them for support and they were extremely helpful even when the issue turned out to be a kernel bug and not an issue related to storage at all. The Blockbridge servers also phone home and are monitored by them and if there are any problems, they are available 24x7 to address them. We're a small organization and don't have a dedicated storage administrator so having Blockbridge there to backstop us is helpful.

Because their driver directly integrates Blockbridge storage with Proxmox, we've rarely had to login to the Blockbridge units for anything. The performance and reliability has been outstanding. For us the solution has been set it and forget it.

I am also happy to answer any questions you might have relating to our environment or our Blockbridge setup.

 

[anush.intech]

@dignus Our team had been using Ceph before, and we were constantly dealing with slow read and write speeds. But since we switched to Blockbridge, everything has been running so much faster delivering great IOPS performance as well.
What I really appreciate about Blockbridge is its flexibility. We're able to customize our storage setup to fit our specific needs, and the system can easily adapt as our workloads change, it also supports live cluster to cluster migration which proxmox has introduced recently. Plus, the user interface is clean and easy to navigate, so managing our storage environment is a breeze.
Another big plus is the Blockbridge team themselves. They're incredibly knowledgeable and responsive, always willing to help us troubleshoot any issues that come up. It's clear that they really care about their customers and are dedicated to providing the best possible service.

I would say go for it.

 

[dignus]

Thanks for the replies! One thing I do wonder is how the compression performs in a real life scenario and what the compression gains are.

Reason for asking: currently we have a storage solution that supports trim and thus we have only 65% of allocated storage in use in the storage system. I understand BlockBridge doesn’t support trim, but compression might help in this case.

 

[anush.intech]

Thanks for the replies! One thing I do wonder is how the compression performs in a real life scenario and what the compression gains are.

Reason for asking: currently we have a storage solution that supports trim and thus we have only 65% of allocated storage in use in the storage system. I understand BlockBridge doesn’t support trim, but compression might help in this case.

Hi,
You are right on the part, the trim is not supported. But, I rather felt it was never required because normally what trim does is it clears off white space and gives it for reuse. In BB, their thin provisioning only stores the blocks with data in it. If its a whitespace, it simply does not consider it used. Hence, you don’t need to trim anymore. In fact we used to trim each of our VMs in ceph but now not a single trim is required. Also, we have not tested compression yet but it depends on what type of data are you storing and accordingly you might be able to save around 20% more i guess.

 

[Martin Vassilev]

I tried CEPH on the proxmox machines, it was not good. Since I have a dedicated blockbridge storage, there are no problems at all. Low latency, high performance.
They have scheme with only 2 node , full replication , one VM voter for quorum which is perfect for me.
I use it with lvm over iscsi.

 

[pille99]

no offence but these Serversare no high performance. they wrote on the website to one storagebox 3 million IOPs for round about 20 Drives. this is the cheapest enterprise hardware like netapp, dell, ibm etc. buy cheap shit and charge a lot. how i know ? i built a server myself, with the goal only one bottleneck which sets the overall speed. my harddrive, and only ONE of them, hat 10 million IOPs. that is high performance.

dont get me wrong - the complete equiptment is highly adusted to proxmox and works very well, but far away from real high performace

 

[dignus]

Yeah, not 10 miljoen QD1 random IOPS

 

[iptgeek]

@dignus

Another +100 for Blockbridge. Absolutely hands down the best storage solution that we have come across for Proxmox. It just works - really nicely integrated with PVE and the blockbridge team are really excellent to deal with - nothing is ever too much trouble. Really happy to chat off forum, PM me if you think that would be helpful.

 

[bbgeek17]

Hi @pille99 ,
No offense taken. 10 million IOPS is fantastic! That would require 16+ of the top-end, latest generation, Gen-4 micron 9400s running at maximum load. You have a supercomputer, for sure!

If suitable for datacenter use, we'll look into adding your configuration to our list. Please DM me some information (I want to avoid taking over the thread).

• Are you measuring sustained write IOPS without drive caching?
• Is the storage fully redundant (i.e., supports drive loss)?
• Does the configuration support power loss without data loss?
• What drives are you using?

And to clarify, we don't sell hardware. The platforms on our site are configurations that we recommend for performance, reliability, and global accessibility of replacement parts. Customers are free to choose and supply their hardware.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[dignus]

I bet you @pille99 is testing on a RAM drive. And a fast one at that

 

[alexskysilk]

i built a server myself, with the goal only one bottleneck which sets the overall speed. my harddrive, and only ONE of them, hat 10 million IOPs. that is high performance.

It is POSSIBLE to do what you're after on your own.

If you dont have any delivery deadlines, can solve all problems on your own in a timely manner, have reliable and tested failover/failback, and have the budget to replace/wait for all the missteps during design and implementation. In other words, if your time has no value and there's no real cost to downtime.

the reason commercial solutions costs as much as they do (well, discounting sales overhead) is because the vendor takes all that away from you and onto themselves. Folks who run storage for real can't afford to spend their entire professional life fixing holes and putting out fires in production- also, its very useful to blame the vendor when shit goes sideways and your job isnt in jeopardy

 

[pille99]

No offense taken. 10 million IOPS is fantastic! That would require 16+ of the top-end, latest generation, Gen-4 micron 9400s running at maximum load. You have a supercomputer, for sure!

If suitable for datacenter use, we'll look into adding your configuration to our list. Please DM me some information (I want to avoid taking over the thread).

it was the goal, to create a real high performance Server. i think i archived it. but the main mission was: a server (in this case a cluster with 10 hardware servers) which doesnt need an aircondition. 2 years ago, i finished a test, 24 hours on 23 degrees (running only 5 servers). from the numbers my configuration is multiplied 11 times faster as a 2.5 mil Netapp config (found a PDF file on netapp). in my tests (i used esx with vsan) copy a 5 gb file was almost not shown up anymore - maybe 1 or 2 secs (windows). i also use a different hardware configuration and not server, network storage. the project is frozen because i need an investor and its difficult to find (i am not a sales person).
but i dont want to flame the threat. btw: a proper configuration contains multiple parts, the storage is just one.
dont get me wront - the mentioned storage is, what the comments confirmed, highly adjusted to proxmox, which makes it a good product.