> Thanks a lot for that tip! Blockbridge sounds really interesting. You also write that you are actually using Blockbridge. Any notable requirements or things to consider when designing a solution with it? Which integration are you using in Proxmox for that storage?
We are using Blockbridge's NVMe-oF plugin for Proxmox which has worked flawlessly. It is very easy to configure (we scripted it easily). Essentially, each virtual disk has a virtual LUN created and mapped to QEMU automatically. All of the native VM storage functions of Proxmox work as you would expect (snapshots, storage live migrations, VM live migrations between hosts, backups with Proxmox Backup, etc.).
Blockbridge supports VMware and all of its features including VAAI, as well as compression, VVOLs, etc., but I wouldn't recommend anyone choose VMware for any new projects. For those environments that can't switch, Blockbridge is extremely fast storage for VMware workloads.
Ceph is also an option. We have quite a few Ceph clusters (10?), with Petabytes of HDD storage as well as about a Petabyte of SSD storage. That said, the big issue with Ceph is performance, specifically latency and write iops, especially for erasure coded pools. It makes sense that it costs of lot in terms of latency due to CPU usage and network latency to compute the CRUSH rule(s) and replicate blocks, including its use of an object storage system as the base for block storage (RADOS). Bandwidth is plentiful with larger block sizes, as long as latency isn't a concern. Our Blockbridge systems have about 1/10th the latency of Ceph, so are more suitable for high-performance workloads that require this level of latency. Ceph SSD storage is our go to for "general purpose" storage (think boot drives or drives that are not used all that much).
Ceph has had quite a few bugs which can cause high stress (human stress) when upgrading large clusters, so plenty of testing and planning is needed for high-availability. There are also situations that can cause disruptions such as network flapping if a switch is failing or ports go bad. Flapping can really cause confusion for any distributed system, if it happens fast enough. We have had situations where Ceph got confused as to which object was newer in a triple replicated configuration when one of our switches decided it was going to die soon.
There are also cases where a single bad OSD (either software crashing or SSD hardware issues) can cause latency in the Ceph cluster. Finding and failing it takes a bit of work unless you have significant monitoring in place, but without automation, it can become an issue pretty quickly.
In the end, Ceph can be great for a lot of things, but it's definitely not "free". It requires quite a bit of knowledge to manage properly and due to its distributed nature, you have to understand distributed systems and the various failure modes. It definitely isn't something you set up and forget.
Blockbridge upgrades are all managed and have 24x7 support. We've been running their systems for more than seven years and have never had any issues. Updates are transparent and non-disruptive, including the Proxmox plugin updates on Proxmox nodes.
If you are running a high-end enterprise Fibre Channel setup, you could add Blockbridge on top. We did this years ago with a multi-PB fiber channel SAN, migrating it slowly to iSCSI on Blockbridge. But it only makes sense if you're dealing with expensive storage since you'll need to buy servers for Blockbridge anyway.
NFS is another option, but QCOW performance is a somewhat poor and there are currently some limitations around snapshots. Finding a highly-available low-latency NFS solution is tough, though.
Note that we only use redundant "everything", so switches (with MLAG), NICs, servers, etc. are all redundant, both for Blockbridge and Ceph systems. It is really the only practical approach to gain as high of availability as possible for network and storage connectivity. In "huge" environments, where an entire rack is a failure domain, I may not recommend this, and let a whole rack fail instead of dealing with the complexities added with these redundancies, but for most environments, this isn't reality.
> OK, what's a big deployment in your case? Also how fast is the network connection you are using, 25 GbE, 100 GbE or even faster?
Just to answer some of your questions that you asked others - we use both 25Gbps and 100Gbps switch ports throughout our environment, with at least 2 active 100Gbps connections connected to active Blockbridge nodes, along with multiple 100Gbps dedicated connections between physical servers that make up a Blockbridge single "node" (a pair of servers). Our servers have dual 25Gbps and some have both dual 25Gbps and dual 100Gbps for separation of workload and storage traffic. We have tested 200Gbps connections too, which are commonly used for inter-Blockbridge server communications where latency must be minimal, but with plenty of available bandwidth for replication.
> I have lot's of experience with CEPH but only with a fraction of the capacity needed now, about 30-40TB. With capacity needs which are 20 times more, I'm fairly concerned about CPU time needed and also about performance and stability, if you scale up CEPH to hundreds of TBs.
It depends on how dense your storage nodes are, as well as how much storage traffic is required. If you only have 5 nodes and each has 750TiB of storage in them, you have to really start to think about how long it will take to recover a node this size, and you should be using dedicated nodes just for storage. Even 64-core nodes are going to use quite a bit of CPU capacity to manage this much storage. Plus, your metadata servers (MONs) should be plenty fast with low latency storage.
Think of this way, with 768,000GiB of available storage (750TiB), where it is 70% full, with 3GiB/sec of recovery performance (triple mirror would cause 9GiB/sec of writes in most cases), you would be waiting nearly 50 hours for it to recover. It might be faster, but likely not due to OSD software throughput limitations. Large storage nodes have this disadvantage - the time to recovery starts to become "scary".
> From
@ness1602:
> Big deployments are ceph storages > 100TB i think in my case. We usually start with 10g, but now are moving to 40g, and probably next year to 100g ,because the equipment is finally afordable. For corosync 1gb is enough, and everything is redundant through bonding .
> Stability is okay, i don't usually go for more than 20 nodes , i tend to break clusters up just in case.
Note that having multiple independent connections for Corosync is needed for production environments. Even with bonding (LACP connections), the time for an LAG to re-establish its connection after a link failure may be too long for Corosync.
Plus, there are situations where two switches (stacked switches, for example) can take a while to fail-over and may disrupt traffic, which can cause Corosync to fail that node. The same can happen if there is ever an issue MLAG communication for whatever reason (switch software bugs, bad ports, bad cables, etc.).
I speak from experience that it is highly advised to use both 1Gbps and 10Gbps/25Gbps/100Gbps connections on a server for two Corosync links.
One last situation that can arise is saturation of bandwidth of a port/LAG. If there is enough packet loss due to the port being saturated with traffic, Corosync traffic won't be forwarded in time and will fail. This can easily happen on a 1Gbps port (or even 2x1Gbps LAG). Some documentation indicates to dedicate ports to Corosync, but that isn't very practical.
20 nodes is fine. The reasoning behind this is that Corosync is a single master that replicates to all secondaries, and this master is round-robin'd between all nodes in the cluster. The more nodes, the longer this takes, and the more potential delays can be incurred if some nodes have (relatively) long waits for tasks to finish while it is their turn to be master.
> - What about GlusterFS? Is it stable enough to serve large amounts of storage in an enterprise setup?
GlusterFS is essentially deprecated and has been mostly used as a file server replacement, not a VM host storage platform. CephFS is RedHat's answer, even though it is significantly more complex than GlusterFS since it requires Ceph.
Hope this helps!
) Anyway, that's semantics. 