I have a cluster made of three PVEs. Without considering virtual machines and data backup, which storage method is the fastest?
- Thread starter ggea
- Start date
ZFS would also be local storage. Replicated local storge but still local storage. Local storage should be faster as there is no network latency added, but you will miss alot of the features and redundancy you usually want when running a cluster.
And you didn't told us what hardware is used? For ceph you for example want as fast as possible NICs (40Gbit, 100Gbit and so on) and enterprise grade SSDs with PLP.
And you didn't told us what hardware is used? For ceph you for example want as fast as possible NICs (40Gbit, 100Gbit and so on) and enterprise grade SSDs with PLP.
I am not sure if you really want plain the fastest. Have you thought about what happens if a physical drive fails? You may want to set up your storage redundant so in case one drive fails, not all data is lost.
Hello @ggea ,
For full disclosure, I work for a storage company. We routinely have folks reaching out with the exact same question.
The first thing you need to do is define "fastest." If you are looking for the best interactive performance, you are likely interested in latency. The latency issues with distributed and hyper-converged block storage are architectural and common. As others have pointed out, upgrading and expanding your hardware is a possible route to improving latency. However, remember that you cannot achieve the latency of local storage.
Consider a commercial solution if low latency and high availability are mission-critical. If a commercial solution is outside of your budget, you are probably best sticking with Ceph if high availability is required. Otherwise, get yourself a good RAID controller that supports NVMe.
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
For full disclosure, I work for a storage company. We routinely have folks reaching out with the exact same question.
The first thing you need to do is define "fastest." If you are looking for the best interactive performance, you are likely interested in latency. The latency issues with distributed and hyper-converged block storage are architectural and common. As others have pointed out, upgrading and expanding your hardware is a possible route to improving latency. However, remember that you cannot achieve the latency of local storage.
Consider a commercial solution if low latency and high availability are mission-critical. If a commercial solution is outside of your budget, you are probably best sticking with Ceph if high availability is required. Otherwise, get yourself a good RAID controller that supports NVMe.
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
I am using local ZFS with scheduled replication to simplify my setup. I've tried CEPH in the past and always had issues with VMs running very slowly whenever there is something major going on with CEPH. With local ZFS it only impacts one node vs several. Granted I lose HA but ZFS replication is good enough for me as most of the VMs are static. Plus I run the backups several times a day using PBS.
an array of nvme in RAID10. But that is a pointless question as speed is not the only consideration for storage, or even the most important.