This might have been asked before and maybe there are or they were reasons not to consider this. However, I'd like to start a public discussion about SDS in PVE and how to enable or optimize the PVE stack for various use cases.
So for now we've got support for classic three-tier-infrastructures using NVME-oF, iSCSI and FC as well as Ceph as a real distributed SDS plus replicated ZFS. That's more choices that anyone else offers bundled into the product.
I'm into HCI for many years already and evaluated almost every product on the commercial and open source market. I've been designing and building vSAN architectures with thousands of NVMe devices and the majority of evaluations on my lab hardware is SDS.
While I agree that Ceph is a real great and strong competitor to vSAN at scale, it lacks the performance at the edge. This is, where DRBD/LinStor comes into play. I operate my own 2-node (plus qDevice) edgle cluster that's build out of 2 Dell R740 with 9 SAS HDD and 4 NVMe each. Everyone who tried to use SAS HDDs or just 2 hosts with 4 SSDs each with Ceph knows, that it works but doesn't perform at all. On the other side, this 2-node scenario is kind of perfect for a DRDB mirrored setup and so I went with LinStor, replicating using 4 direct 25GBe links where NVMe is my T0 and HDD is my T1 storage. This setup is working and performing for more than two years already. Compared to replicated ZFS it's sync-replicated and has a very (!) low memory footprint. Using LinstorGateway it also offers additional interfaces that can be used by the workload directly (k8s i.e.).
Q: Fine, but if everything's perfect, what's the case for this thread?
A: If you're a geek like I am, that's all fine. If you want to sell it to the customers, things get more complicated. Most customers expect:
So for now we've got support for classic three-tier-infrastructures using NVME-oF, iSCSI and FC as well as Ceph as a real distributed SDS plus replicated ZFS. That's more choices that anyone else offers bundled into the product.
I'm into HCI for many years already and evaluated almost every product on the commercial and open source market. I've been designing and building vSAN architectures with thousands of NVMe devices and the majority of evaluations on my lab hardware is SDS.
While I agree that Ceph is a real great and strong competitor to vSAN at scale, it lacks the performance at the edge. This is, where DRBD/LinStor comes into play. I operate my own 2-node (plus qDevice) edgle cluster that's build out of 2 Dell R740 with 9 SAS HDD and 4 NVMe each. Everyone who tried to use SAS HDDs or just 2 hosts with 4 SSDs each with Ceph knows, that it works but doesn't perform at all. On the other side, this 2-node scenario is kind of perfect for a DRDB mirrored setup and so I went with LinStor, replicating using 4 direct 25GBe links where NVMe is my T0 and HDD is my T1 storage. This setup is working and performing for more than two years already. Compared to replicated ZFS it's sync-replicated and has a very (!) low memory footprint. Using LinstorGateway it also offers additional interfaces that can be used by the workload directly (k8s i.e.).
Q: Fine, but if everything's perfect, what's the case for this thread?
A: If you're a geek like I am, that's all fine. If you want to sell it to the customers, things get more complicated. Most customers expect:
- A single control plane
- A unified software maintenance / patch lifecycle management
- At least a roughly tested baseline from the manufacturer
- No hassle to validate various HCLs against each other before performing updates
