Shared storage over Fiber channel

K

kosnar

New Member
Aug 23, 2024
13
1
3
Hi all,

I am looking for a fast storage for HA cluster. Currently I use Ceph, but it is terribly slow (I have 3 nodes with enterprise class nvme drives. I read many articles about it and its performace. By my opinion the bottleneck is in network synchronization. So I am looking for another solution.

According to https://pve.proxmox.com/wiki/Storage and several other post I tried Fiber Channel Storage (SAN) + shared LVM (LUNs mapped via multipath, shared LVM on top of them). This solution has better performance, but I miss the snapshot feature.

So the questions:
1) Is there another possibility for FC SAN then LVM? (requred features shared, live migration, snapshots, live backup)
2) Is there another storage as a alternative to Ceph but much faster.
3) Why LVM supports live backup but not support snapshot?

Thanks R.
 
1) For FC SAN instead of LVM you could use a filesystem and nfs (likely ha setup) for shared, live migration, snapshots, live backup.
2) To circumstate ha-nfs take a look at blockbridge from user bbgeek17 which solution is integrated into Proxmox or eg. ocfs2. EDIT: or gfs2.
3) LVM supports snapshot but that's depending on local/shared access and webui integration in pve.

Normally you can get every storage to work with pve but all their functionality isn't avallable from pve webui and you have to do them
by host cronjobs or manually on shell, eg. reset a vm/lxc from snapshot etc, so it's just a question of comfort and knowledge.
 
Last edited:
Just curious: how much slow is your Ceph and with what infrastructure (Severs, OSD, network)? I would like to check your numbers with some that I've got from some clusters. Do your apps use Firebird database by chance? Got a few cases with that workload that I'm still to get working properly under Ceph (they queue all iops in 4k-q1 and perform poorly compared to local disks).
 
Just f.y.i.

Im using proxmox in a cluster setup with a fiberchanel network and Synology as NFS storage.
With "Snapshot Replication" from Synology local snapshots are created and a copy to an external location (also a synology station) is done by a replication job.

Works flawless.
 
RonaldDJ said:
Im using proxmox in a cluster setup with a fiberchanel network and Synology as NFS storage.
With "Snapshot Replication" from Synology local snapshots are created and a copy to an external location (also a synology station) is done by a replication job.
Click to expand...
If understand this well, you are using NFS to access the storage, not FiberChannel even if your network uses fiber instead of cooper? Or can a Synology share some volumes using FC?
 
I do use the fiberchannel to access the storage, this is a 10Gb network (seperate up and downlink)
Synology is capable of sharing folders using the NFS protocol, in this way the cluster nodes have access to the shared storage.

The synology itself can create the snapshots an external backup using replication.
 
kosnar said:
Why LVM supports live backup but not support snapshot?
Click to expand...
Live backup is independend of the storage plugin and implemented in QEMU. Sadly both have snapshot in their name, yet work on different layers.

kosnar said:
Is there another storage as a alternative to Ceph but much faster.
Click to expand...
CEPH and your SAN are both shared storages, but work on different levels: CEPH is a distributed shared storage, as the data is distributed over different nodes and a SAN is normally a dedicted shared storage system, e.g. a box with the storage in it and some network protocol to access them. Both are not interchangeable and you will not have a great experience running CEPH on LUNs of SAN. You built a SAN-on-a-SAN setup.
 
  • Like
Reactions: Johannes S
kosnar said:
Hi all,

I am looking for a fast storage for HA cluster. Currently I use Ceph, but it is terribly slow (I have 3 nodes with enterprise class nvme drives. I read many articles about it and its performace. By my opinion the bottleneck is in network synchronization. So I am looking for another solution.

According to https://pve.proxmox.com/wiki/Storage and several other post I tried Fiber Channel Storage (SAN) + shared LVM (LUNs mapped via multipath, shared LVM on top of them). This solution has better performance, but I miss the snapshot feature.

So the questions:
1) Is there another possibility for FC SAN then LVM? (requred features shared, live migration, snapshots, live backup)
2) Is there another storage as a alternative to Ceph but much faster.
3) Why LVM supports live backup but not support snapshot?

Thanks R.
Click to expand...

Consider local NVMe per node + LINSTOR (DRBD9) replication—usually lower latency than 3-node Ceph for 4k/QD1, full Proxmox integration; snapshots via ZFS/LVM-thin, live backup via QEMU.
Caveat: not self-healing like Ceph and best for small/medium clusters—would 2× replication + re-sync on failure meet your HA/RPO needs?
 
You must log in or register to reply here.
Top Bottom