Shared storage over Fiber channel

[kosnar]

Hi all,

I am looking for a fast storage for HA cluster. Currently I use Ceph, but it is terribly slow (I have 3 nodes with enterprise class nvme drives. I read many articles about it and its performace. By my opinion the bottleneck is in network synchronization. So I am looking for another solution.

According to https://pve.proxmox.com/wiki/Storage and several other post I tried Fiber Channel Storage (SAN) + shared LVM (LUNs mapped via multipath, shared LVM on top of them). This solution has better performance, but I miss the snapshot feature.

So the questions:
1) Is there another possibility for FC SAN then LVM? (requred features shared, live migration, snapshots, live backup)
2) Is there another storage as a alternative to Ceph but much faster.
3) Why LVM supports live backup but not support snapshot?

Thanks R.

 

[waltar]

1) For FC SAN instead of LVM you could use a filesystem and nfs (likely ha setup) for shared, live migration, snapshots, live backup.
2) To circumstate ha-nfs take a look at blockbridge from user bbgeek17 which solution is integrated into Proxmox or eg. ocfs2. EDIT: or gfs2.
3) LVM supports snapshot but that's depending on local/shared access and webui integration in pve.

Normally you can get every storage to work with pve but all their functionality isn't avallable from pve webui and you have to do them
by host cronjobs or manually on shell, eg. reset a vm/lxc from snapshot etc, so it's just a question of comfort and knowledge.

 

[bbgeek17]

3) Why LVM supports live backup but not support snapshot?

In short, the way the metadata is allocated is not friendly for multi-writer access.
You can find more details here https://kb.blockbridge.com/technote/proxmox-lvm-shared-storage/

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[VictorSTS]

Just curious: how much slow is your Ceph and with what infrastructure (Severs, OSD, network)? I would like to check your numbers with some that I've got from some clusters. Do your apps use Firebird database by chance? Got a few cases with that workload that I'm still to get working properly under Ceph (they queue all iops in 4k-q1 and perform poorly compared to local disks).

 

[RonaldDJ]

Just f.y.i.

Im using proxmox in a cluster setup with a fiberchanel network and Synology as NFS storage.
With "Snapshot Replication" from Synology local snapshots are created and a copy to an external location (also a synology station) is done by a replication job.

Works flawless.

 

[VictorSTS]

Im using proxmox in a cluster setup with a fiberchanel network and Synology as NFS storage.
With "Snapshot Replication" from Synology local snapshots are created and a copy to an external location (also a synology station) is done by a replication job.

If understand this well, you are using NFS to access the storage, not FiberChannel even if your network uses fiber instead of cooper? Or can a Synology share some volumes using FC?

 

[DBS]

I do use the fiberchannel to access the storage, this is a 10Gb network (seperate up and downlink)
Synology is capable of sharing folders using the NFS protocol, in this way the cluster nodes have access to the shared storage.

The synology itself can create the snapshots an external backup using replication.

 

[bbgeek17]

I do use the fiberchannel to access the storage, this is a 10Gb network (seperate up and downlink)

It seems to me that you mean Fibre Channel SFP ethernet transceiver, while the conversation is about Fibre Channel storage transfer protocol and associated connections.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[DBS]

It seems to me that you mean Fibre Channel SFP ethernet transceiver, while the conversation is about Fibre Channel storage transfer protocol and associated connections.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

You are right, apologies.

Regards,
ronald