CEPH on three separate nodes

[mk2024]

Hi,
A client want to have 3 nodes in 3 datacenter with iSCSI local storage connected, it's possible to configure on the top one shared storage for all hosts?? CEPH??

 

[bbgeek17]

Does the client have Dark Fiber between the 3 datacenters with very low latency? If they do, then its theoretically possible. But there are no guarantees, of course.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[mk2024]

There is 1 ms (10GB/s local network) latency beetween nodes between all sites and 2x25GB/s iSCSI to storage

 

[bbgeek17]

In theory, it might work.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[VictorSTS]

Every write and around 2/3 of the reads will be impacted by that 1ms latency, which is at least like two orders of magnitude higher that that of local drives. Besides, Ceph is supported on local drives only. This may be doable but will be under performing and unsupported.

 

[alexskysilk]

A client want to have 3 nodes in 3 datacenter with iSCSI local storage connected

This is confusing. what is meant by "iscsi local storage?"

it's possible to configure on the top one shared storage for all hosts??

sure.

CEPH??

not on top of iscsi. theoretically possible but pointless; iscsi is already sharable.

 

[bbgeek17]

This is confusing

My read was that Op has 3 DCs, each having SAN box attached to the local server in that DC.
I do agree that this setup will end up in a significant storage waste. Ceph replication on top of local RAID5(?) in SAN.
As you said, theoretically it might work. How well it works and whether it is worth it - is up to Op to decide.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox