About ceph over vpn performance question.

[akong]

Proxmox ceph node(A) <----> firewall <---------------internet--------------> proxmox ve client(B) (use vpn connect to ceph node and mount rbd)
My question is (A) create vm and storage set to ceph rbd.The VM very fast.
But (B) create vm and set to remote storage rbd is very slow.I has see iops almost not not working.How to test it?

 

[B.Otto]

I doubt ceph over VPN will work well, ceph needs a good network connection, and routing is not what you want for that.

 

[akong]

So, Could ceph just work on lan?

 

[B.Otto]

Pretty much, see also the requirements (https://pve.proxmox.com/pve-docs/chapter-pveceph.html):

We recommend a network bandwidth of at least 10 GbE or more, which is used exclusively for Ceph. A meshed network setup [4] is also an option if there are no 10 GbE switches available.

Since Ceph replicates syncrounously, a write command will only be confirmed once at least one copy has been written on the other side. Small bandwidth and high network latency makes it so that you will wait for a long time for your iops to be completed.

You could try using ZFS instead with (asynchronous) replication, but I dont have much experience with that. Also iirc that still requires the hosts to be in the same cluster and Proxmox also requires low latencies for the corosync cluster network.

Generally, making a stretched cluster is not feasible with Proxmox right now.

 

[ph0x]

A cluster over WAN is a bad idea in general.