Multi site Ceph

[a.snorins]

Hello.

We have 2 proxmox clusters in production.
They contain Ceph cluster with cephfs is on it.

The task is to sync files between sites. Site A is master and site B is for backup.
On site A cephfs files are created and deleted after x months.
Site B cephfs must contain the same data what is on site A. (sync or async)

cephfs-mirror will be in next ceph release so we must search another way.
For now we want to create multi-site Ceph for backup / disaster recovery.


Before doing this in production, we have created 2 clusters in the lab.
Followed by instruction in https://docs.ceph.com/en/latest/radosgw/multisite/ we errors when starting a process "systemctl start ceph-radosgw@rgw.`hostname -s`".

Mar  4 14:10:17 lab1 radosgw[40619]: 2021-03-04T14:10:17.876+0200 7fc70306b700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
Mar  4 14:10:17 lab1 radosgw[40619]: failed to fetch mon config (--no-mon-config to skip)

Maybe you have some ideas how make this task up and running!?


Proxmox version 6.3.1
Veph version 15.2.8

P.S. rsync is last option for this task.

 

[Alwin Antreich]

They contain Ceph cluster with cephfs is on it.

What do you intent to store on the CephFS?

Followed by instruction in https://docs.ceph.com/en/latest/radosgw/multisite/ we errors when starting a process "systemctl start ceph-radosgw@rgw.`hostname -s`".

That is to provide a S3 compatible storage but not for CephFS.

 

[kyriazis]

What do you intent to store on the CephFS?


That is to provide a S3 compatible storage but not for CephFS.

I have a similar scenario to the original poster.

I am planning a geo-backup scenario of a ceph cluster. For the time being, we only want to replicate / backup the CephFS portion for disaster recovery.

I am not sure why you are asking what is the intended CephFS storage. If the main purpose is backups, why does it matter?

Thinking ahead into I recall reading somewhere that there are plans for multi-site CephFS. Does anybody know what is the timeframe for this? Will I be able to convert 2 existing clusters (where cluster A rsync's to cluster B) into a multi-site setup where clients on both sites can perform reads & writes and they are replicated into the other cluster?

Thanks!

 

[Alwin Antreich]

I am planning a geo-backup scenario of a ceph cluster. For the time being, we only want to replicate / backup the CephFS portion for disaster recovery.

CephFS mirroring has been recently introduced in Ceph Pacific. You may need to wait a bit till it's on PVE available.

I am not sure why you are asking what is the intended CephFS storage. If the main purpose is backups, why does it matter?

To find out the intent, as it might be that a different solution fits better. And it doesn't hurt to ask.

 

[kyriazis]

CephFS mirroring has been recently introduced in Ceph Pacific. You may need to wait a bit till it's on PVE available.


To find out the intent, as it might be that a different solution fits better. And it doesn't hurt to ask.

I am fine doing rsync's at the beginning, and then switching to CephFS mirroring.

The initial use of the 2nd cluster would be for backup, however thinking that in the future multi-site cephFS would be the ideal goal (being able to write from both/all sites). Any particular way that the clusters should be set up to make this easier later, when multi-site cephFS would be available? Will cephFS multi-site assume ceph multi-site? Meaning, if cephFS multi-site is set up, will I be getting rdb and/or rados multi-site "for free"?

The use case is storage/archival of large datasets (gigabyte to terabyte per file), per-user storage area (not home directories) and executables used to support a CI infrastructure. Mostly read-only, but expect it to expand as storage of CI results. All intranet. It would be cumbersome to expect machines/hosts that mount cephFS to have to do 2 mounts, one (local) for read-only and one (remote) for writing.

Thank you!

 

[Alwin Antreich]

The initial use of the 2nd cluster would be for backup, however thinking that in the future multi-site cephFS would be the ideal goal (being able to write from both/all sites). Any particular way that the clusters should be set up to make this easier later, when multi-site cephFS would be available?

Probably you may need to re-create the CephFS setup on the backup site.

Will cephFS multi-site assume ceph multi-site? Meaning, if cephFS multi-site is set up, will I be getting rdb and/or rados multi-site "for free"?

No. You will need to setup every service separately.

The use case is storage/archival of large datasets (gigabyte to terabyte per file), per-user storage area (not home directories) and executables used to support a CI infrastructure. Mostly read-only, but expect it to expand as storage of CI results. All intranet. It would be cumbersome to expect machines/hosts that mount cephFS to have to do 2 mounts, one (local) for read-only and one (remote) for writing.

Depends on your CI/CD engine. We use Gitlab with S3 storage and there you will have RGW multi-site capability.