[SOLVED] rbd mirror - cannot setup peer on site-a

[seanetjersey]

Hello,

I would be very grateful for some help, I have been trying to setup an rbd mirror on a pool between two proxmox ceph clusters.
I have followed the instructions here : https://pve.proxmox.com/wiki/Ceph_RBD_Mirroring
And also here : https://satishdotpatel.github.io/ceph-rbd-mirroring/

Where I think I am :
- pools on site-a and site-b setup with the same name
- mirroring enabled on the pools
- client auth setup and keyring copied to site-b
- site-a conf file copied to site-b
- installed the rbd-mirror daemon in site-b, and configured the /etc/systemd/system/ceph-rbd-mirror@.service ExecStart line to use the client username and keyring from site-a.
- daemon starts without error (so far as I can tell)

root@pve11:/etc/systemd/system# systemctl status ceph-rbd-mirror@rbd-mirror.pve11.service
● ceph-rbd-mirror@rbd-mirror.pve11.service - Ceph rbd mirror daemon
     Loaded: loaded (/etc/systemd/system/ceph-rbd-mirror@.service; enabled; preset: enabled)
     Active: active (running) since Mon 2024-07-01 11:05:15 BST; 6min ago
   Main PID: 1000105 (rbd-mirror)
      Tasks: 16
     Memory: 13.8M
        CPU: 148ms
     CGroup: /system.slice/system-ceph\x2drbd\x2dmirror.slice/ceph-rbd-mirror@rbd-mirror.pve11.service
             └─1000105 /usr/bin/rbd-mirror -f --cluster site-a --id rbd-mirror-peer-a --keyring=/etc/pve/priv/site-a.client.rbd-mirror-peer-a.keyring --setuser root --setgroup root

Jul 01 11:05:15 pve11 systemd[1]: Started ceph-rbd-mirror@rbd-mirror.pve11.service - Ceph rbd mirror daemon.
root@pve11:/etc/systemd/system#

- from site-b, the peer is setup :

root@pve11:/var/log/ceph# rbd mirror pool info cephpool01
Mode: image
Site Name: site-b

Peer Sites:

UUID: b271253c-ec99-4a4b-907c-94da92ef2851
Name: site-a
Direction: rx-only
Client: client.rbd-mirror-peer
root@pve11:/var/log/ceph#

- but on site-a, no peer is setup :

root@pve01:/etc/ceph# rbd mirror pool info cephpool01
Mode: image
Site Name: site-a

Peer Sites: none
root@pve01:/etc/ceph#

Any help gratefully received - I am completely stuck.

Thank you in advance

 

[seanetjersey]

Hi,
just to update, this is solved.
The destination cluster had CEPH errors which were unresolved.
(In my case the CEPH errors related to the fact it is a test/lab environment and with a single host, and CEPH wasnt able to honour the replication rule requiring data be spread across different hosts. I edited the crush map, and recreated the replication pool for good measure, restarted the services and then it worked.)

 

[cardax]

Hello @seanetjersey I'm following the guide:
https://pve.proxmox.com/wiki/Ceph_RBD_Mirroring
but I've a situationo similar to you about site-a does't show peer when i ask info.

root@node01:/var/log/ceph# rbd mirror pool info pool01
Mode: image
Site Name: 16c6e89a-34cc-4672-86a1-b265f0f897dd

Peer Sites: none

did you follow the same guide? I'm not really shure how site-a is able to know site-b

 

[Jordan.zhang]

Hi seanetjersey
I am trying to test RBD mirroring between two PVE clusters, but I don’t know how to import the virtual machine disk. The system always prompts 'vm-100-disk-0: non-existent or non-regular file.' Can you provide the complete disk import operation?"