rbd-mirror

[koniambo]

Hi,

I have 2 clusters ceph one is hyper converged proxmox and the other one is cephadm managed. I want to mirrored them to test if it's possible.
But by following the proxmox documentation about rbd-mirroring, rbd mirror wont start am I missing something ?

Setting up rbd-mirror (19.2.3-pve2) ...
Created symlink '/etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target' → '/usr/lib/systemd/system/ceph-rbd-mirror.target'.
Created symlink '/etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target' → '/usr/lib/systemd/system/ceph-rbd-mirror.target'.
Processing triggers for man-db (2.13.1-1) ...
root@hp365-25-1:~# systemctl enable ceph-rbd-mirror.target
root@hp365-25-1:~# cp /usr/lib/systemd/system/ceph-rbd-mirror@.service /etc/systemd/system/ceph-rbd-mirror@.service
root@hp365-25-1:~# sed -i -e 's/setuser ceph.*/setuser root --setgroup root/' /etc/systemd/system/ceph-rbd-mirror@.service
root@hp365-25-1:~# systemctl enable --now ceph-rbd-mirror@rbd-mirror.$(hostname).service
Created symlink '/etc/systemd/system/ceph-rbd-mirror.target.wants/ceph-rbd-mirror@rbd-mirror.hp365-25-1.intra.inist.fr.service' → '/etc/systemd/system/ceph-rbd-mirror@.service'.
Job for ceph-rbd-mirror@rbd-mirror.hp365-25-1.intra.inist.fr.service failed because the control process exited with error code.
See "systemctl status ceph-rbd-mirror@rbd-mirror.hp365-25-1.intra.inist.fr.service" and "journalctl -xeu ceph-rbd-mirror@rbd-mirror.hp365-25-1.intra.inist.fr.service" for details.
root@hp365-25-1:~# systemctl status ceph-rbd-mirror@rbd-mirror.hp365-25-1.intra.inist.fr.service
× ceph-rbd-mirror@rbd-mirror.hp365-25-1.intra.inist.fr.service - Ceph rbd mirror daemon
Loaded: loaded (/etc/systemd/system/ceph-rbd-mirror@.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Wed 2025-11-05 17:33:24 CET; 5min ago
Duration: 77ms
Invocation: 753a5e4d33624cf7808423e52bd19d50
Main PID: 4032250 (code=exited, status=1/FAILURE)

nov. 05 17:33:24 hp365-25-1.intra.inist.fr systemd[1]: ceph-rbd-mirror@rbd-mirror.hp365-25-1.intra.inist.fr.service: Scheduled restart job, restart counter is at 3.
nov. 05 17:33:24 hp365-25-1.intra.inist.fr systemd[1]: ceph-rbd-mirror@rbd-mirror.hp365-25-1.intra.inist.fr.service: Start request repeated too quickly.
nov. 05 17:33:24 hp365-25-1.intra.inist.fr systemd[1]: ceph-rbd-mirror@rbd-mirror.hp365-25-1.intra.inist.fr.service: Failed with result 'exit-code'.
nov. 05 17:33:24 hp365-25-1.intra.inist.fr systemd[1]: Failed to start ceph-rbd-mirror@rbd-mirror.hp365-25-1.intra.inist.fr.service - Ceph rbd mirror daemon.
nov. 05 17:35:52 hp365-25-1.intra.inist.fr systemd[1]: ceph-rbd-mirror@rbd-mirror.hp365-25-1.intra.inist.fr.service: Start request repeated too quickly.
nov. 05 17:35:52 hp365-25-1.intra.inist.fr systemd[1]: ceph-rbd-mirror@rbd-mirror.hp365-25-1.intra.inist.fr.service: Failed with result 'exit-code'.
nov. 05 17:35:52 hp365-25-1.intra.inist.fr systemd[1]: Failed to start ceph-rbd-mirror@rbd-mirror.hp365-25-1.intra.inist.fr.service - Ceph rbd mirror daemon.
nov. 05 17:38:06 hp365-25-1.intra.inist.fr systemd[1]: ceph-rbd-mirror@rbd-mirror.hp365-25-1.intra.inist.fr.service: Start request repeated too quickly.
nov. 05 17:38:06 hp365-25-1.intra.inist.fr systemd[1]: ceph-rbd-mirror@rbd-mirror.hp365-25-1.intra.inist.fr.service: Failed with result 'exit-code'.
nov. 05 17:38:06 hp365-25-1.intra.inist.fr systemd[1]: Failed to start ceph-rbd-mirror@rbd-mirror.hp365-25-1.intra.inist.fr.service - Ceph rbd mirror daemon.

Thanks for your help !

 

[Sviat]

Hi,

It looks like missed the part for site A

ceph auth get-or-create client.rbd-mirror-peer-a mon 'profile rbd' osd 'profile rbd' -o /etc/pve/priv/site-b.client.rbd-mirror-peer-a.keyring

But better to check it step by step https://pve.proxmox.com/wiki/Ceph_RBD_Mirroring .
Note that the steps order is makes matter.
Make sense to revert everything back and start from beginning.
Do not expect that it will works perfect from the box, mirroring is too sensitive and require fast and stable network.

 

[koniambo]

Hi,

Thanks for your answer, I forget to copy it but I bootstrapped the mirror by using thus command :

rbd mirror pool peer bootstrap create --site-name <local_site_name> <pool_mirrored> > boostrap.txt
rbd mirror pool peer bootstrap create --site-name cluster_proxmox ceph_proxmox > boostrap.txt

As I understand this command, it create the cephx client automatically ?