[TUTORIAL] What Replaces SRM and vSphere Replication on Proxmox?

bbgeek17

Distinguished Member
Nov 20, 2020
6,941
2,920
278
Blockbridge
www.blockbridge.com
Hi Everyone,

Disaster recovery keeps coming up with folks moving off VMware, usually as some version of "what replaces SRM?" We've answered pieces of this in threads for years, so we finally sat down and wrote the whole thing up properly.

Mapping VMware DR Features to Proxmox VE: SRM, vSphere Replication, and Site Recovery https://kb.blockbridge.com/technote/proxmox-vmware-disaster-recovery/

The technote maps each VMware DR feature to its nearest native PVE equivalent: pvesr and pve-zsync, the PBS two-site pattern, the HA manager, PDM, stretched clusters with Ceph stretch mode, and array-based replication. It also walks through what doing SRM's job by hand involves: configurations, identity, addressing, start order, and testing. There's a feature mapping table at the end if you want the one-screen version. We stuck to native tooling; third-party tools exist but are out of scope here.

Everything was checked against current docs (PVE 9.2 / PBS 4.2 / PDM 1.1) or confirmed in our lab, and where we couldn't verify something, we say so. One thing we'll say up front: we have never run Ceph RBD mirroring failover ourselves, and public operational reports are scarce. If you've run it through a real or rehearsed site failure, we'd love to learn more and update the doc.

If you're doing DR on PVE in a way we didn't cover, or you think we got something wrong, post it here. We'll update the article.

As always, let me know if you have questions, corrections, or if we've missed anything!

Enjoy, The Blockbridge Team!


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Hi,

Thanks for your link.

We have run Ceph RBD mirroring (by pool, by image, snapshot-based and journal-based, one-way, two-ways). Not in production environment. Working as expected. Requires some care during implementation...

It needs some user friendly integration in WebUI, otherwise only advanced users can deploy.

But it is an essential feature - the multi-cluster equivalent of intra-cluster pvesr.

No doubt Proxmox Team is working on this.

Christophe.
 
Thanks, @christophe! That's more configuration coverage than most reports we've found. "The multi-cluster equivalent of intra-cluster pvesr" is a good way to put it.

Since you've run both modes in both directions, a few things you're in a rare position to answer:

1. What did "some care during implementation" involve? For example, journal mode needs the journaling feature enabled per image, which requires librbd and doesn't mix with the krbd storage option. Was it that kind of thing, or something else?

2. What did you observe on the primary side while a peer was unreachable? Journal growth in journal mode, or mirror-snapshot buildup in snapshot mode? Space behavior during an outage is where we have the least data.

3. Did you ever end up in split-brain after a forced promote? The docs say resync deletes the image and copies from scratch. Curious how long that took at your sizes, or whether you found a better path.


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Hi @bbgeek17,

In our context, we are preparing configurations intended to be instantiated by non-specialist departments; we are therefore prioritizing ease of implementation.

1. At the very beginning we followed multiple docs, mainly https://pve.proxmox.com/wiki/Ceph_RBD_Mirroring,https://docs.ceph.com/en/reef/rbd/rbd-mirroring/ and https://docs.ceph.com/en/latest/rados/operations/user-management/#user-management. A mix not so happy, an ill-advised combination led to an implementation that didn't work! A second try followed exclusively and strictly PVE doc. Journal mode is very interesting, particularly at the pool level, as it significantly simplifies management for operations teams : move a virtual machine disk to this pool, and it is automatically registered for replication! However, it comes at a cost: it doubles the number of write operations... In fact, we primarily conducted investigations using snapshot mode. krbd were not used in our lab tests but shouldn't be problematic with recent kernels / Ceph versions?

2. Good point! The tests conducted so far assume two healthy clusters to start with. The issue of a failing backup system will need to be added to our list!

3. Yes, every time we bring the disks back to the first Ceph cluster : split-brain! Because in the Cluster 1 configuration, the image was set to "primary"... So, we need to do the following on cluster 1 (primary / main) :


Code:
rbd mirror image demote ceph-disks/vm-101-disk-1
Code:
rbd mirror image resync ceph-disks/vm-101-disk-1
Showing :
Code:
rbd mirror pool status ceph-disks --verbose
health: WARNING
daemon health: OK
image health: WARNING
images: 1 total
    1 unknown

DAEMONS
service 7614135:
  instance_id: 7614216
  client_id: cluster1-test
  hostname: vpx1-ceph
  version: 19.2.3
  leader: true
  health: OK

IMAGES
vm-101-disk-1:
  global_id:   43c71802-a1a2-4c4f-b48a-fc33882aa4d2
  state:       up+error
  description: split-brain
  service:     cluster1-test on vpx1-ceph
  last_update: 2026-05-28 16:50:49

Then after resync completed :

Code:
rbd mirror pool status ceph-disks --verbose
health: WARNING
daemon health: OK
image health: WARNING
images: 1 total
    1 unknown

DAEMONS
service 7614135:
  instance_id: 7614216
  client_id: cluster1-test
  hostname: vpx1-ceph
  version: 19.2.3
  leader: true
  health: OK

IMAGES
vm-101-disk-1:
  global_id:   43c71802-a1a2-4c4f-b48a-fc33882aa4d2
  state:       up+unknown
  description: remote image is not primary
  service:     cluster1-test on vpx1-ceph
  last_update: 2026-05-28 16:52:18

Followed by
Code:
rbd mirror image promote ceph-disks/vm-101-disk-1
Wait fot Ceph status :
Code:
...
IMAGES
vm-101-disk-1:
  global_id:   43c71802-a1a2-4c4f-b48a-fc33882aa4d2
  state:       up+stopped
  description: local image is primary
  service:     cluster1-test on vpx1-ceph
  last_update: 2026-05-28 16:52:48

Then, finally, on the second cluster :
Code:
 rbd mirror image demote ceph-disks/vm-101-disk-1

I cannot say for sure whether the image was destroyed before the resynchronization; the operation took two minutes for a 32GB image. That is another good question to consider before moving to production!

All in all, the system isn't exactly user-friendly and, in its current state, cannot be offered to non-expert teams; however, all the individual components are there and work quite well.

Christophe.
 
Last edited:
Thanks, this is great detail. The command transcript especially.

With krbd the snapshot mode should be fine since it doesn't need the journaling feature on the image. Journal mode is a different story though. There was a full patch series for kernel rbd journaling back around the 5.3 Kernel days (https://patchwork.kernel.org/cover/11063519/) but it never merged, so journal mode still needs librbd even on current kernels.

On resync the image does, indeed, get destroyed prior to the resync. This is mentioned by RBD developers in their DR deck: resync deletes the demoted image and copies everything back from the primary, from scratch (https://ceph.io/assets/pdfs/events/...ed mirroring robust for disaster recovery.pdf). Your two minutes for 32GB fits that, though the exact rate depends on how full the image actually was. Do you happen to know? If it was mostly full, that's 270 MB/s, which would put the bottleneck in the resync itself rather than the network. Either way it's a complete re-copy, so what matters for production is the allocated data per image over the site link. A mostly-full 10TB image over a 1Gbit link works out to more than a day of copying, per image, and you're unprotected until it finishes.

The critical thing about your observation is that this is the routine path for disaster recovery, not particularly specific to your infrastructure. An unplanned failover leaves the old primary still believing it owns the image, so the return trip goes through split-brain and a from-scratch resync every time. That's a full copy of every image over the site link just to return to normal.

One more thing from that same deck worth noting for your use case: image group mirroring is still on their ongoing list, so each disk of a multi-disk VM syncs on its own schedule and arrives at the other site at a slightly different point in time. For a database VM with data and log on separate disks, the DR copy isn't one consistent moment.

Interesting point about pool-level journal mode. Auto-enrollment would make life easier for an ops team, but paying double writes for it is steep.

Your summary matches our impression from the outside: the pieces are all there, but you really need to know what you're doing.

Cheers!


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: Johannes S
This post has been very informative, thank you for sharing.

I am still contemplating what is better way, to follow Ceph rbd mirroring or going PBS restore way for the DR. As Ceph rbd mirroring still lacks feature to clone from rbd mirror secondary image[0] to RW rbd image, I am leaning more into PBS.

My main concern regarding PBS is how fast can about 50TB environment can be restored and put into operation. Regarding Ceph my main concern is the complexity of rbd-mirroring and (not low) probability of destroying a production environment if an admin does not practice restores frequently.

I am considering building my own automation tools to restore backups from PBS in more organized way. I have very good experience with building Proxmox automations together with AI.

Any thoughts are welcome.

[0] - https://tracker.ceph.com/issues/71739
 
My main concern regarding PBS is how fast can about 50TB environment can be restored and put into operation.
From scratch? the same as any other restoration RTO calculation- the longest of
(- time to read from source
- time to transfer
- time to write to destination) * processing coefficient (1.2 is safe with typical current hardware.)

HOWEVER, PBS has live restore which means you can be operational in the interim.
 
Thanks for the tracker link @robertlukan . It also affects testing - cloning from the secondary is a common way to practice a recovery without touching replication.

For the 50TB question, @alexskysilk'` math is the way to figure it. Two things to add:

Live restore : It starts the VM before its data is local. The alloc-track driver and the stream job read not-yet-restored data , verify, and decompresse at least the 4MB chunk containing it from PBS, and the same read writes that data onto the local disk, so each region pays the fault-in cost once. First touch of cold data adds around 5 to 10 milliseconds on a fast local network ( HDD seek territory even on NVMe ) while the background job restores everything else sequentially. So the VM boots up relatively quickly and warms as it runs. But, plan for reduced IO on cold data until the restore completes.

Second, with many guests, restore order matters more than any single restore time. Restores run per guest, so your real RTO comes down to which VMs you bring up first and how many restores you can run at once before the PBS datastore can't read any faster. A few timed test restores, including some in parallel, will give you those numbers. They will also verify your backups.

You are on point about the frequent restore exercising, no matter which route you take! The recovery that works is the one that got rehearsed.


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: Johannes S
Here's a slight refinement to my live restore description above, after more time in the source. There are three different sizes at work. I had them blurred together.

The guest read is whatever size the guest asks for. The fetch from PBS is always the whole 4MB chunk containing it. The chunk is verified and decompressed by the PBS client code that runs inside the QEMU process on the restoring host. The write to the local disk covers just the region the guest read, rounded up to the devices native write/allocation granularity (depends on the specifics of the underlying block device). It is not the whole chunk. The rest of the chunk stays in a 256MB memory cache inside that same QEMU process. Reads of nearby data may find it there instead of going back to the server. The background restore job uses the same cache.

It's important to keep the cache size in perspective. 256MB is relatively small next to a multi-TB image. Its effectiveness depends on the size of the disks and the randomness of the workload. Clustered reads benefit (i.e., spatial locality). Scattered random reads across a large cold disk mostly do not.

So "each region pays the cost once" is not a guarantee. A later read from a different part of the same chunk may have to fetch it again if that chunk is no longer in the cache. The 5 to 10 millisecond first read of cold data still stands.

We're updating the technote to match, along with the RBD findings from earlier in the thread.


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: Johannes S
Thanks for the write up, and to @christophe for the RBD transcript. Very useful.


Disclosure: we build KVMDR, a third party DR tool for Proxmox and oVirt, so I know you kept third party tools out of scope. A few of the gaps raised here are the ones we built around, so I’ll share how we handle them in case it helps anyone comparing options.


Testing without touching replication: our Simulation runs the DR copy on an isolated network while protection keeps running, so you can rehearse as often as you like without demoting or cloning anything.


Consistency across disks: disks of the same VM, and linked VMs in a consistency group, recover to the same point in time.


Failback: after failover we reverse replicate only the changes back to the source. No split brain, no full resync, so a 10TB VM doesn’t mean a day of copying to get home.


Addressing: automatic Re IP on failover.


Happy to answer questions here, and glad to be corrected if we’ve misread any of the native tooling.

Ravi
 
Maybe I can also share my progress on this topic. I decided to follow the Proxmox Backup Server (PBS) path due to its simplicity and the assurance that production storage cannot be impacted.

As I mentioned in my post from August, I built my own automation tool for this: https://github.com/RobertLukan/restore-engine. It is a bit more than just simple automation, so if anyone is interested, you can read more about it on GitHub.

My main concern with using PBS was performance, and I have to say it really isn't an issue if you have good hardware and a solid configuration. As mentioned before, I need to restore about 50 TB of data, with roughly 25 TB of that being SQL data. The main bottleneck is restoring a single 8 TB SQL server, which takes about 3 hours. I am still using default settings on PBS and PVE, so there is room to tune that further. Restoring the complete set of ~130 VMs takes about 5 hours, primarily due to that single large server. (I haven't measured restoring all VMs simultaneously, as our restore workflow is a bit more complex.)

The restores run in parallel, though I limited my automation to 20 concurrent restores. At that threshold, I noticed PVE (qemu) processes competing for storage locks, resulting in timeouts while trying to acquire locks to create Ceph RBD images. Switching to kRBD might be better here, as I've noticed before that librbd has certain performance limitations.

Overall, restore speeds are fantastic—I am hitting 10 GB/s total write speeds to the Ceph cluster. Here is the hardware backing this:
  • Ceph Cluster: 5 nodes with 40 NVMe drives total.
  • PBS Server: 22 Ă— 15 TB NVMe drives in RAIDZ2, paired with a dedicated Intel Optane L2ARC cache.
Frankly, the current setup is bottlenecked by the restore speed of the largest VM image, which comes down to a PBS limitation. It could probably be tuned further, though I don't really need to right now.

I am still not convinced that rbd-mirror is the right way to go(yet). Another issue with storage-level replication is that Windows (qemu) isn't aware of storage snapshotting, which can lead to disk corruption that an operator has to clean up later—definitely not something I want to deal with during a Disaster Recovery (DR) activation. While you can script around this using PVE to ask QEMU to invoke VSS and take the snapshot between freeze and thaw, it adds complexity.

IMHO, there are still two features missing from rbd-mirror:
  1. Support for cloning from secondary images (tracked here: https://tracker.ceph.com/issues/71739).
  2. Finalized and tested consistency group mirroring(some PRs were recently merged into main)
The second point is arguably even more important, as you never want individual disks of the same VM snapshotted at different timestamps.
 
  • Like
Reactions: Johannes S
Great write up @robertlukan, and impressive numbers. 10 GB/s into Ceph is serious hardware.

Agree on consistency groups being the bigger gap. Individual disks landing at different timestamps is worse than a slightly older but consistent copy, especially for SQL with data and logs on separate disks.

Curious about two things:
  1. What’s your backup frequency, i.e. your effective RPO for the SQL servers?
  2. Have you tried live restore for the 8TB SQL VM? It could bring it up long before the 3 hours finish, though cold reads would be slow at first.
 
I had to "redesign" this hardware around 2 x 40Gbps and 4 x 10Gbps to get the best recovery performance. For "fun," I am running Ceph EC 2+2 with FastEC enabled. It is fine for DR, though I wouldn't use it for production just yet as FastEC still has a few critical bugs. That said, the performance with FastEC is very good!

Backups are made once a day, as there is no business reason to run them more often. The SQL team has its own transaction log backups, which are SQL-native backups. So, if we are not happy with the daily backup, we can still restore to a specific point in time. This works fine for us at this stage.

Yes, I have tried restoring the 8TB SQL VM, though performance isn't great when other restores are happening at the same time. I was thinking about putting the backups for that SQL server on a different ZFS vdev, though I'm not sure if that would help much.

I have not tried these settings yet: PBS_RESTORE_MAX_THREADS=8 and PBS_RESTORE_FETCH_CONCURRENCY=32. Also, there are new improvements coming in pve-qemu-kvm(11.1.3-2) regarding restore speeds, which will be worth testing once it arrives. Overall, I still have some room for improvement and testing in the upcoming weeks.
 
  • Like
Reactions: Johannes S
I'd treat cyber recovery as a separate requirement from the SRM-style failover path. Replication can give you a recent copy, but after a long attacker dwell time that copy can include the compromise. Keep older protected points alongside the fast replica and test two or three in isolation before promoting one. The runbook should identify the recommended recovery point for each workload, not just the failover target.

Disclosure: I work at CyberSnap.
 
My main concern with using PBS was performance,
Since PBS works at the QEMU level, there is inherent limitations on the type of performance you can get. If your workloads dont fit in your backup window you may want to look at rbd based workflows. Back in antedeluvian times (before PBS) I used this to reasonable effect: https://github.com/Corsinvest/cv4pve-barc

Guest quiescense could still be an issue for orchestration but otherwise its probably the fastest mode possible. Its also much less likely to impact foreground performance the PBS (no fleecing.)
 
alexskysilk's RBD route is worth testing for that 8 TB SQL VM, but I'd keep the SQL log chain as the consistency mechanism: restore a crash-consistent RBD snapshot or replica as the base, start SQL, then apply native log backups. That avoids the PBS/QEMU restore bottleneck without pretending RBD alone gives application consistency. Before adding a separate ZFS vdev, check whether the 3 hours is spent reading PBS, writing Ceph RBD images, or waiting on locks; only the first is likely to benefit from datastore layout.