Feature request: VM replication over Syncthing

jazzl0ver

Renowned Member
Mar 6, 2013
69
1
73
Hi,

It would be awesome to have Syncthing integrated into Proxmox VE as a VM replication backend. Currently, replication works only for ZFS file systems, while Syncthing will work on any mountable fs. A lot of other benefits exist: only changed blocks are transferred, unlimited replicas, rate limiting, API that simplifies integration.

Thanks in advance!
 
I don't think SyncThing is meant for synchronizing raw image files like that. For replication, a mechanism which does on-demand, delta-based synchronization is needed.
 
Hi, fabian. This is actually what Syncthing does: it sends only deltas and runs on demand (default is hourly, but can easily be changed in the config) or when a file (dis)appears.
 
Hi, fabian. This is actually what Syncthing does: it sends only deltas and runs on demand (default is hourly, but can easily be changed in the config) or when a file (dis)appears.

I know what syncthing does - I just really doubt that it works for this use case. ZFS works so well because it can take snapshots almost for free (it's a Copy-On-Write file system, with snapshotting as core feature integrated through the whole architecture), and generate a delta stream from that just by reading what has changed since then. syncthing would need to re-scan the whole image to find out which parts/blocks have changed, which is a very expensive operation (images can easily be hundreds of GB, with multiple images per guest, with replication up to every minute!).

you can easily try it yourself - setup syncthing to replicate a directory storage with non-trivial amount of data stored (let's say a few hundred GB), setup replication every 15 minutes, and have some I/O load inside the guests using the replicated image files while running your experiment. the resulting replicas won't be consistent, and you won't have a backup of your config (the latter is easy to solve, the former not so much) - so please don't use this for anything real ;) you'll see the difference to 'zfs send/recv'. there is more stuff (like SyncThing using temporary copies thus needed twice the space, I am not sure whether SyncThing even supports syncing constantly changing files, ...)
 
Syncthing is really cool, but i don't think this usecase would be currently supported by syncthing.
These are my concerns:

1.) File permissions, extended attributes and other advanced metadata might not fully sync
2.) Syncthing can only write files under single user/owner.
3.) Syncing database storages, sqlite files and similar sounds like recipe for disaster unless one node is fully synced before second starts writing to it.
4.) It fails to sync when file is changed during sync. It will try to resync hour later in such case. But if file changes constantly, it will probably never reach sync.

I think syncthing might get improved to fix these issues in the future, but it will likely be a slow progress.
I like syncthing and i would really like to see it being able to perform full system sync. (or at least be able to sync whole home directory safely)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!