Feature request: VM replication over Syncthing

[jazzl0ver]

Hi,

It would be awesome to have Syncthing integrated into Proxmox VE as a VM replication backend. Currently, replication works only for ZFS file systems, while Syncthing will work on any mountable fs. A lot of other benefits exist: only changed blocks are transferred, unlimited replicas, rate limiting, API that simplifies integration.

Thanks in advance!

 

[fabian]

I don't think SyncThing is meant for synchronizing raw image files like that. For replication, a mechanism which does on-demand, delta-based synchronization is needed.

 

[jazzl0ver]

Hi, fabian. This is actually what Syncthing does: it sends only deltas and runs on demand (default is hourly, but can easily be changed in the config) or when a file (dis)appears.

 

[fabian]

Hi, fabian. This is actually what Syncthing does: it sends only deltas and runs on demand (default is hourly, but can easily be changed in the config) or when a file (dis)appears.

I know what syncthing does - I just really doubt that it works for this use case. ZFS works so well because it can take snapshots almost for free (it's a Copy-On-Write file system, with snapshotting as core feature integrated through the whole architecture), and generate a delta stream from that just by reading what has changed since then. syncthing would need to re-scan the whole image to find out which parts/blocks have changed, which is a very expensive operation (images can easily be hundreds of GB, with multiple images per guest, with replication up to every minute!).

you can easily try it yourself - setup syncthing to replicate a directory storage with non-trivial amount of data stored (let's say a few hundred GB), setup replication every 15 minutes, and have some I/O load inside the guests using the replicated image files while running your experiment. the resulting replicas won't be consistent, and you won't have a backup of your config (the latter is easy to solve, the former not so much) - so please don't use this for anything real you'll see the difference to 'zfs send/recv'. there is more stuff (like SyncThing using temporary copies thus needed twice the space, I am not sure whether SyncThing even supports syncing constantly changing files, ...)

 

[jazzl0ver]

Thanks for the detailed explanation! You're right.

 

[harvie]

Syncthing is really cool, but i don't think this usecase would be currently supported by syncthing.
These are my concerns:

1.) File permissions, extended attributes and other advanced metadata might not fully sync
2.) Syncthing can only write files under single user/owner.
3.) Syncing database storages, sqlite files and similar sounds like recipe for disaster unless one node is fully synced before second starts writing to it.
4.) It fails to sync when file is changed during sync. It will try to resync hour later in such case. But if file changes constantly, it will probably never reach sync.

I think syncthing might get improved to fix these issues in the future, but it will likely be a slow progress.
I like syncthing and i would really like to see it being able to perform full system sync. (or at least be able to sync whole home directory safely)

 

[rhY]

I was hoping to just syncthing the vzdump backups I already make locally. That would be amazing for offsite backup. Is there a better option for what I'm talking about without having to create a VPN, static IP bridge, etc..?

 

[Dark26]

i think Promox backup server is what you need.

 

[rhY]

Experimenting with it now. Still not sure how to get the backups offsite. But I'm just now playing around. Thanks.

 

[Dunuin]

Experimenting with it now. Still not sure how to get the backups offsite. But I'm just now playing around. Thanks.

With PBS there are sync jobs built-in so an offsite PBS could pull your backups from the onsite PBS.

And I don't see why syncthing shouldn't be able to sync your VZDump backups. Just keep in mind that VZDump really wastes space as it can't do incremental backups. You won't have ransomware protection and you have to trust the hoster of your offsite backups.

So yes, highly recommended to have a look at PBS which does everything way better (but with higher hardware requirements).

 

[golf800]

Just to Share:
Recently implemented:
(1) offsite VM running Debian with (a) WG Docker "Server" and (b) nfs-kernel-server
(2) PBS onsite with WG Client tunneled to offiste VM and nfs-common to attach to offsite VM
(3) PBS Daily backup (deduplicate) to offsite nfs-kernel-server
(4) running a week now without issues. 1st backup took a while but subsequent backups ~15% of time of 1st backup.
Hope these can help some members