PVE replication and ZFS Snapshot

[jacklayne]

Hello everyone,

I'm trying to figure out how to replicate the VM between two nodes and get a snapshot as backup.

Example:

• Node-1 and Node-2
• VM-100 running on Node-1
• VM-100 replicated on Node-2

If I get a snapshot on Node-1 this will be replicated on Node-2

NODE-1

NAME                                                 AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD     REFER  WRITTEN
vmdata/vm-100-disk-0                                  475G  5.31G     1.18M   5.31G             0B         0B     5.31G     268K
vmdata/vm-100-disk-0@today                               -   932K         -       -              -          -     5.31G    5.31G
vmdata/vm-100-disk-0@__replicate_100-0_1578244680__      -   272K         -       -              -          -     5.31G     928K

NODE-2

NAME                                                 AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD     REFER  WRITTEN
vmdata/vm-100-disk-0                                  471G  5.31G      924K   5.31G             0B         0B     5.31G        0
vmdata/vm-100-disk-0@today                               -   924K         -       -              -          -     5.31G    5.31G
vmdata/vm-100-disk-0@__replicate_100-0_1578244680__      -     0B         -       -              -          -     5.31G     920K

The two nodes are kept in sync.

But whether I get a snapshot on Node-2 this will be removed during the next sync.

I tried to "hold" the snapshot, but then it seems the replication get broken and the snapshot get renamed:

Before the replication ( NODE-2 Only )

NAME                                                 AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD     REFER  WRITTEN
vmdata/vm-100-disk-0                                  471G  5.31G     1.15M   5.31G             0B         0B     5.31G        0
vmdata/vm-100-disk-0@today                               -  1.15M         -       -              -          -     5.31G    5.31G
vmdata/vm-100-disk-0@__replicate_100-0_1578245040__      -     0B         -       -              -          -     5.31G    1.10M
vmdata/vm-100-disk-0@snap-node-2                         -     0B         -       -              -          -     5.31G        0

After the replication ( Node-2 Only )

NAME                                                 AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD     REFER  WRITTEN
vmdata/vm-100-disk-0                                  471G  5.31G     1.15M   5.31G             0B         0B     5.31G        0
vmdata/vm-100-disk-0@today                               -  1.15M         -       -              -          -     5.31G    5.31G
vmdata/vm-100-disk-0@__replicate_100-0_1578245040__      -     0B         -       -              -          -     5.31G    1.10M
vmdata/vm-100-disk-0@recv-99303-1                        -     0B         -       -              -          -     5.31G        0

PVE replication status

JobID      Enabled    Target                           LastSync             NextSync   Duration  FailCount State
100-0      Yes        local/lab-proxmox-01  2020-01-05_18:24:00  2020-01-05_18:30:00   1.416106          1 command 'set -o pipefail && pvesm export vmdata:vm-100-disk-0 zfs - -with-snapshots 1 -snapshot __replicate_100-0_1578245100__ -base __replicate_100-0_1578245040__ | /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=lab-proxmox-01' root@10.0.60.91 -- pvesm import vmdata:vm-100-disk-0 zfs - -with-snapshots 1 -base __replicate_100-0_1578245040__' failed: exit code 1

As soon as I release the snapshost "vmdata/vm-100-disk-0@recv-99303-1", it get removed and the replication resume working.

Is this working as expected?
How I can use the snapshot to get an incremental backup and use the replication at the same time? Should I get the snapshot on Node-1 instead of Node-2?

Thanks
JackLayne

 

[mailinglists]

Is working as expected.
Replication is not backup.
Try pve-zsync to have more snapshots on both sides.
Try other ZFS backup scripts or write your own to have more snaps on destination than on source.

 

[narrateourale]

How I can use the snapshot to get an incremental backup and use the replication at the same time? Should I get the snapshot on Node-1 instead of Node-2?

Replication is using ZFS send & receive. So if you create a new snapshot on the replication target node (node 2) it will be reverted because in order to send the next incremental snapshot from the source the target needs to be on the same snapshot to start from.

 

[jacklayne]

Hello,

sorry for my late answer and thank you for your clarification.
I tested several solutions, including save the snapshot ( full, incremental and "differential" ) in a file to send it to software like borg or restic.
Unfortunately these software deduplicate only at file level and not block-level.
Furthermore there is the problem of snapshot consolidation in order to rotate the snapshots.
Only way I found is take a new full backup in order to rotate/delete the old one; if this work with small/medium size VMs it becomes more complicated with larger VMs ( like file server ).
It seems the only valid solution is using zfs send|receive to another pool.

I read someone's using borg successfully with containers, but no solution with VMs.
Any suggest?

Thanks