[SOLVED] ERROR IN ZFS REPLICATION

[fabricio.biri]

Hi folks!
I have a node and installed another just for replication. I've created a zfs where VM disks are installed. The second node have almost 100% free space, but when I start the replication process this happens:
2023-06-15 11:25:01 100-0: start replication job
2023-06-15 11:25:01 100-0: guest => VM 100, running => 3094
2023-06-15 11:25:01 100-0: volumes => local-zfs:vm-100-disk-0,local-zfs:vm-100-disk-1
2023-06-15 11:25:02 100-0: freeze guest filesystem
2023-06-15 11:25:02 100-0: create snapshot '__replicate_100-0_1686839101__' on local-zfs:vm-100-disk-0
2023-06-15 11:25:02 100-0: thaw guest filesystem
2023-06-15 11:25:02 100-0: end replication job with error: zfs error: cannot create snapshot 'local-zfs/vm-100-disk-0@__replicate_100-0_1686839101__': out of space

NODE 1 (ORIGIN)
root@teslapve:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
local-zfs 2.72T 672G 96K /local-zfs
local-zfs/subvol-101-disk-0 330M 99.7G 330M /local-zfs/subvol-101-disk-0
local-zfs/vm-100-disk-0 1.21T 757G 1.13T -
local-zfs/vm-100-disk-1 1.51T 795G 1.39T -

NODE 2 (DESTINATION)
root@newton-pve:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
local-zfs 606M 3.60T 104K /local-zfs
local-zfs/subvol-101-disk-0 604M 99.4G 604M /local-zfs/subvol-101-disk-0

Both nodes are in version 7.1-13

Can anyone help me please? Thanks in advance.

 

[shanreich]

Most likely your disk vm-100-disk-0 is thickly provisioned. That means when taking a snapshot (happens when you run replication) you need at least as much space as the volume currently occupies.

You can check if a volume is thickly provisioned by running the command zfs get all local-zfs/vm-100-disk-0 and looking at the refreservation

You either need more space or set the volume to be thinly provisioned (for existing volumes this means setting the refreservation to 0). The downside of thin-provisioned volumes is that you have to be very careful because you can run out of space and lose data so keep that in mind.

 

[fabricio.biri]

Most likely your disk vm-100-disk-0 is thickly provisioned. That means when taking a snapshot (happens when you run replication) you need at least as much space as the volume currently occupies.

You can check if a volume is thickly provisioned by running the command zfs get all local-zfs/vm-100-disk-0 and looking at the refreservation

You either need more space or set the volume to be thinly provisioned (for existing volumes this means setting the refreservation to 0). The downside of thin-provisioned volumes is that you have to be very careful because you can run out of space and lose data so keep that in mind.

root@teslapve:~# zfs get all local-zfs/vm-100-disk-0 | grep refreservation
local-zfs/vm-100-disk-0 refreservation 1.21T local
local-zfs/vm-100-disk-0 usedbyrefreservation 85.3G -

The amount used is very small right? Do you think it's necessary to enable thin provision anyway?

 

[shanreich]

 local-zfs/vm-100-disk-0 refreservation 1.21T local

The refreservation is the deciding factor (not the usedbyrefreservation) - this is 1.2 TB. I suggest you familiarize yourself with the meaning of the terms [1] (see the Section 'Thick-Provisioned Zvols and Snapshots'). Nevertheless, in order to be able to run the replication you will have to change to thinly-provisioned volumes (or expand storage)

[1] https://jro.io/truenas/openzfs/

 

[fabricio.biri]

 local-zfs/vm-100-disk-0 refreservation 1.21T local

The refreservation is the deciding factor (not the usedbyrefreservation) - this is 1.2 TB. I suggest you familiarize yourself with the meaning of the terms [1] (see the Section 'Thick-Provisioned Zvols and Snapshots'). Nevertheless, in order to be able to run the replication you will have to change to thinly-provisioned volumes (or expand storage)

[1] https://jro.io/truenas/openzfs/

I've read and undertood the meaning. I think thin provision is good for me. I've increased the pool to 7 TB but the error persists. And the error occurs at the first disc replication. This disc have only 1.2T

 

[fabricio.biri]

After disable refreservation, the replication worked. Thanks