ZFS replication gives "our of space"

[greg]

Greetings
I'm trying to have a replication job between two nodes of a cluster, like this:

pvesr run -id 111-2 -verbose

After transfering 173G, the process stops with "cannot receive incremental stream: out of space". The free space on the receiving end is 1.65T.

What am I doing wrong?

Thanks in advance

Regards

 

[fabian]

please post the VM config, as well as 'zpool status' on both nodes, and 'zfs get all $DATASET' on both nodes for all disks of that VM.

 

[greg]

Thanks for your answer. Here are the info:


VM config:

arch: amd64
cores: 1
hostname: xxxxx
memory: 1000
net0: name=eth1,bridge=vmbr1,hwaddr=xxxx,ip=xxxxx,type=veth
net1: name=eth0,bridge=vmbr0,gw=xxxxx,hwaddr=xxxxx,ip=xxxx,type=veth
net2: name=eth3,bridge=vmbr0,firewall=1,gw=xxxx4,hwaddr=xxxx,ip=xxxx,type=veth
onboot: 1
ostype: debian
parent: daily_20181126
rootfs: ct_B:subvol-111-disk-1,size=1000G
swap: 4000
unprivileged: 1


not sure the meaning of the "parent" field, there are no more snapshot for this VM.

On the source :

# zpool status
pool: ct_B
state: ONLINE
scan: scrub repaired 0B in 16h45m with 0 errors on Sun Jul 12 17:09:03 2020
config:

NAME STATE READ WRITE CKSUM
ct_B ONLINE 0 0 0
ata-HGST_HUS7240xxxxxX-part1 ONLINE 0 0 0
ata-HGST_HUS72402xxxxxxpart1 ONLINE 0 0 0



NAME PROPERTY VALUE SOURCE
ct_B/ct/subvol-111-disk-1 type filesystem -
ct_B/ct/subvol-111-disk-1 creation mar. oct. 30 14:43 2018 -
ct_B/ct/subvol-111-disk-1 used 1,83T -
ct_B/ct/subvol-111-disk-1 available 893G -
ct_B/ct/subvol-111-disk-1 referenced 1,82T -
ct_B/ct/subvol-111-disk-1 compressratio 1.00x -
ct_B/ct/subvol-111-disk-1 mounted yes -
ct_B/ct/subvol-111-disk-1 quota none default
ct_B/ct/subvol-111-disk-1 reservation none default
ct_B/ct/subvol-111-disk-1 recordsize 128K default
ct_B/ct/subvol-111-disk-1 mountpoint /ct_B/ct/subvol-111-disk-1 default
ct_B/ct/subvol-111-disk-1 sharenfs off default
ct_B/ct/subvol-111-disk-1 checksum on default
ct_B/ct/subvol-111-disk-1 compression off default
ct_B/ct/subvol-111-disk-1 atime on default
ct_B/ct/subvol-111-disk-1 devices on default
ct_B/ct/subvol-111-disk-1 exec on default
ct_B/ct/subvol-111-disk-1 setuid on default
ct_B/ct/subvol-111-disk-1 readonly off default
ct_B/ct/subvol-111-disk-1 zoned off default
ct_B/ct/subvol-111-disk-1 snapdir hidden default
ct_B/ct/subvol-111-disk-1 aclinherit restricted default
ct_B/ct/subvol-111-disk-1 createtxg 176767 -
ct_B/ct/subvol-111-disk-1 canmount on default
ct_B/ct/subvol-111-disk-1 xattr on default
ct_B/ct/subvol-111-disk-1 copies 1 default
ct_B/ct/subvol-111-disk-1 version 5 -
ct_B/ct/subvol-111-disk-1 utf8only off -
ct_B/ct/subvol-111-disk-1 normalization none -
ct_B/ct/subvol-111-disk-1 casesensitivity sensitive -
ct_B/ct/subvol-111-disk-1 vscan off default
ct_B/ct/subvol-111-disk-1 nbmand off default
ct_B/ct/subvol-111-disk-1 sharesmb off default
ct_B/ct/subvol-111-disk-1 refquota none default
ct_B/ct/subvol-111-disk-1 refreservation none default
ct_B/ct/subvol-111-disk-1 guid 947576344036297752 -
ct_B/ct/subvol-111-disk-1 primarycache all default
ct_B/ct/subvol-111-disk-1 secondarycache all default
ct_B/ct/subvol-111-disk-1 usedbysnapshots 11,1G -
ct_B/ct/subvol-111-disk-1 usedbydataset 1,82T -
ct_B/ct/subvol-111-disk-1 usedbychildren 0B -
ct_B/ct/subvol-111-disk-1 usedbyrefreservation 0B -
ct_B/ct/subvol-111-disk-1 logbias latency default
ct_B/ct/subvol-111-disk-1 dedup off default
ct_B/ct/subvol-111-disk-1 mlslabel none default
ct_B/ct/subvol-111-disk-1 sync standard default
ct_B/ct/subvol-111-disk-1 dnodesize legacy default
ct_B/ct/subvol-111-disk-1 refcompressratio 1.00x -
ct_B/ct/subvol-111-disk-1 written 4,50M -
ct_B/ct/subvol-111-disk-1 logicalused 1,83T -
ct_B/ct/subvol-111-disk-1 logicalreferenced 1,82T -
ct_B/ct/subvol-111-disk-1 volmode default default
ct_B/ct/subvol-111-disk-1 filesystem_limit none default
ct_B/ct/subvol-111-disk-1 snapshot_limit none default
ct_B/ct/subvol-111-disk-1 filesystem_count none default
ct_B/ct/subvol-111-disk-1 snapshot_count none default
ct_B/ct/subvol-111-disk-1 snapdev hidden default
ct_B/ct/subvol-111-disk-1 acltype off default
ct_B/ct/subvol-111-disk-1 context none default
ct_B/ct/subvol-111-disk-1 fscontext none default
ct_B/ct/subvol-111-disk-1 defcontext none default
ct_B/ct/subvol-111-disk-1 rootcontext none default
ct_B/ct/subvol-111-disk-1 relatime off default
ct_B/ct/subvol-111-disk-1 redundant_metadata all default
ct_B/ct/subvol-111-disk-1 overlay off default

on the starget:

# zpool status
pool: ct_B
state: ONLINE
scan: scrub repaired 0B in 0 days 01:49:20 with 0 errors on Sun Jul 12 02:13:22 2020
config:

NAME STATE READ WRITE CKSUM
ct_B ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-HGST_HUS72602xxxxx-part3 ONLINE 0 0 0
ata-HGST_HUS72602xxxxx-part3 ONLINE 0 0 0
ata-HGST_HUS72602xxxxx-part3 ONLINE 0 0 0


On the target the zfs fs doesn't exists prior to the sync.

 

[greg]

It's weird! now a 1.36T zfs fs exists on the target ?!?

 

[fabian]

your source dataset is 1.8T big..

 

[greg]

yes the target is supposed to have 2T more free... I'm assuming there's a background copy process I missed somehow... I created the replication task in the GUI so I guess it's been automatically queued and my manual run is interfering...
what's really weird is that I ran manually because the GUI told me the task failed for some obscure reason ("command 'set -o pipefail && pvesm export").
I will let it finish and see ...

Thanks a lot!