[SOLVED] Replication - space quota exceeded

Petr.114

Member
Jun 25, 2019
28
1
8
30
Hello,
we are having problem with host replication.
The host have 16GB disk, which is about 25% full. But when replication starts,after 2GB trasnfered it ends with an error cannot receive incremental stream: destination rpool/data/subvol-150-disk-0 space quota exceeded
The error is about space quota, but we have not assigned any space quota.

Where can be the problem?
Thank you in advance.

2021-04-22 11:25:02 150-0: start replication job
2021-04-22 11:25:02 150-0: guest => CT 150, running => 1
2021-04-22 11:25:02 150-0: volumes => local-zfs:subvol-150-disk-0
2021-04-22 11:25:02 150-0: freeze guest filesystem
2021-04-22 11:25:02 150-0: create snapshot '__replicate_150-0_1619083500__' on local-zfs:subvol-150-disk-0
2021-04-22 11:25:02 150-0: thaw guest filesystem
2021-04-22 11:25:02 150-0: using secure transmission, rate limit: 10 MByte/s
2021-04-22 11:25:02 150-0: incremental sync 'local-zfs:subvol-150-disk-0' (__replicate_150-0_1617300025__ => __replicate_150-0_1619083500__)
2021-04-22 11:25:02 150-0: using a bandwidth limit of 10000000 bps for transferring 'local-zfs:subvol-150-disk-0'
2021-04-22 11:25:03 150-0: send from @__replicate_150-0_1617300025__ to rpool/data/subvol-150-disk-0@__replicate_150-0_1619083500__ estimated size is 2.01G
2021-04-22 11:25:03 150-0: total estimated size is 2.01G
2021-04-22 11:25:04 150-0: TIME SENT SNAPSHOT rpool/data/subvol-150-disk-0@__replicate_150-0_1619083500__
2021-04-22 11:25:04 150-0: 11:25:04 14.6M rpool/data/subvol-150-disk-0@__replicate_150-0_1619083500__
2021-04-22 11:25:05 150-0: 11:25:05 24.1M rpool/data/subvol-150-disk-0@__replicate_150-0_1619083500__
2021-04-22 11:25:06 150-0: 11:25:06 33.6M rpool/data/subvol-150-disk-0@__replicate_150-0_1619083500__
...
2021-04-22 11:28:39 150-0: 11:28:39 2.02G rpool/data/subvol-150-disk-0@__replicate_150-0_1619083500__
2021-04-22 11:28:40 150-0: 11:28:40 2.02G rpool/data/subvol-150-disk-0@__replicate_150-0_1619083500__
2021-04-22 11:28:41 150-0: 11:28:41 2.02G rpool/data/subvol-150-disk-0@__replicate_150-0_1619083500__
2021-04-22 11:28:47 150-0: cannot receive incremental stream: destination rpool/data/subvol-150-disk-0 space quota exceeded.
2021-04-22 11:28:47 150-0: cannot rollback 'rpool/data/subvol-150-disk-0': out of space
2021-04-22 11:28:47 150-0: command 'zfs recv -F -- rpool/data/subvol-150-disk-0' failed: exit code 1
2021-04-22 11:28:47 150-0: delete previous replication snapshot '__replicate_150-0_1619083500__' on local-zfs:subvol-150-disk-0
2021-04-22 11:28:47 150-0: end replication job with error: command 'set -o pipefail && pvesm export local-zfs:subvol-150-disk-0 zfs - -with-snapshots 1 -snapshot __replicate_150-0_1619083500__ -base __replicate_150-0_1617300025__ | /usr/bin/cstream -t 10000000 | /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=backup' root@192.168.0.18 -- pvesm import local-zfs:subvol-150-disk-0 zfs - -with-snapshots 1 -allow-rename 0 -base __replicate_150-0_1617300025__' failed: exit code 1
root@prox4:~$ zfs get space rpool/data/subvol-150-disk-0
NAME PROPERTY VALUE SOURCE
rpool/data/subvol-150-disk-0 name rpool/data/subvol-150-disk-0 -
rpool/data/subvol-150-disk-0 available 12.2G -
rpool/data/subvol-150-disk-0 used 3.82G -
rpool/data/subvol-150-disk-0 usedbysnapshots 16.5M -
rpool/data/subvol-150-disk-0 usedbydataset 3.80G -
rpool/data/subvol-150-disk-0 usedbyrefreservation 0B -
rpool/data/subvol-150-disk-0 usedbychildren 0B -

root@prox4:~$ zfs get quota rpool/data/subvol-150-disk-0
NAME PROPERTY VALUE SOURCE
rpool/data/subvol-150-disk-0 quota none default
root@backup:~$ zfs get quota rpool/data/subvol-150-disk-0
NAME PROPERTY VALUE SOURCE
rpool/data/subvol-150-disk-0 quota none default
root@prox4:~$ cat /etc/pve/nodes/prox4/lxc/150.conf
#192.168.0.65
#
#state 27/11/2019%3A asi na tom bezi zaznam packetu z rPi na Barakove a Milicove ohledne prejezdu LCA; taky webova aplikace od @lmezl
arch: amd64
cores: 1
hostname: lxc-lca
memory: 1024
nameserver: 192.168.7.2
net0: name=eth0,bridge=vmbr0,gw=192.168.0.1,hwaddr=5E:E0:C4:3A:6C:B2,ip=192.168.0.65/24,ip6=auto,type=veth
onboot: 1
ostype: debian
rootfs: local-zfs:subvol-150-disk-0,size=16G
searchdomain: lan.cutter.cz
startup: up=10
swap: 1024
 

Fabian_E

Proxmox Staff Member
Staff member
Aug 1, 2019
1,738
301
88
Hi,
does the destination's rpool/data or rpool have a quota configured?

What is your pveversion -v?
 

Petr.114

Member
Jun 25, 2019
28
1
8
30
Hello
There are not set quotas on rpool or rpool/data
root@backup:~$ zfs get quota rpool && zfs get quota rpool/data
NAME PROPERTY VALUE SOURCE
rpool quota none default
NAME PROPERTY VALUE SOURCE
rpool/data quota none default
Iam running proxmox VE 6.3-6, details bellow
proxmox-ve: 6.3-1 (running kernel: 5.4.78-2-pve)
pve-manager: 6.3-6 (running version: 6.3-6/2184247e)
pve-kernel-5.4: 6.3-8
pve-kernel-helper: 6.3-8
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.103-1-pve: 5.4.103-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.8
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-5
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-9
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.1-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-1
pve-cluster: 6.2-1
pve-container: 3.3-4
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-5
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-10
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1
Iam planning upgrade to 6.4-x at friday, but i dont expect it to fix this problem.
 

Fabian_E

Proxmox Staff Member
Staff member
Aug 1, 2019
1,738
301
88
2021-04-22 11:28:47 150-0: cannot rollback 'rpool/data/subvol-150-disk-0': out of space
I feel like this might be the root cause. ZFS needs a little bit of space for a rollback operation. When receiving with the -F flag it does first attempt to rollback to the most recent snapshot.

What is the output of
Code:
zfs list -o space rpool
zpool list rpool
on the destination side?
 

Petr.114

Member
Jun 25, 2019
28
1
8
30
Thank you for trying to help.
Looks like there is planty of free space on the pool.

Code:
root@backup:~$ zfs list -o space rpool
NAME   AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
rpool  4.23T  11.1T        0B    140K             0B      11.1T
root@backup:~$ zpool list rpool
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rpool  21.8T  15.3T  6.50T        -         -    13%    70%  1.00x    ONLINE  -
 

Fabian_E

Proxmox Staff Member
Staff member
Aug 1, 2019
1,738
301
88
Did you do anything special (e.g. ZFS upgrade) before it started happening? Does it also happen with any other dataset?

What is the output of
Code:
zfs get all rpool/data/subvol-150-disk-0
zfs get all rpool/data
zpool get all rpool
on both source and target?
 

Petr.114

Member
Jun 25, 2019
28
1
8
30
I did not do anything, pool is showing that i can upgrade for some time, but i did not proceed.

The same thing happened on other proxmox server with different container so i tried to delete the replication (it destroyed CTs dataset too) and set it up again, but it did not helped and ended with error too. Then i found there is not needed snapshot, so i deleted that snapshot and then it started to work ¯\_(ツ)_/¯
But on the current CT 150, there is not any snapshot.

I had to upload the output externally due to character limit
 

Fabian_E

Proxmox Staff Member
Staff member
Aug 1, 2019
1,738
301
88
Probably this is the reason:

source:
Code:
rpool/data/subvol-150-disk-0  compression           on                             inherited from rpool
rpool/data/subvol-150-disk-0  logicalused           20.7G                          -
rpool/data/subvol-150-disk-0  logicalreferenced     20.6G                          -
rpool/data/subvol-150-disk-0  refquota              16G                            received

destination:
Code:
rpool/data/subvol-150-disk-0  compression           off                            inherited from rpool
rpool/data/subvol-150-disk-0  refquota              16G                            received

as the uncompressed usage is likely bigger then the refquota.
 

Petr.114

Member
Jun 25, 2019
28
1
8
30
Oh, good point.
So i will set zfs set compression=on rpool/data on destination and if i get it right, i need to delete the replication and subvol-150-disk-0 dataset to start it freshly with compression enabled. Is there any possibility to compress other VM/CT datasets so i can prevent problems with other replications? It is not very convenient to delete all replications and starting them over.

Thank you very much Fabian.
 

Fabian_E

Proxmox Staff Member
Staff member
Aug 1, 2019
1,738
301
88
Is there any possibility to compress other VM/CT datasets so i can prevent problems with other replications? It is not very convenient to delete all replications and starting them over.
Maybe doing a full send/receive locally on the target itself works? I mean to create a local compressed copy (with all snapshots) with a different name. Then remove the original and rename back. Haven't tested it, but feels like it could be a way to do it.
 

Petr.114

Member
Jun 25, 2019
28
1
8
30
Well i checked out how to use zfs send and zfs recv, but iam still not 100% sure and i really dont want to screw anything up.
Is this procedure with -R option right?

Code:
zfs send -R rpool/data/vm-172-disk-1 | zfs recv rpool/data/vm-172-disk-1-new
zfs rename rpool/data/vm-172-disk-1 zpool/data/vm-172-disk-1-old
zfs rename rpool/data/vm-172-disk-1-new rpool/data/vm-172-disk-1
if everything is okay then
Code:
zfs destroy zpool/data/vm-172-disk-1-old

Thanks for your help.
 

Fabian_E

Proxmox Staff Member
Staff member
Aug 1, 2019
1,738
301
88
Well i checked out how to use zfs send and zfs recv, but iam still not 100% sure and i really dont want to screw anything up.
Is this procedure with -R option right?

Code:
zfs send -R rpool/data/vm-172-disk-1 | zfs recv rpool/data/vm-172-disk-1-new
zfs rename rpool/data/vm-172-disk-1 zpool/data/vm-172-disk-1-old
zfs rename rpool/data/vm-172-disk-1-new rpool/data/vm-172-disk-1
if everything is okay then
Code:
zfs destroy zpool/data/vm-172-disk-1-old

Thanks for your help.
Basically correct. You might also want to set the -p flag for the send command, so the properties like refquota will be copied too. But you don't want the compression property to be copied, so you also need to set -o compression=on for the receive command.

Note that I'm writing this from the top of my head so not 100% sure. Best to try it out, see if all snapshots are there and replication still works after replacing the dataset.
 

Petr.114

Member
Jun 25, 2019
28
1
8
30
Well, i have used this command zfs send -Rp rpool/data/vm-172-disk-1 | zfs recv rpool/data/vm-172-disk-1-new but it didnt work, so i found out, that i need to create snapshot first and send the snapshot. Apparently there is not need for -o compression=on, because its already present on dataset, since i activated this option for rpool/data (-p should be enough).

root@backup:~$ zfs send -Rp rpool/data/vm-119-disk-0 | zfs recv rpool/data/vm-119-disk-0-new
Error: Unsupported flag with filesystem or bookmark.
cannot receive: failed to read from stream

Then I tried this procedure
root@backup:~$ zfs snapshot rpool/data/vm-119-disk-0@snap
root@backup:~$ zfs send -R rpool/data/vm-119-disk-0@snap | zfs recv rpool/data/vm-119-disk-0-new
root@backup:~$ zfs rename rpool/data/vm-119-disk-0 rpool/data/vm-119-disk-0-old
root@backup:~$ zfs rename rpool/data/vm-119-disk-0-new rpool/data/vm-119-disk-0
everything looks allright
root@backup:~$ zfs destroy -r rpool/data/vm-119-disk-0-old
iam not sure if it is okay to remove @snap from new dataset
root@backup:~$ zfs destroy rpool/data/vm-119-disk-0@snap

Also there is present more snapshots
root@backup:~$ zfs destroy rpool/data/vm-119-disk-0@
rpool/data/vm-119-disk-0@__replicate_119-0_1620385981__
rpool/data/vm-119-disk-0@__replicate_119-0_1620386460__
rpool/data/vm-119-disk-0@snap

Iam unsure if all data were transfered sucessfully since i think "logicalused" should be the same whether compressed or uncompressed. But maybe i am wrong....i think i am surely wrong.
zfs get all rpool/data/vm-119-disk-0 and zfs get all rpool/data/vm-119-disk-0-new
 
Last edited:

Fabian_E

Proxmox Staff Member
Staff member
Aug 1, 2019
1,738
301
88
Well, i have used this command zfs send -Rp rpool/data/vm-172-disk-1 | zfs recv rpool/data/vm-172-disk-1-new but it didnt work, so i found out, that i need to create snapshot first and send the snapshot. Apparently there is not need for -o compression=on, because its already present on dataset, since i activated this option for rpool/data (-p should be enough).

root@backup:~$ zfs send -Rp rpool/data/vm-119-disk-0 | zfs recv rpool/data/vm-119-disk-0-new
Error: Unsupported flag with filesystem or bookmark.
cannot receive: failed to read from stream

Then I tried this procedure
root@backup:~$ zfs snapshot rpool/data/vm-119-disk-0@snap
root@backup:~$ zfs send -R rpool/data/vm-119-disk-0@snap | zfs recv rpool/data/vm-119-disk-0-new
root@backup:~$ zfs rename rpool/data/vm-119-disk-0 rpool/data/vm-119-disk-0-old
root@backup:~$ zfs rename rpool/data/vm-119-disk-0-new rpool/data/vm-119-disk-0
everything looks allright
root@backup:~$ zfs destroy rpool/data/vm-119-disk-0-old
Right, -R requires using a snapshot.

iam not sure if it is okay to remove @snap from new dataset
root@backup:~$ zfs destroy -r rpool/data/vm-119-disk-0@snap
It's fine, but you don't need the -r flag ;)

Also there is present more snapshots
root@backup:~$ zfs destroy rpool/data/vm-119-disk-0@
rpool/data/vm-119-disk-0@__replicate_119-0_1620385981__
rpool/data/vm-119-disk-0@__replicate_119-0_1620386460__
rpool/data/vm-119-disk-0@snap
There should be the same snapshots as on the original dataset. Did the replication run while you were doing this?

Iam unsure if all data were transfered sucessfully since i think "logicalused" should be the same whether compressed or uncompressed. But maybe i am wrong....i think i am surely wrong.
zfs get all rpool/data/vm-119-disk-0 and zfs get all rpool/data/vm-119-disk-0-new
Yes, it should be, but this might be an explanation.
 

Petr.114

Member
Jun 25, 2019
28
1
8
30
It's fine, but you don't need the -r flag ;)
My bad, -r flag had to be in previous command zfs destroy -r rpool/data/vm-119-disk-0-old
I'll edit it in my previous post, so people who find this will get it right.

There should be the same snapshots as on the original dataset. Did the replication run while you were doing this?
Yeah, snapshots were the same. Replication was not set up for this VM

I think i got everything i needed.

Thank you very much for your assistance Fabian.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!