[SOLVED] Replication - space quota exceeded

P

Petr.114

Well-Known Member
Jun 25, 2019
37
2
48
32
Hello,
we are having problem with host replication.
The host have 16GB disk, which is about 25% full. But when replication starts,after 2GB trasnfered it ends with an error cannot receive incremental stream: destination rpool/data/subvol-150-disk-0 space quota exceeded
The error is about space quota, but we have not assigned any space quota.

Where can be the problem?
Thank you in advance.

2021-04-22 11:25:02 150-0: start replication job
2021-04-22 11:25:02 150-0: guest => CT 150, running => 1
2021-04-22 11:25:02 150-0: volumes => local-zfs:subvol-150-disk-0
2021-04-22 11:25:02 150-0: freeze guest filesystem
2021-04-22 11:25:02 150-0: create snapshot '__replicate_150-0_1619083500__' on local-zfs:subvol-150-disk-0
2021-04-22 11:25:02 150-0: thaw guest filesystem
2021-04-22 11:25:02 150-0: using secure transmission, rate limit: 10 MByte/s
2021-04-22 11:25:02 150-0: incremental sync 'local-zfs:subvol-150-disk-0' (__replicate_150-0_1617300025__ => __replicate_150-0_1619083500__)
2021-04-22 11:25:02 150-0: using a bandwidth limit of 10000000 bps for transferring 'local-zfs:subvol-150-disk-0'
2021-04-22 11:25:03 150-0: send from @__replicate_150-0_1617300025__ to rpool/data/subvol-150-disk-0@__replicate_150-0_1619083500__ estimated size is 2.01G
2021-04-22 11:25:03 150-0: total estimated size is 2.01G
2021-04-22 11:25:04 150-0: TIME SENT SNAPSHOT rpool/data/subvol-150-disk-0@__replicate_150-0_1619083500__
2021-04-22 11:25:04 150-0: 11:25:04 14.6M rpool/data/subvol-150-disk-0@__replicate_150-0_1619083500__
2021-04-22 11:25:05 150-0: 11:25:05 24.1M rpool/data/subvol-150-disk-0@__replicate_150-0_1619083500__
2021-04-22 11:25:06 150-0: 11:25:06 33.6M rpool/data/subvol-150-disk-0@__replicate_150-0_1619083500__
...
2021-04-22 11:28:39 150-0: 11:28:39 2.02G rpool/data/subvol-150-disk-0@__replicate_150-0_1619083500__
2021-04-22 11:28:40 150-0: 11:28:40 2.02G rpool/data/subvol-150-disk-0@__replicate_150-0_1619083500__
2021-04-22 11:28:41 150-0: 11:28:41 2.02G rpool/data/subvol-150-disk-0@__replicate_150-0_1619083500__
2021-04-22 11:28:47 150-0: cannot receive incremental stream: destination rpool/data/subvol-150-disk-0 space quota exceeded.
2021-04-22 11:28:47 150-0: cannot rollback 'rpool/data/subvol-150-disk-0': out of space
2021-04-22 11:28:47 150-0: command 'zfs recv -F -- rpool/data/subvol-150-disk-0' failed: exit code 1
2021-04-22 11:28:47 150-0: delete previous replication snapshot '__replicate_150-0_1619083500__' on local-zfs:subvol-150-disk-0
2021-04-22 11:28:47 150-0: end replication job with error: command 'set -o pipefail && pvesm export local-zfs:subvol-150-disk-0 zfs - -with-snapshots 1 -snapshot __replicate_150-0_1619083500__ -base __replicate_150-0_1617300025__ | /usr/bin/cstream -t 10000000 | /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=backup' root@192.168.0.18 -- pvesm import local-zfs:subvol-150-disk-0 zfs - -with-snapshots 1 -allow-rename 0 -base __replicate_150-0_1617300025__' failed: exit code 1
root@prox4:~$ zfs get space rpool/data/subvol-150-disk-0
NAME PROPERTY VALUE SOURCE
rpool/data/subvol-150-disk-0 name rpool/data/subvol-150-disk-0 -
rpool/data/subvol-150-disk-0 available 12.2G -
rpool/data/subvol-150-disk-0 used 3.82G -
rpool/data/subvol-150-disk-0 usedbysnapshots 16.5M -
rpool/data/subvol-150-disk-0 usedbydataset 3.80G -
rpool/data/subvol-150-disk-0 usedbyrefreservation 0B -
rpool/data/subvol-150-disk-0 usedbychildren 0B -

root@prox4:~$ zfs get quota rpool/data/subvol-150-disk-0
NAME PROPERTY VALUE SOURCE
rpool/data/subvol-150-disk-0 quota none default
root@backup:~$ zfs get quota rpool/data/subvol-150-disk-0
NAME PROPERTY VALUE SOURCE
rpool/data/subvol-150-disk-0 quota none default
root@prox4:~$ cat /etc/pve/nodes/prox4/lxc/150.conf
#192.168.0.65
#
#state 27/11/2019%3A asi na tom bezi zaznam packetu z rPi na Barakove a Milicove ohledne prejezdu LCA; taky webova aplikace od @lmezl
arch: amd64
cores: 1
hostname: lxc-lca
memory: 1024
nameserver: 192.168.7.2
net0: name=eth0,bridge=vmbr0,gw=192.168.0.1,hwaddr=5E:E0:C4:3A:6C:B2,ip=192.168.0.65/24,ip6=auto,type=veth
onboot: 1
ostype: debian
rootfs: local-zfs:subvol-150-disk-0,size=16G
searchdomain: lan.cutter.cz
startup: up=10
swap: 1024
 
Hi,
does the destination's rpool/data or rpool have a quota configured?

What is your pveversion -v?
 
Hello
There are not set quotas on rpool or rpool/data
root@backup:~$ zfs get quota rpool && zfs get quota rpool/data
NAME PROPERTY VALUE SOURCE
rpool quota none default
NAME PROPERTY VALUE SOURCE
rpool/data quota none default
Iam running proxmox VE 6.3-6, details bellow
proxmox-ve: 6.3-1 (running kernel: 5.4.78-2-pve)
pve-manager: 6.3-6 (running version: 6.3-6/2184247e)
pve-kernel-5.4: 6.3-8
pve-kernel-helper: 6.3-8
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.103-1-pve: 5.4.103-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.8
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-5
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-9
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.1-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-1
pve-cluster: 6.2-1
pve-container: 3.3-4
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-5
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-10
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1
Iam planning upgrade to 6.4-x at friday, but i dont expect it to fix this problem.
 
Petr.114 said:
2021-04-22 11:28:47 150-0: cannot rollback 'rpool/data/subvol-150-disk-0': out of space
Click to expand...
I feel like this might be the root cause. ZFS needs a little bit of space for a rollback operation. When receiving with the -F flag it does first attempt to rollback to the most recent snapshot.

What is the output of
Code: 
zfs list -o space rpool
zpool list rpool
on the destination side?
 
Thank you for trying to help.
Looks like there is planty of free space on the pool.

Code: 
root@backup:~$ zfs list -o space rpool
NAME   AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
rpool  4.23T  11.1T        0B    140K             0B      11.1T
root@backup:~$ zpool list rpool
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rpool  21.8T  15.3T  6.50T        -         -    13%    70%  1.00x    ONLINE  -
 
Did you do anything special (e.g. ZFS upgrade) before it started happening? Does it also happen with any other dataset?

What is the output of
Code: 
zfs get all rpool/data/subvol-150-disk-0
zfs get all rpool/data
zpool get all rpool
on both source and target?
 
I did not do anything, pool is showing that i can upgrade for some time, but i did not proceed.

The same thing happened on other proxmox server with different container so i tried to delete the replication (it destroyed CTs dataset too) and set it up again, but it did not helped and ended with error too. Then i found there is not needed snapshot, so i deleted that snapshot and then it started to work ¯\_(ツ)_/¯
But on the current CT 150, there is not any snapshot.

I had to upload the output externally due to character limit
 
Probably this is the reason:

source:
Code: 
rpool/data/subvol-150-disk-0  compression           on                             inherited from rpool
rpool/data/subvol-150-disk-0  logicalused           20.7G                          -
rpool/data/subvol-150-disk-0  logicalreferenced     20.6G                          -
rpool/data/subvol-150-disk-0  refquota              16G                            received

destination:
Code: 
rpool/data/subvol-150-disk-0  compression           off                            inherited from rpool
rpool/data/subvol-150-disk-0  refquota              16G                            received

as the uncompressed usage is likely bigger then the refquota.
 
Oh, good point.
So i will set zfs set compression=on rpool/data on destination and if i get it right, i need to delete the replication and subvol-150-disk-0 dataset to start it freshly with compression enabled. Is there any possibility to compress other VM/CT datasets so i can prevent problems with other replications? It is not very convenient to delete all replications and starting them over.

Thank you very much Fabian.
 
Petr.114 said:
Is there any possibility to compress other VM/CT datasets so i can prevent problems with other replications? It is not very convenient to delete all replications and starting them over.
Click to expand...
Maybe doing a full send/receive locally on the target itself works? I mean to create a local compressed copy (with all snapshots) with a different name. Then remove the original and rename back. Haven't tested it, but feels like it could be a way to do it.
 
Well i checked out how to use zfs send and zfs recv, but iam still not 100% sure and i really dont want to screw anything up.
Is this procedure with -R option right?

Code: 
zfs send -R rpool/data/vm-172-disk-1 | zfs recv rpool/data/vm-172-disk-1-new
zfs rename rpool/data/vm-172-disk-1 zpool/data/vm-172-disk-1-old
zfs rename rpool/data/vm-172-disk-1-new rpool/data/vm-172-disk-1
if everything is okay then
Code: 
zfs destroy zpool/data/vm-172-disk-1-old

Thanks for your help.
 
Petr.114 said:
Well i checked out how to use zfs send and zfs recv, but iam still not 100% sure and i really dont want to screw anything up.
Is this procedure with -R option right?

Code: 
zfs send -R rpool/data/vm-172-disk-1 | zfs recv rpool/data/vm-172-disk-1-new
zfs rename rpool/data/vm-172-disk-1 zpool/data/vm-172-disk-1-old
zfs rename rpool/data/vm-172-disk-1-new rpool/data/vm-172-disk-1
if everything is okay then
Code: 
zfs destroy zpool/data/vm-172-disk-1-old

Thanks for your help.
Click to expand...
Basically correct. You might also want to set the -p flag for the send command, so the properties like refquota will be copied too. But you don't want the compression property to be copied, so you also need to set -o compression=on for the receive command.

Note that I'm writing this from the top of my head so not 100% sure. Best to try it out, see if all snapshots are there and replication still works after replacing the dataset.
 
Well, i have used this command zfs send -Rp rpool/data/vm-172-disk-1 | zfs recv rpool/data/vm-172-disk-1-new but it didnt work, so i found out, that i need to create snapshot first and send the snapshot. Apparently there is not need for -o compression=on, because its already present on dataset, since i activated this option for rpool/data (-p should be enough).

root@backup:~$ zfs send -Rp rpool/data/vm-119-disk-0 | zfs recv rpool/data/vm-119-disk-0-new
Error: Unsupported flag with filesystem or bookmark.
cannot receive: failed to read from stream

Then I tried this procedure
root@backup:~$ zfs snapshot rpool/data/vm-119-disk-0@snap
root@backup:~$ zfs send -R rpool/data/vm-119-disk-0@snap | zfs recv rpool/data/vm-119-disk-0-new
root@backup:~$ zfs rename rpool/data/vm-119-disk-0 rpool/data/vm-119-disk-0-old
root@backup:~$ zfs rename rpool/data/vm-119-disk-0-new rpool/data/vm-119-disk-0
everything looks allright
root@backup:~$ zfs destroy -r rpool/data/vm-119-disk-0-old
iam not sure if it is okay to remove @snap from new dataset
root@backup:~$ zfs destroy rpool/data/vm-119-disk-0@snap

Also there is present more snapshots
root@backup:~$ zfs destroy rpool/data/vm-119-disk-0@
rpool/data/vm-119-disk-0@__replicate_119-0_1620385981__
rpool/data/vm-119-disk-0@__replicate_119-0_1620386460__
rpool/data/vm-119-disk-0@snap

Iam unsure if all data were transfered sucessfully since i think "logicalused" should be the same whether compressed or uncompressed. But maybe i am wrong....i think i am surely wrong.
zfs get all rpool/data/vm-119-disk-0 and zfs get all rpool/data/vm-119-disk-0-new
 
Last edited:
Petr.114 said:
Well, i have used this command zfs send -Rp rpool/data/vm-172-disk-1 | zfs recv rpool/data/vm-172-disk-1-new but it didnt work, so i found out, that i need to create snapshot first and send the snapshot. Apparently there is not need for -o compression=on, because its already present on dataset, since i activated this option for rpool/data (-p should be enough).

root@backup:~$ zfs send -Rp rpool/data/vm-119-disk-0 | zfs recv rpool/data/vm-119-disk-0-new
Error: Unsupported flag with filesystem or bookmark.
cannot receive: failed to read from stream

Then I tried this procedure
root@backup:~$ zfs snapshot rpool/data/vm-119-disk-0@snap
root@backup:~$ zfs send -R rpool/data/vm-119-disk-0@snap | zfs recv rpool/data/vm-119-disk-0-new
root@backup:~$ zfs rename rpool/data/vm-119-disk-0 rpool/data/vm-119-disk-0-old
root@backup:~$ zfs rename rpool/data/vm-119-disk-0-new rpool/data/vm-119-disk-0
everything looks allright
root@backup:~$ zfs destroy rpool/data/vm-119-disk-0-old
Click to expand...
Right, -R requires using a snapshot.

Petr.114 said:
iam not sure if it is okay to remove @snap from new dataset
root@backup:~$ zfs destroy -r rpool/data/vm-119-disk-0@snap
Click to expand...
It's fine, but you don't need the -r flag ;)

Petr.114 said:
Also there is present more snapshots
root@backup:~$ zfs destroy rpool/data/vm-119-disk-0@
rpool/data/vm-119-disk-0@__replicate_119-0_1620385981__
rpool/data/vm-119-disk-0@__replicate_119-0_1620386460__
rpool/data/vm-119-disk-0@snap
Click to expand...
There should be the same snapshots as on the original dataset. Did the replication run while you were doing this?

Petr.114 said:
Iam unsure if all data were transfered sucessfully since i think "logicalused" should be the same whether compressed or uncompressed. But maybe i am wrong....i think i am surely wrong.
zfs get all rpool/data/vm-119-disk-0 and zfs get all rpool/data/vm-119-disk-0-new
Click to expand...
Yes, it should be, but this might be an explanation.
 
Fabian_E said:
It's fine, but you don't need the -r flag ;)
Click to expand...
My bad, -r flag had to be in previous command zfs destroy -r rpool/data/vm-119-disk-0-old
I'll edit it in my previous post, so people who find this will get it right.

Fabian_E said:
There should be the same snapshots as on the original dataset. Did the replication run while you were doing this?
Click to expand...
Yeah, snapshots were the same. Replication was not set up for this VM

I think i got everything i needed.

Thank you very much for your assistance Fabian.
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom