Replication error "out of space"

Jun 24, 2021
9
0
1
54
Hi everyone.
Problem: I have a cluster with three nodes.
In node A, I have a 1TB zfs volume with a VM (640 GB).
In node B I have a 1TB zfs volume on which I want to replicate the VM.
Both volumes are called "vmstore".
When I go to set up replication I get an error telling me that there is no space ...
The volume of node A is 68.82% occupied, is that the problem?
Do I need to have more free space?
Obvious that the volume of node B is empty.
TK
 

t.lamprecht

Proxmox Staff Member
Staff member
Jul 28, 2015
5,176
1,491
164
South Tyrol/Italy
shop.proxmox.com
Hi,

can you please post the output of the two commands
Bash:
zpool list
zfs list
from both nodes, just to be sure about space usage and availability?

Also, where do you get the error when a replication itself starts or already when setting up a new one?
 
Jun 24, 2021
9
0
1
54
Hi,

Node A

zpool list

Code:
root@pve:~# zpool list
NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rpool     464G  79.1G   385G        -         -     4%    17%  1.00x    ONLINE  -
vmstore   928G   360G   568G        -         -    12%    38%  1.00x    ONLINE  -

zfs list
Code:
root@pve:~# zfs list
NAME                       USED  AVAIL     REFER  MOUNTPOINT
rpool                     79.1G   370G      104K  /rpool
rpool/ROOT                1.23G   370G       96K  /rpool/ROOT
rpool/ROOT/pve-1          1.23G   370G     1.23G  /
rpool/data                77.9G   370G       96K  /rpool/data
rpool/data/vm-102-disk-0  77.9G   370G     77.9G  -
vmstore                    619G   280G       96K  /vmstore
vmstore/vm-100-disk-0      619G   539G      360G  -

Node B

zpool list
Code:
root@pveslave:~# zpool list
NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
vmstore   928G  1.16M   928G        -         -     0%     0%  1.00x    ONLINE  -

zfs list
Code:
root@pveslave:~# zfs list
NAME      USED  AVAIL     REFER  MOUNTPOINT
vmstore  1.16M   899G       96K  /vmstore

Schermata da 2022-01-17 11-57-02.png


I configure the replication, start the replication and after about 4 seconds the error message arrives.
TK
 
Last edited by a moderator:

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
7,272
1,334
164
likely your volume is thick-provisioned (has a refreservation set - you can check with zfs get all vmstore/vm-100-disk-0).

in this case, creating a snapshot requires at least as much free space as the volume currently uses:

- a thick provisioned volume has it's full size reserved (to make sure you can always write the full volume)
- creating a snapshot means the currently referenced data lives as long as that snapshot exists
- to ensure you can still fully (over)write the volume, the total amount of space reserved needs to grow by the data referenced by the snapshot (== currently used data)

if there isn't enough free space, creating the snapshot will fail. the logic gets a bit more involved when you add multiple snapshots (you then only need space for the blocks changed since the last snapshots), and for some pool setups there can be more overhead (raidz parity, etc.pp.).

so yeah, you either need more free space, or set the volume to be thin-provisioned (no reserved space, only actual usage is accounted).

for existing volumes, you can do that by setting the refreservation to 0. for future volumes, you can configure the storage to be thin-provisioned and PVE will not set a refreservation. the downside is that you can run out of space by writing inside the VM (since ZFS no longer ensures the space is there at volume/snapshot creation time), which can cause data loss or undefined behaviour (if you think of it from the guest's POV, this is like putting a lying disk or one of those fake USB drives into your server - it says it has 2TB of space, but after you've written 1TB it starts spewing errors) - so you need to carefully monitor your usage and trust your guests.

here's what the ZFS docs (man zfsprops) say about refreservation / thin/sparse volumes

Code:
refreservation=size|none|auto      

The minimum amount of space guaranteed to a dataset, not including its descendents.  When the amount of space used is below this value, the dataset is treated as if it were taking up the amount of space specified by refreservation.  The refreservation reservation is accounted for in the parent datasets' space used, and counts against the parent datasets' quotas and reservations.     
If refreservation is set, a snapshot is only allowed if there is enough free pool space outside of this reservation to accommodate       the current number of "referenced" bytes in the dataset.      

If refreservation is set to auto, a volume is thick provisioned (or "not sparse").  refreservation=auto is only supported on vol‐       umes.  See volsize in the Native Properties section for more information about sparse volumes.

....

volsize=size
...

The reservation is kept equal to the volume's logical size to prevent unexpected behavior for consumers.  Without the reservation,  the volume could run out of space, resulting in undefined behavior or data corruption, depending on how the volume is used.  These effects can also occur when the volume size is changed while it is in use (particularly when shrinking the size).  Extreme care should be used when adjusting the volume size.

Though not recommended, a "sparse volume" (also known as "thin provisioned") can be created by specifying the -s option to the zfs create -V command, or by changing the value of the refreservation property (or reservation property on pool version 8 or earlier) after the volume has been created.  A "sparse volume" is a volume where the value of refreservation is less than the size of the volume plus the space required to store its metadata.  Consequently, writes to a sparse volume can fail with ENOSPC when the pool is low on space.  For a sparse volume, changes to volsize are not reflected in the refreservation.  A volume that is not sparse is said to be "thick provisioned".  A sparse volume can become thick provisioned by setting refreservation to auto.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!