Replication Space Issues

Alexandre Aguirre

Well-Known Member
Apr 4, 2018
Passo Fundo - RS, Brasil
Hello good day !

I'm having problems with replication in the following question, my vm was with 300GB and I had to increase another 300GB, so far so good, the vm was with 600GB, kept performing the replication perfectly, but now that comes the problem ... ...... when I started passing data to this vm, the replica started to use all the zfs disk space from node 2, the vm has 600GB and the zfs storage from node 2 has 900GB, and after doing the replicas, he started using all the space generating error out of space

Would anyone know inform me the possible problem?

Thank you in advance for your attention
is anything else using the same pool? What does
zfs list -o space -r <pool> -t all
show on both nodes?
Using 'refreservation' reserves space for the single dataset in question. From the man page:
The minimum amount of space guaranteed to a dataset, not including its descendents. When the amount of space used is below this value, the dataset is treated as if it were taking up the amount of space specified by
refreservation. The refreservation reservation is accounted for in the parent datasets' space used, and counts against the parent datasets' quotas and reservations.
If refreservation is set, a snapshot is only allowed if there is enough free pool space outside of this reservation to accommodate the current number of "referenced" bytes in the dataset.

Since you created a snapshot and a snapshot is a new dataset, the data it references has to be outside of the 'refreservation'. Otherwise ZFS couldn't guarantee for 'refreservation' for the parent dataset anymore.

The following illustrates the behavior:
root@rob0 ~ # zfs create -V 10M src/myvol      
root@rob0 ~ # zfs set refreservation=20M src/myvol
root@rob0 ~ # dd if=somedata of=/dev/zvol/src/myvol
7304+1 records in
7304+1 records out
3740064 bytes (3.7 MB, 3.6 MiB) copied, 0.805947 s, 4.6 MB/s
root@rob0 ~ # zfs list -o space -r src -t all
src        19.8M  20.2M        0B     24K             0B      20.2M
src/myvol  36.2M    20M        0B   3.62M          16.4M         0B
root@rob0 ~ # zfs snap src/myvol@mysnap  
root@rob0 ~ # zfs list -o space -r src -t all
src               16.1M  23.8M        0B     24K             0B      23.8M
src/myvol         36.1M  23.6M        0B   3.62M            20M         0B
src/myvol@mysnap      -     0B         -       -              -          -

So you might need to use 'reservation' instead of 'refreservation'. Note that then the child datasets like snapshots will share the amount of reserved space with the parent dataset.


The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!