ZFS Replication Error: out of space

Deco20 · Oct 14, 2025

Hello everyone,

I have been a silent reader here in the forum for quite some time. Since there are already threads on so many topics/problems here, that has always been sufficient so far.
Now I am currently facing a problem/phenomenon that I cannot quite understand, and although there are some similar cases on the topic, I have not found anything that directly applies to my situation.

I have three nodes in my PVE environment. On one (pve1) there is a PBS VM (I know that PBS is not the best alternative as a VM), which I replicate to my other nodes. This replication has been failing to one of the nodes for some time now. It ran for a few more days to the other node, but is now also failing.

Code:

2025-10-14 18:20:08 100-0: start replication job
2025-10-14 18:20:08 100-0: guest => VM 100, running => 1680
2025-10-14 18:20:08 100-0: volumes => local-zfs1:vm-100-disk-0,local-zfs1:vm-100-disk-1,local-zfs1:vm-100-disk-2
2025-10-14 18:20:09 100-0: freeze guest filesystem
2025-10-14 18:20:09 100-0: create snapshot '__replicate_100-0_1760458808__' on local-zfs1:vm-100-disk-0
2025-10-14 18:20:09 100-0: create snapshot '__replicate_100-0_1760458808__' on local-zfs1:vm-100-disk-1
2025-10-14 18:20:09 100-0: create snapshot '__replicate_100-0_1760458808__' on local-zfs1:vm-100-disk-2
2025-10-14 18:20:09 100-0: thaw guest filesystem
2025-10-14 18:20:09 100-0: using secure transmission, rate limit: none
2025-10-14 18:20:09 100-0: incremental sync 'local-zfs1:vm-100-disk-0' (__replicate_100-0_1760389801__ => __replicate_100-0_1760458808__)
2025-10-14 18:20:10 100-0: send from @__replicate_100-0_1760389801__ to local-zfs1/vm-100-disk-0@__replicate_100-0_1760458808__ estimated size is 3.94G
2025-10-14 18:20:10 100-0: total estimated size is 3.94G
2025-10-14 18:20:10 100-0: TIME        SENT   SNAPSHOT local-zfs1/vm-100-disk-0@__replicate_100-0_1760458808__
2025-10-14 18:20:11 100-0: 18:20:11   65.0M   local-zfs1/vm-100-disk-0@__replicate_100-0_1760458808__
2025-10-14 18:20:12 100-0: 18:20:12    177M   local-zfs1/vm-100-disk-0@__replicate_100-0_1760458808__
2025-10-14 18:20:13 100-0: 18:20:13    289M   local-zfs1/vm-100-disk-0@__replicate_100-0_1760458808__
2025-10-14 18:20:14 100-0: 18:20:14    401M   local-zfs1/vm-100-disk-0@__replicate_100-0_1760458808__
2025-10-14 18:20:15 100-0: 18:20:15    513M   local-zfs1/vm-100-disk-0@__replicate_100-0_1760458808__
2025-10-14 18:20:16 100-0: 18:20:16    625M   local-zfs1/vm-100-disk-0@__replicate_100-0_1760458808__
2025-10-14 18:20:17 100-0: 18:20:17    737M   local-zfs1/vm-100-disk-0@__replicate_100-0_1760458808__
2025-10-14 18:20:18 100-0: 18:20:18    849M   local-zfs1/vm-100-disk-0@__replicate_100-0_1760458808__
2025-10-14 18:20:18 100-0: cannot receive incremental stream: out of space
2025-10-14 18:20:18 100-0: command 'zfs recv -F -- local-zfs1/vm-100-disk-0' failed: exit code 1
2025-10-14 18:20:18 100-0: warning: cannot send 'local-zfs1/vm-100-disk-0@__replicate_100-0_1760458808__': signal received
2025-10-14 18:20:18 100-0: cannot send 'local-zfs1/vm-100-disk-0': I/O error
2025-10-14 18:20:18 100-0: command 'zfs send -Rpv -I __replicate_100-0_1760389801__ -- local-zfs1/vm-100-disk-0@__replicate_100-0_1760458808__' failed: exit code 1
2025-10-14 18:20:18 100-0: delete previous replication snapshot '__replicate_100-0_1760458808__' on local-zfs1:vm-100-disk-0
2025-10-14 18:20:18 100-0: delete previous replication snapshot '__replicate_100-0_1760458808__' on local-zfs1:vm-100-disk-1
2025-10-14 18:20:18 100-0: delete previous replication snapshot '__replicate_100-0_1760458808__' on local-zfs1:vm-100-disk-2
2025-10-14 18:20:18 100-0: end replication job with error: command 'set -o pipefail && pvesm export local-zfs1:vm-100-disk-0 zfs - -with-snapshots 1 -snapshot __replicate_100-0_1760458808__ -base __replicate_100-0_1760389801__ | /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=pve2' -o 'UserKnownHostsFile=/etc/pve/nodes/pve2/ssh_known_hosts' -o 'GlobalKnownHostsFile=none' root@192.168.178.22 -- pvesm import local-zfs1:vm-100-disk-0 zfs - -with-snapshots 1 -snapshot __replicate_100-0_1760458808__ -allow-rename 0 -base __replicate_100-0_1760389801__' failed: exit code 255

Of course, I wanted to get to the bottom of this, but I can't find the source of the problem.

Here is the hardware of the PBS-VM:

Both replication targets have similar configuration, here is what I see on one of them (pve2).
Disks --> ZFS, there is a pool called “local-zfs1” (replication target) that is about half full:

However, when I go to the “local-zfs1” storage, the overview shows me that the storage is completely full:

I don't understand how this difference arises and whether that might be the problem.

I hope someone can help me with this.
Thanks a lot.

Deco20

gridiron · Oct 14, 2025

Can you post the results of zfs get all?

Deco20 · Oct 15, 2025

Hello,

here is the output of zfs get all . I have to cut it into pieces, because it's too long for one post.

Part 1/4:

Code:

root@pve2:~# zfs get all
NAME                                                     PROPERTY              VALUE                     SOURCE
local-zfs1                                               type                  filesystem                -
local-zfs1                                               creation              Fri Jul 11 10:39 2025     -
local-zfs1                                               used                  199G                      -
local-zfs1                                               available             470M                      -
local-zfs1                                               referenced            96K                       -
local-zfs1                                               compressratio         1.16x                     -
local-zfs1                                               mounted               yes                       -
local-zfs1                                               quota                 none                      default
local-zfs1                                               reservation           none                      default
local-zfs1                                               recordsize            128K                      default
local-zfs1                                               mountpoint            /local-zfs1               default
local-zfs1                                               sharenfs              off                       default
local-zfs1                                               checksum              on                        default
local-zfs1                                               compression           on                        local
local-zfs1                                               atime                 on                        default
local-zfs1                                               devices               on                        default
local-zfs1                                               exec                  on                        default
local-zfs1                                               setuid                on                        default
local-zfs1                                               readonly              off                       default
local-zfs1                                               zoned                 off                       default
local-zfs1                                               snapdir               hidden                    default
local-zfs1                                               aclmode               discard                   default
local-zfs1                                               aclinherit            restricted                default
local-zfs1                                               createtxg             1                         -
local-zfs1                                               canmount              on                        default
local-zfs1                                               xattr                 on                        default
local-zfs1                                               copies                1                         default
local-zfs1                                               version               5                         -
local-zfs1                                               utf8only              off                       -
local-zfs1                                               normalization         none                      -
local-zfs1                                               casesensitivity       sensitive                 -
local-zfs1                                               vscan                 off                       default
local-zfs1                                               nbmand                off                       default
local-zfs1                                               sharesmb              off                       default
local-zfs1                                               refquota              none                      default
local-zfs1                                               refreservation        none                      default
local-zfs1                                               guid                  7349834316013210724       -
local-zfs1                                               primarycache          all                       default
local-zfs1                                               secondarycache        all                       default
local-zfs1                                               usedbysnapshots       0B                        -
local-zfs1                                               usedbydataset         96K                       -
local-zfs1                                               usedbychildren        199G                      -
local-zfs1                                               usedbyrefreservation  0B                        -
local-zfs1                                               logbias               latency                   default
local-zfs1                                               objsetid              54                        -
local-zfs1                                               dedup                 off                       default
local-zfs1                                               mlslabel              none                      default
local-zfs1                                               sync                  standard                  default
local-zfs1                                               dnodesize             legacy                    default
local-zfs1                                               refcompressratio      1.00x                     -
local-zfs1                                               written               96K                       -
local-zfs1                                               logicalused           109G                      -
local-zfs1                                               logicalreferenced     42K                       -
local-zfs1                                               volmode               default                   default
local-zfs1                                               filesystem_limit      none                      default
local-zfs1                                               snapshot_limit        none                      default
local-zfs1                                               filesystem_count      none                      default
local-zfs1                                               snapshot_count        none                      default
local-zfs1                                               snapdev               hidden                    default
local-zfs1                                               acltype               off                       default
local-zfs1                                               context               none                      default
local-zfs1                                               fscontext             none                      default
local-zfs1                                               defcontext            none                      default
local-zfs1                                               rootcontext           none                      default
local-zfs1                                               relatime              on                        default
local-zfs1                                               redundant_metadata    all                       default
local-zfs1                                               overlay               on                        default
local-zfs1                                               encryption            off                       default
local-zfs1                                               keylocation           none                      default
local-zfs1                                               keyformat             none                      default
local-zfs1                                               pbkdf2iters           0                         default
local-zfs1                                               special_small_blocks  0                         default
local-zfs1                                               prefetch              all                       default
local-zfs1/vm-100-disk-0                                 type                  volume                    -
local-zfs1/vm-100-disk-0                                 creation              Mon Jul 28 16:38 2025     -
local-zfs1/vm-100-disk-0                                 used                  36.4G                     -
local-zfs1/vm-100-disk-0                                 available             20.8G                     -
local-zfs1/vm-100-disk-0                                 referenced            12.5G                     -
local-zfs1/vm-100-disk-0                                 compressratio         1.58x                     -
local-zfs1/vm-100-disk-0                                 reservation           none                      default
local-zfs1/vm-100-disk-0                                 volsize               20G                       local
local-zfs1/vm-100-disk-0                                 volblocksize          16K                       default
local-zfs1/vm-100-disk-0                                 checksum              on                        default
local-zfs1/vm-100-disk-0                                 compression           on                        inherited from local-zfs1
local-zfs1/vm-100-disk-0                                 readonly              off                       default
local-zfs1/vm-100-disk-0                                 createtxg             224619                    -
local-zfs1/vm-100-disk-0                                 copies                1                         default
local-zfs1/vm-100-disk-0                                 refreservation        20.3G                     received
local-zfs1/vm-100-disk-0                                 guid                  3799072871789392869       -
local-zfs1/vm-100-disk-0                                 primarycache          all                       default
local-zfs1/vm-100-disk-0                                 secondarycache        all                       default
local-zfs1/vm-100-disk-0                                 usedbysnapshots       3.53G                     -
local-zfs1/vm-100-disk-0                                 usedbydataset         12.5G                     -
local-zfs1/vm-100-disk-0                                 usedbychildren        0B                        -
local-zfs1/vm-100-disk-0                                 usedbyrefreservation  20.3G                     -
local-zfs1/vm-100-disk-0                                 logbias               latency                   default
local-zfs1/vm-100-disk-0                                 objsetid              548                       -
local-zfs1/vm-100-disk-0                                 dedup                 off                       default
local-zfs1/vm-100-disk-0                                 mlslabel              none                      default
local-zfs1/vm-100-disk-0                                 sync                  standard                  default
local-zfs1/vm-100-disk-0                                 refcompressratio      1.56x                     -
local-zfs1/vm-100-disk-0                                 written               0                         -
local-zfs1/vm-100-disk-0                                 logicalused           25.4G                     -
local-zfs1/vm-100-disk-0                                 logicalreferenced     19.5G                     -
local-zfs1/vm-100-disk-0                                 volmode               default                   default
local-zfs1/vm-100-disk-0                                 snapshot_limit        none                      default
local-zfs1/vm-100-disk-0                                 snapshot_count        none                      default
local-zfs1/vm-100-disk-0                                 snapdev               hidden                    default
local-zfs1/vm-100-disk-0                                 context               none                      default
local-zfs1/vm-100-disk-0                                 fscontext             none                      default
local-zfs1/vm-100-disk-0                                 defcontext            none                      default
local-zfs1/vm-100-disk-0                                 rootcontext           none                      default
local-zfs1/vm-100-disk-0                                 redundant_metadata    all                       default
local-zfs1/vm-100-disk-0                                 encryption            off                       default
local-zfs1/vm-100-disk-0                                 keylocation           none                      default
local-zfs1/vm-100-disk-0                                 keyformat             none                      default
local-zfs1/vm-100-disk-0                                 pbkdf2iters           0                         default
local-zfs1/vm-100-disk-0                                 snapshots_changed     Tue Oct 14  1:20:02 2025  -
local-zfs1/vm-100-disk-0                                 prefetch              all                       default
local-zfs1/vm-100-disk-0@__replicate_100-1_1760044205__  type                  snapshot                  -
local-zfs1/vm-100-disk-0@__replicate_100-1_1760044205__  creation              Thu Oct  9 23:10 2025     -
local-zfs1/vm-100-disk-0@__replicate_100-1_1760044205__  used                  3.53G                     -
local-zfs1/vm-100-disk-0@__replicate_100-1_1760044205__  referenced            12.7G                     -
local-zfs1/vm-100-disk-0@__replicate_100-1_1760044205__  compressratio         1.54x                     -
local-zfs1/vm-100-disk-0@__replicate_100-1_1760044205__  volsize               20G                       local
local-zfs1/vm-100-disk-0@__replicate_100-1_1760044205__  createtxg             1690686                   -
local-zfs1/vm-100-disk-0@__replicate_100-1_1760044205__  guid                  9857876554623719724       -
local-zfs1/vm-100-disk-0@__replicate_100-1_1760044205__  primarycache          all                       default
local-zfs1/vm-100-disk-0@__replicate_100-1_1760044205__  secondarycache        all                       default
local-zfs1/vm-100-disk-0@__replicate_100-1_1760044205__  defer_destroy         off                       -
local-zfs1/vm-100-disk-0@__replicate_100-1_1760044205__  userrefs              0                         -
local-zfs1/vm-100-disk-0@__replicate_100-1_1760044205__  objsetid              114254                    -
local-zfs1/vm-100-disk-0@__replicate_100-1_1760044205__  mlslabel              none                      default
local-zfs1/vm-100-disk-0@__replicate_100-1_1760044205__  refcompressratio      1.54x                     -
local-zfs1/vm-100-disk-0@__replicate_100-1_1760044205__  written               12.7G                     -
local-zfs1/vm-100-disk-0@__replicate_100-1_1760044205__  logicalreferenced     19.5G                     -
local-zfs1/vm-100-disk-0@__replicate_100-1_1760044205__  context               none                      default
local-zfs1/vm-100-disk-0@__replicate_100-1_1760044205__  fscontext             none                      default
local-zfs1/vm-100-disk-0@__replicate_100-1_1760044205__  defcontext            none                      default
local-zfs1/vm-100-disk-0@__replicate_100-1_1760044205__  rootcontext           none                      default
local-zfs1/vm-100-disk-0@__replicate_100-1_1760044205__  encryption            off                       default
local-zfs1/vm-100-disk-0@__replicate_100-1_1760044205__  prefetch              all                       default
local-zfs1/vm-100-disk-0@__replicate_100-0_1760389801__  type                  snapshot                  -
local-zfs1/vm-100-disk-0@__replicate_100-0_1760389801__  creation              Mon Oct 13 23:10 2025     -
local-zfs1/vm-100-disk-0@__replicate_100-0_1760389801__  used                  0B                        -

Deco20 · Oct 15, 2025

Part 2/4:

Code:

local-zfs1/vm-100-disk-0@__replicate_100-0_1760389801__  referenced            12.5G                     -
local-zfs1/vm-100-disk-0@__replicate_100-0_1760389801__  compressratio         1.56x                     -
local-zfs1/vm-100-disk-0@__replicate_100-0_1760389801__  volsize               20G                       local
local-zfs1/vm-100-disk-0@__replicate_100-0_1760389801__  createtxg             1765721                   -
local-zfs1/vm-100-disk-0@__replicate_100-0_1760389801__  guid                  13069191831976352812      -
local-zfs1/vm-100-disk-0@__replicate_100-0_1760389801__  primarycache          all                       default
local-zfs1/vm-100-disk-0@__replicate_100-0_1760389801__  secondarycache        all                       default
local-zfs1/vm-100-disk-0@__replicate_100-0_1760389801__  defer_destroy         off                       -
local-zfs1/vm-100-disk-0@__replicate_100-0_1760389801__  userrefs              0                         -
local-zfs1/vm-100-disk-0@__replicate_100-0_1760389801__  objsetid              30036                     -
local-zfs1/vm-100-disk-0@__replicate_100-0_1760389801__  mlslabel              none                      default
local-zfs1/vm-100-disk-0@__replicate_100-0_1760389801__  refcompressratio      1.56x                     -
local-zfs1/vm-100-disk-0@__replicate_100-0_1760389801__  written               3.36G                     -
local-zfs1/vm-100-disk-0@__replicate_100-0_1760389801__  logicalreferenced     19.5G                     -
local-zfs1/vm-100-disk-0@__replicate_100-0_1760389801__  context               none                      default
local-zfs1/vm-100-disk-0@__replicate_100-0_1760389801__  fscontext             none                      default
local-zfs1/vm-100-disk-0@__replicate_100-0_1760389801__  defcontext            none                      default
local-zfs1/vm-100-disk-0@__replicate_100-0_1760389801__  rootcontext           none                      default
local-zfs1/vm-100-disk-0@__replicate_100-0_1760389801__  encryption            off                       default
local-zfs1/vm-100-disk-0@__replicate_100-0_1760389801__  prefetch              all                       default
local-zfs1/vm-100-disk-1                                 type                  volume                    -
local-zfs1/vm-100-disk-1                                 creation              Mon Jul 28 16:41 2025     -
local-zfs1/vm-100-disk-1                                 used                  66.3G                     -
local-zfs1/vm-100-disk-1                                 available             33.0G                     -
local-zfs1/vm-100-disk-1                                 referenced            30.8G                     -
local-zfs1/vm-100-disk-1                                 compressratio         1.02x                     -
local-zfs1/vm-100-disk-1                                 reservation           none                      default
local-zfs1/vm-100-disk-1                                 volsize               32G                       local
local-zfs1/vm-100-disk-1                                 volblocksize          16K                       default
local-zfs1/vm-100-disk-1                                 checksum              on                        default
local-zfs1/vm-100-disk-1                                 compression           on                        inherited from local-zfs1
local-zfs1/vm-100-disk-1                                 readonly              off                       default
local-zfs1/vm-100-disk-1                                 createtxg             224657                    -
local-zfs1/vm-100-disk-1                                 copies                1                         default
local-zfs1/vm-100-disk-1                                 refreservation        32.5G                     received
local-zfs1/vm-100-disk-1                                 guid                  13801723570134263249      -
local-zfs1/vm-100-disk-1                                 primarycache          all                       default
local-zfs1/vm-100-disk-1                                 secondarycache        all                       default
local-zfs1/vm-100-disk-1                                 usedbysnapshots       3.03G                     -
local-zfs1/vm-100-disk-1                                 usedbydataset         30.8G                     -
local-zfs1/vm-100-disk-1                                 usedbychildren        0B                        -
local-zfs1/vm-100-disk-1                                 usedbyrefreservation  32.5G                     -
local-zfs1/vm-100-disk-1                                 logbias               latency                   default
local-zfs1/vm-100-disk-1                                 objsetid              97                        -
local-zfs1/vm-100-disk-1                                 dedup                 off                       default
local-zfs1/vm-100-disk-1                                 mlslabel              none                      default
local-zfs1/vm-100-disk-1                                 sync                  standard                  default
local-zfs1/vm-100-disk-1                                 refcompressratio      1.02x                     -
local-zfs1/vm-100-disk-1                                 written               0                         -
local-zfs1/vm-100-disk-1                                 logicalused           34.6G                     -
local-zfs1/vm-100-disk-1                                 logicalreferenced     31.5G                     -
local-zfs1/vm-100-disk-1                                 volmode               default                   default
local-zfs1/vm-100-disk-1                                 snapshot_limit        none                      default
local-zfs1/vm-100-disk-1                                 snapshot_count        none                      default
local-zfs1/vm-100-disk-1                                 snapdev               hidden                    default
local-zfs1/vm-100-disk-1                                 context               none                      default
local-zfs1/vm-100-disk-1                                 fscontext             none                      default
local-zfs1/vm-100-disk-1                                 defcontext            none                      default
local-zfs1/vm-100-disk-1                                 rootcontext           none                      default
local-zfs1/vm-100-disk-1                                 redundant_metadata    all                       default
local-zfs1/vm-100-disk-1                                 encryption            off                       default
local-zfs1/vm-100-disk-1                                 keylocation           none                      default
local-zfs1/vm-100-disk-1                                 keyformat             none                      default
local-zfs1/vm-100-disk-1                                 pbkdf2iters           0                         default
local-zfs1/vm-100-disk-1                                 snapshots_changed     Mon Oct 13 23:50:03 2025  -
local-zfs1/vm-100-disk-1                                 prefetch              all                       default
local-zfs1/vm-100-disk-1@__replicate_100-1_1760044205__  type                  snapshot                  -
local-zfs1/vm-100-disk-1@__replicate_100-1_1760044205__  creation              Thu Oct  9 23:10 2025     -
local-zfs1/vm-100-disk-1@__replicate_100-1_1760044205__  used                  3.03G                     -
local-zfs1/vm-100-disk-1@__replicate_100-1_1760044205__  referenced            30.7G                     -
local-zfs1/vm-100-disk-1@__replicate_100-1_1760044205__  compressratio         1.02x                     -
local-zfs1/vm-100-disk-1@__replicate_100-1_1760044205__  volsize               32G                       local
local-zfs1/vm-100-disk-1@__replicate_100-1_1760044205__  createtxg             1690691                   -
local-zfs1/vm-100-disk-1@__replicate_100-1_1760044205__  guid                  5568967649910646726       -
local-zfs1/vm-100-disk-1@__replicate_100-1_1760044205__  primarycache          all                       default
local-zfs1/vm-100-disk-1@__replicate_100-1_1760044205__  secondarycache        all                       default
local-zfs1/vm-100-disk-1@__replicate_100-1_1760044205__  defer_destroy         off                       -
local-zfs1/vm-100-disk-1@__replicate_100-1_1760044205__  userrefs              0                         -
local-zfs1/vm-100-disk-1@__replicate_100-1_1760044205__  objsetid              113760                    -
local-zfs1/vm-100-disk-1@__replicate_100-1_1760044205__  mlslabel              none                      default
local-zfs1/vm-100-disk-1@__replicate_100-1_1760044205__  refcompressratio      1.02x                     -
local-zfs1/vm-100-disk-1@__replicate_100-1_1760044205__  written               30.7G                     -
local-zfs1/vm-100-disk-1@__replicate_100-1_1760044205__  logicalreferenced     31.3G                     -
local-zfs1/vm-100-disk-1@__replicate_100-1_1760044205__  context               none                      default
local-zfs1/vm-100-disk-1@__replicate_100-1_1760044205__  fscontext             none                      default
local-zfs1/vm-100-disk-1@__replicate_100-1_1760044205__  defcontext            none                      default
local-zfs1/vm-100-disk-1@__replicate_100-1_1760044205__  rootcontext           none                      default
local-zfs1/vm-100-disk-1@__replicate_100-1_1760044205__  encryption            off                       default
local-zfs1/vm-100-disk-1@__replicate_100-1_1760044205__  prefetch              all                       default
local-zfs1/vm-100-disk-1@__replicate_100-0_1760389801__  type                  snapshot                  -
local-zfs1/vm-100-disk-1@__replicate_100-0_1760389801__  creation              Mon Oct 13 23:10 2025     -
local-zfs1/vm-100-disk-1@__replicate_100-0_1760389801__  used                  0B                        -
local-zfs1/vm-100-disk-1@__replicate_100-0_1760389801__  referenced            30.8G                     -
local-zfs1/vm-100-disk-1@__replicate_100-0_1760389801__  compressratio         1.02x                     -
local-zfs1/vm-100-disk-1@__replicate_100-0_1760389801__  volsize               32G                       local
local-zfs1/vm-100-disk-1@__replicate_100-0_1760389801__  createtxg             1765723                   -
local-zfs1/vm-100-disk-1@__replicate_100-0_1760389801__  guid                  6181428074804667181       -
local-zfs1/vm-100-disk-1@__replicate_100-0_1760389801__  primarycache          all                       default
local-zfs1/vm-100-disk-1@__replicate_100-0_1760389801__  secondarycache        all                       default
local-zfs1/vm-100-disk-1@__replicate_100-0_1760389801__  defer_destroy         off                       -
local-zfs1/vm-100-disk-1@__replicate_100-0_1760389801__  userrefs              0                         -
local-zfs1/vm-100-disk-1@__replicate_100-0_1760389801__  objsetid              29768                     -
local-zfs1/vm-100-disk-1@__replicate_100-0_1760389801__  mlslabel              none                      default
local-zfs1/vm-100-disk-1@__replicate_100-0_1760389801__  refcompressratio      1.02x                     -
local-zfs1/vm-100-disk-1@__replicate_100-0_1760389801__  written               3.06G                     -
local-zfs1/vm-100-disk-1@__replicate_100-0_1760389801__  logicalreferenced     31.5G                     -
local-zfs1/vm-100-disk-1@__replicate_100-0_1760389801__  context               none                      default
local-zfs1/vm-100-disk-1@__replicate_100-0_1760389801__  fscontext             none                      default
local-zfs1/vm-100-disk-1@__replicate_100-0_1760389801__  defcontext            none                      default
local-zfs1/vm-100-disk-1@__replicate_100-0_1760389801__  rootcontext           none                      default
local-zfs1/vm-100-disk-1@__replicate_100-0_1760389801__  encryption            off                       default
local-zfs1/vm-100-disk-1@__replicate_100-0_1760389801__  prefetch              all                       default
local-zfs1/vm-100-disk-2                                 type                  volume                    -
local-zfs1/vm-100-disk-2                                 creation              Mon Jul 28 16:41 2025     -
local-zfs1/vm-100-disk-2                                 used                  66.3G                     -
local-zfs1/vm-100-disk-2                                 available             33.0G                     -
local-zfs1/vm-100-disk-2                                 referenced            30.8G                     -
local-zfs1/vm-100-disk-2                                 compressratio         1.02x                     -
local-zfs1/vm-100-disk-2                                 reservation           none                      default
local-zfs1/vm-100-disk-2                                 volsize               32G                       local
local-zfs1/vm-100-disk-2                                 volblocksize          16K                       default
local-zfs1/vm-100-disk-2                                 checksum              on                        default
local-zfs1/vm-100-disk-2                                 compression           on                        inherited from local-zfs1
local-zfs1/vm-100-disk-2                                 readonly              off                       default
local-zfs1/vm-100-disk-2                                 createtxg             224668                    -
local-zfs1/vm-100-disk-2                                 copies                1                         default
local-zfs1/vm-100-disk-2                                 refreservation        32.5G                     received
local-zfs1/vm-100-disk-2                                 guid                  3882436471822493934       -
local-zfs1/vm-100-disk-2                                 primarycache          all                       default
local-zfs1/vm-100-disk-2                                 secondarycache        all                       default
local-zfs1/vm-100-disk-2                                 usedbysnapshots       3.03G                     -
local-zfs1/vm-100-disk-2                                 usedbydataset         30.8G                     -
local-zfs1/vm-100-disk-2                                 usedbychildren        0B                        -
local-zfs1/vm-100-disk-2                                 usedbyrefreservation  32.5G                     -
local-zfs1/vm-100-disk-2                                 logbias               latency                   default
local-zfs1/vm-100-disk-2                                 objsetid              657                       -
local-zfs1/vm-100-disk-2                                 dedup                 off                       default
local-zfs1/vm-100-disk-2                                 mlslabel              none                      default
local-zfs1/vm-100-disk-2                                 sync                  standard                  default
local-zfs1/vm-100-disk-2                                 refcompressratio      1.02x                     -
local-zfs1/vm-100-disk-2                                 written               0                         -
local-zfs1/vm-100-disk-2                                 logicalused           34.6G                     -
local-zfs1/vm-100-disk-2                                 logicalreferenced     31.5G                     -
local-zfs1/vm-100-disk-2                                 volmode               default                   default
local-zfs1/vm-100-disk-2                                 snapshot_limit        none                      default
local-zfs1/vm-100-disk-2                                 snapshot_count        none                      default
local-zfs1/vm-100-disk-2                                 snapdev               hidden                    default

Deco20 · Oct 15, 2025

Part 3/4:

Code:

local-zfs1/vm-100-disk-2                                 context               none                      default
local-zfs1/vm-100-disk-2                                 fscontext             none                      default
local-zfs1/vm-100-disk-2                                 defcontext            none                      default
local-zfs1/vm-100-disk-2                                 rootcontext           none                      default
local-zfs1/vm-100-disk-2                                 redundant_metadata    all                       default
local-zfs1/vm-100-disk-2                                 encryption            off                       default
local-zfs1/vm-100-disk-2                                 keylocation           none                      default
local-zfs1/vm-100-disk-2                                 keyformat             none                      default
local-zfs1/vm-100-disk-2                                 pbkdf2iters           0                         default
local-zfs1/vm-100-disk-2                                 snapshots_changed     Mon Oct 13 23:10:06 2025  -
local-zfs1/vm-100-disk-2                                 prefetch              all                       default
local-zfs1/vm-100-disk-2@__replicate_100-1_1760044205__  type                  snapshot                  -
local-zfs1/vm-100-disk-2@__replicate_100-1_1760044205__  creation              Thu Oct  9 23:10 2025     -
local-zfs1/vm-100-disk-2@__replicate_100-1_1760044205__  used                  3.03G                     -
local-zfs1/vm-100-disk-2@__replicate_100-1_1760044205__  referenced            30.7G                     -
local-zfs1/vm-100-disk-2@__replicate_100-1_1760044205__  compressratio         1.02x                     -
local-zfs1/vm-100-disk-2@__replicate_100-1_1760044205__  volsize               32G                       local
local-zfs1/vm-100-disk-2@__replicate_100-1_1760044205__  createtxg             1690700                   -
local-zfs1/vm-100-disk-2@__replicate_100-1_1760044205__  guid                  816492300571015962        -
local-zfs1/vm-100-disk-2@__replicate_100-1_1760044205__  primarycache          all                       default
local-zfs1/vm-100-disk-2@__replicate_100-1_1760044205__  secondarycache        all                       default
local-zfs1/vm-100-disk-2@__replicate_100-1_1760044205__  defer_destroy         off                       -
local-zfs1/vm-100-disk-2@__replicate_100-1_1760044205__  userrefs              0                         -
local-zfs1/vm-100-disk-2@__replicate_100-1_1760044205__  objsetid              114029                    -
local-zfs1/vm-100-disk-2@__replicate_100-1_1760044205__  mlslabel              none                      default
local-zfs1/vm-100-disk-2@__replicate_100-1_1760044205__  refcompressratio      1.02x                     -
local-zfs1/vm-100-disk-2@__replicate_100-1_1760044205__  written               30.7G                     -
local-zfs1/vm-100-disk-2@__replicate_100-1_1760044205__  logicalreferenced     31.3G                     -
local-zfs1/vm-100-disk-2@__replicate_100-1_1760044205__  context               none                      default
local-zfs1/vm-100-disk-2@__replicate_100-1_1760044205__  fscontext             none                      default
local-zfs1/vm-100-disk-2@__replicate_100-1_1760044205__  defcontext            none                      default
local-zfs1/vm-100-disk-2@__replicate_100-1_1760044205__  rootcontext           none                      default
local-zfs1/vm-100-disk-2@__replicate_100-1_1760044205__  encryption            off                       default
local-zfs1/vm-100-disk-2@__replicate_100-1_1760044205__  prefetch              all                       default
local-zfs1/vm-100-disk-2@__replicate_100-0_1760389801__  type                  snapshot                  -
local-zfs1/vm-100-disk-2@__replicate_100-0_1760389801__  creation              Mon Oct 13 23:10 2025     -
local-zfs1/vm-100-disk-2@__replicate_100-0_1760389801__  used                  0B                        -
local-zfs1/vm-100-disk-2@__replicate_100-0_1760389801__  referenced            30.8G                     -
local-zfs1/vm-100-disk-2@__replicate_100-0_1760389801__  compressratio         1.02x                     -
local-zfs1/vm-100-disk-2@__replicate_100-0_1760389801__  volsize               32G                       local
local-zfs1/vm-100-disk-2@__replicate_100-0_1760389801__  createtxg             1765725                   -
local-zfs1/vm-100-disk-2@__replicate_100-0_1760389801__  guid                  1198892948880385769       -
local-zfs1/vm-100-disk-2@__replicate_100-0_1760389801__  primarycache          all                       default
local-zfs1/vm-100-disk-2@__replicate_100-0_1760389801__  secondarycache        all                       default
local-zfs1/vm-100-disk-2@__replicate_100-0_1760389801__  defer_destroy         off                       -
local-zfs1/vm-100-disk-2@__replicate_100-0_1760389801__  userrefs              0                         -
local-zfs1/vm-100-disk-2@__replicate_100-0_1760389801__  objsetid              29781                     -
local-zfs1/vm-100-disk-2@__replicate_100-0_1760389801__  mlslabel              none                      default
local-zfs1/vm-100-disk-2@__replicate_100-0_1760389801__  refcompressratio      1.02x                     -
local-zfs1/vm-100-disk-2@__replicate_100-0_1760389801__  written               3.06G                     -
local-zfs1/vm-100-disk-2@__replicate_100-0_1760389801__  logicalreferenced     31.5G                     -
local-zfs1/vm-100-disk-2@__replicate_100-0_1760389801__  context               none                      default
local-zfs1/vm-100-disk-2@__replicate_100-0_1760389801__  fscontext             none                      default
local-zfs1/vm-100-disk-2@__replicate_100-0_1760389801__  defcontext            none                      default
local-zfs1/vm-100-disk-2@__replicate_100-0_1760389801__  rootcontext           none                      default
local-zfs1/vm-100-disk-2@__replicate_100-0_1760389801__  encryption            off                       default
local-zfs1/vm-100-disk-2@__replicate_100-0_1760389801__  prefetch              all                       default
local-zfs1/vm-130-disk-0                                 type                  volume                    -
local-zfs1/vm-130-disk-0                                 creation              Fri Aug 22 13:27 2025     -
local-zfs1/vm-130-disk-0                                 used                  30.1G                     -
local-zfs1/vm-130-disk-0                                 available             20.8G                     -
local-zfs1/vm-130-disk-0                                 referenced            9.76G                     -
local-zfs1/vm-130-disk-0                                 compressratio         1.44x                     -
local-zfs1/vm-130-disk-0                                 reservation           none                      default
local-zfs1/vm-130-disk-0                                 volsize               20G                       local
local-zfs1/vm-130-disk-0                                 volblocksize          16K                       default
local-zfs1/vm-130-disk-0                                 checksum              on                        default
local-zfs1/vm-130-disk-0                                 compression           on                        inherited from local-zfs1
local-zfs1/vm-130-disk-0                                 readonly              off                       default
local-zfs1/vm-130-disk-0                                 createtxg             712172                    -
local-zfs1/vm-130-disk-0                                 copies                1                         default
local-zfs1/vm-130-disk-0                                 refreservation        20.3G                     received
local-zfs1/vm-130-disk-0                                 guid                  12909084920988674074      -
local-zfs1/vm-130-disk-0                                 primarycache          all                       default
local-zfs1/vm-130-disk-0                                 secondarycache        all                       default
local-zfs1/vm-130-disk-0                                 usedbysnapshots       7.98M                     -
local-zfs1/vm-130-disk-0                                 usedbydataset         9.76G                     -
local-zfs1/vm-130-disk-0                                 usedbychildren        0B                        -
local-zfs1/vm-130-disk-0                                 usedbyrefreservation  20.3G                     -
local-zfs1/vm-130-disk-0                                 logbias               latency                   default
local-zfs1/vm-130-disk-0                                 objsetid              15616                     -
local-zfs1/vm-130-disk-0                                 dedup                 off                       default
local-zfs1/vm-130-disk-0                                 mlslabel              none                      default
local-zfs1/vm-130-disk-0                                 sync                  standard                  default
local-zfs1/vm-130-disk-0                                 refcompressratio      1.44x                     -
local-zfs1/vm-130-disk-0                                 written               0                         -
local-zfs1/vm-130-disk-0                                 logicalused           14.1G                     -
local-zfs1/vm-130-disk-0                                 logicalreferenced     14.1G                     -
local-zfs1/vm-130-disk-0                                 volmode               default                   default
local-zfs1/vm-130-disk-0                                 snapshot_limit        none                      default
local-zfs1/vm-130-disk-0                                 snapshot_count        none                      default
local-zfs1/vm-130-disk-0                                 snapdev               hidden                    default
local-zfs1/vm-130-disk-0                                 context               none                      default
local-zfs1/vm-130-disk-0                                 fscontext             none                      default
local-zfs1/vm-130-disk-0                                 defcontext            none                      default
local-zfs1/vm-130-disk-0                                 rootcontext           none                      default
local-zfs1/vm-130-disk-0                                 redundant_metadata    all                       default
local-zfs1/vm-130-disk-0                                 encryption            off                       default
local-zfs1/vm-130-disk-0                                 keylocation           none                      default
local-zfs1/vm-130-disk-0                                 keyformat             none                      default
local-zfs1/vm-130-disk-0                                 pbkdf2iters           0                         default
local-zfs1/vm-130-disk-0                                 snapshots_changed     Wed Oct 15  8:40:03 2025  -
local-zfs1/vm-130-disk-0                                 prefetch              all                       default

Deco20 · Oct 15, 2025

Part 4/4:

Code:

local-zfs1/vm-130-disk-0@__replicate_130-1_1760509803__  type                  snapshot                  -
local-zfs1/vm-130-disk-0@__replicate_130-1_1760509803__  creation              Wed Oct 15  8:30 2025     -
local-zfs1/vm-130-disk-0@__replicate_130-1_1760509803__  used                  7.98M                     -
local-zfs1/vm-130-disk-0@__replicate_130-1_1760509803__  referenced            9.76G                     -
local-zfs1/vm-130-disk-0@__replicate_130-1_1760509803__  compressratio         1.44x                     -
local-zfs1/vm-130-disk-0@__replicate_130-1_1760509803__  volsize               20G                       local
local-zfs1/vm-130-disk-0@__replicate_130-1_1760509803__  createtxg             1791195                   -
local-zfs1/vm-130-disk-0@__replicate_130-1_1760509803__  guid                  15389576321876098090      -
local-zfs1/vm-130-disk-0@__replicate_130-1_1760509803__  primarycache          all                       default
local-zfs1/vm-130-disk-0@__replicate_130-1_1760509803__  secondarycache        all                       default
local-zfs1/vm-130-disk-0@__replicate_130-1_1760509803__  defer_destroy         off                       -
local-zfs1/vm-130-disk-0@__replicate_130-1_1760509803__  userrefs              0                         -
local-zfs1/vm-130-disk-0@__replicate_130-1_1760509803__  objsetid              39318                     -
local-zfs1/vm-130-disk-0@__replicate_130-1_1760509803__  mlslabel              none                      default
local-zfs1/vm-130-disk-0@__replicate_130-1_1760509803__  refcompressratio      1.44x                     -
local-zfs1/vm-130-disk-0@__replicate_130-1_1760509803__  written               9.76G                     -
local-zfs1/vm-130-disk-0@__replicate_130-1_1760509803__  logicalreferenced     14.1G                     -
local-zfs1/vm-130-disk-0@__replicate_130-1_1760509803__  context               none                      default
local-zfs1/vm-130-disk-0@__replicate_130-1_1760509803__  fscontext             none                      default
local-zfs1/vm-130-disk-0@__replicate_130-1_1760509803__  defcontext            none                      default
local-zfs1/vm-130-disk-0@__replicate_130-1_1760509803__  rootcontext           none                      default
local-zfs1/vm-130-disk-0@__replicate_130-1_1760509803__  encryption            off                       default
local-zfs1/vm-130-disk-0@__replicate_130-1_1760509803__  prefetch              all                       default
local-zfs1/vm-130-disk-0@__replicate_130-0_1760510400__  type                  snapshot                  -
local-zfs1/vm-130-disk-0@__replicate_130-0_1760510400__  creation              Wed Oct 15  8:40 2025     -
local-zfs1/vm-130-disk-0@__replicate_130-0_1760510400__  used                  0B                        -
local-zfs1/vm-130-disk-0@__replicate_130-0_1760510400__  referenced            9.76G                     -
local-zfs1/vm-130-disk-0@__replicate_130-0_1760510400__  compressratio         1.44x                     -
local-zfs1/vm-130-disk-0@__replicate_130-0_1760510400__  volsize               20G                       local
local-zfs1/vm-130-disk-0@__replicate_130-0_1760510400__  createtxg             1791197                   -
local-zfs1/vm-130-disk-0@__replicate_130-0_1760510400__  guid                  9977857332145710569       -
local-zfs1/vm-130-disk-0@__replicate_130-0_1760510400__  primarycache          all                       default
local-zfs1/vm-130-disk-0@__replicate_130-0_1760510400__  secondarycache        all                       default
local-zfs1/vm-130-disk-0@__replicate_130-0_1760510400__  defer_destroy         off                       -
local-zfs1/vm-130-disk-0@__replicate_130-0_1760510400__  userrefs              0                         -
local-zfs1/vm-130-disk-0@__replicate_130-0_1760510400__  objsetid              38628                     -
local-zfs1/vm-130-disk-0@__replicate_130-0_1760510400__  mlslabel              none                      default
local-zfs1/vm-130-disk-0@__replicate_130-0_1760510400__  refcompressratio      1.44x                     -
local-zfs1/vm-130-disk-0@__replicate_130-0_1760510400__  written               8.02M                     -
local-zfs1/vm-130-disk-0@__replicate_130-0_1760510400__  logicalreferenced     14.1G                     -
local-zfs1/vm-130-disk-0@__replicate_130-0_1760510400__  context               none                      default
local-zfs1/vm-130-disk-0@__replicate_130-0_1760510400__  fscontext             none                      default
local-zfs1/vm-130-disk-0@__replicate_130-0_1760510400__  defcontext            none                      default
local-zfs1/vm-130-disk-0@__replicate_130-0_1760510400__  rootcontext           none                      default
local-zfs1/vm-130-disk-0@__replicate_130-0_1760510400__  encryption            off                       default
local-zfs1/vm-130-disk-0@__replicate_130-0_1760510400__  prefetch              all                       default

gridiron · Oct 15, 2025

Your local-zfs1 pool is full:

local-zfs1 used 199G -
local-zfs1 available 470M -

I'm not sure why the Disks -> ZFS UI shows 120 GB free, though. Maybe it is not accounting for the usage of snapshots or something like that. Someone a bit more knowledgeable than I about the nuances there will have to comment.

Deco20 · Oct 15, 2025

But I don't know why it's full. Because I've there only these disks:

vm-100 is the PBS, disk-0 for OS, disk1 & disk2 are a zfs pool. Here a picture from the pbs (vm100):

So all of this looks (in my opinion) like about 100 GB at all.
I can't understand from where the other 100 GB should come...

Greetings from Germany
Deco20

gridiron · Oct 15, 2025

Can you post the output of this command?

zfs list -r -o name,used,avail,refer,usedbysnapshots,usedbychildren,usedbyrefreservation local-zfs1

Deco20 · Oct 15, 2025

Yes, here it is:

Code:

root@pve2:~# zfs list -r -o name,used,avail,refer,usedbysnapshots,usedbychildren,usedbyrefreservation local-zfs1
NAME                       USED  AVAIL  REFER  USEDSNAP  USEDCHILD  USEDREFRESERV
local-zfs1                 199G   475M    96K        0B       199G             0B
local-zfs1/vm-100-disk-0  36.4G  20.8G  12.5G     3.53G         0B          20.3G
local-zfs1/vm-100-disk-1  66.3G  33.0G  30.8G     3.03G         0B          32.5G
local-zfs1/vm-100-disk-2  66.3G  33.0G  30.8G     3.03G         0B          32.5G
local-zfs1/vm-130-disk-0  30.1G  20.8G  9.77G        0B         0B          20.3G

Interesting. Here we can see, that the disks are 66,3 G instead of 32G.
For me even more confusing for me.

gridiron · Oct 15, 2025

I suspect the discrepancy you see is related to the refreservation (reserved space for each zvol). It isn't used, so the PVE UI shows ~120 GB as "free," but is reserved and therefore you get an error when you try to run replication. That is my guess but again it would be good to have someone else chime in here.

Deco20 · Oct 19, 2025

Does anyone else have any thoughts on this?

janus57 · Oct 19, 2025

Hi,

What is your original storage and disk size from this replicated VMs ?

What you’re seeing seems to be more or less the same from : https://forum.proxmox.com/threads/replication-needs-so-much-additional-storage-space.111164/

Best regards,

Deco20 · Oct 19, 2025

janus57 said:
What is your original storage and disk size from this replicated VMs ?

Hello, thanks for xour Reply.
I'm not sure what exactly you mean.

The vm100 (pbs) has 3 disks (like the 2nd screenshot on my first post):
disk0 = 20 G (for pbs itself)
disk1 = 32 G (for an Datastore (with zfs) on pbs), mirrored with disk2
disk2 = 32 G (for an Datastore (with zfs) on pbs), mirrored with disk1

All this disks are thin provisioned on a zfs storage.
The vm is (currently) on my node1.
The partition of phsiscal disk, where the zfs lay on is 465 G.

On all other nodes is also a zfs on a partition of the phsiscal disk.
This partition is in node2 and node3 220 G.

Is that what you wanted to know?

janus57 said:
What you’re seeing seems to be more or less the same from : https://forum.proxmox.com/threads/replication-needs-so-much-additional-storage-space.111164/

Thanks a lot for the linked topic.
I read it and found it quite interesting to understand replication. However, my volumes are all thin provisioned (discard=on), so I don't think it applies entirely. Or am I mistaken?

Best regards

janus57 · Oct 19, 2025

Hi,

Can you share the content of your "/etc/pve/storage.cfg" ?

Best regards,

Deco20 · Oct 19, 2025

Sure, here it is:

Code:

root@pve1:~# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content vztmpl,backup,iso

lvmthin: local-lvm
        thinpool data
        vgname pve
        content images,rootdir

zfspool: local-zfs1
        pool local-zfs1
        content images,rootdir
        mountpoint /local-zfs1
        nodes pve1,pve3,pve2

dir: local-directory1
        path /mnt/pve/local-directory1
        content images,vztmpl,backup,rootdir,iso,snippets
        is_mountpoint 1
        nodes pve1

pbs: pbs1-datastore-local-zfs1-pbs1
        datastore local-zfs1-pbs1
        server 192.***
        content backup
        fingerprint ***
        prune-backups keep-all=1
        username BackupUser@pbs

pbs: pbs1-datastore-external-nfs1-nas1
        datastore external-nfs1-nas1
        server 192.***
        content backup
        fingerprint ***
        prune-backups keep-all=1
        username BackupUser@pbs

nfs: external-nfs1-nas1-vzdump
        export /volume1/Proxmox_Backups/pve/vzdump
        path /mnt/pve/external-nfs1-nas1-vzdump
        server 192.***
        content backup
        prune-backups keep-all=1

pbs: pbs1-datastore-local-zfs1-pbs1-namespaceDAILY
        datastore local-zfs1-pbs1
        server 192.***
        content backup
        fingerprint ***
        namespace daily
        prune-backups keep-all=1
        username root@pam

pbs: pbs1-datastore-external-nfs1-nas1-namespaceDAILY
        datastore external-nfs1-nas1
        server 192.***
        content backup
        fingerprint ***
        namespace daily
        prune-backups keep-all=1
        username root@pam

root@pve1:~#

janus57 · Oct 19, 2025

Hi,

Hm, maybe I'm blind but your "local-zfs1" is declared as thick provisioned (don't see the "sparse" flag on your ZFS pool, see : https://pve.proxmox.com/pve-docs/chapter-pvesm.html#storage_zfspool).

So the discard at the VM level will do nothing if I'm not wrong.

Best regards,

Deco20 · Oct 19, 2025

Ahh ok, I understand.
Yes, it is correct that the zfs pool (in datacenter) is thick provisioned.
I thought that the vm disks would still be thin provisioned based on the settings when creating the disks in the hardware options.

So if I see it right, I have to use

zfs set refreservation=none local-zfs1
- if I want to thin provisioning the whole pool
or zfs set refreservation=none local-zfs1/vm-100-disk-1 and zfs set refreservation=none local-zfs1/vm-100-disk-2
- if I only want to thin provisioning this two disk from my pbs storage

Is this correct?

janus57 · Oct 19, 2025

Hi,

Well, I think you should activate the thin provisioning at proxmox storage level first.

After that I don't really know what to tell you, but be sure to have the proxmox part setup first before doing any command on the ZFS part.

If it's just a replication storage, personally I would delete, make sure the proxmox configuration is OK, then do the replication from zero and monitor from then.

Best regards,

ZFS Replication Error: out of space

New Member

Member

New Member

New Member

New Member

New Member

Member

New Member

Member

New Member

Member

New Member

Renowned Member

New Member

Renowned Member

New Member

Renowned Member

New Member

Renowned Member

We value your privacy