Failed replication Cluster

[pvpaulo]

Hi, I have a cluster proxmox
I want confiugrate replication between VM's .
I have 2 Vm' with replication working but 1 VM's with 8TB not working replication and this error appears:

2025-07-22 17:18:07 100-0: start replication job
2025-07-22 17:18:10 100-0: guest => VM 100, running => 1736687
2025-07-22 17:18:10 100-0: volumes => VMs:vm-100-disk-0,VMs:vm-100-disk-1
2025-07-22 17:18:13 100-0: create snapshot '__replicate_100-0_1753215487__' on VMs:vm-100-disk-0
2025-07-22 17:18:13 100-0: end replication job with error: zfs error: cannot create snapshot 'VMs/vm-100-disk-0@__replicate_100-0_1753215487__': out of space

Source:
#zfs list
NAME USED AVAIL REFER MOUNTPOINT
VMs 9.42T 4.99T 96K /VMs
VMs/vm-100-disk-0 8.06T 6.56T 6.49T -
VMs/vm-100-disk-1 41.3G 5.03T 2.95G -
VMs/vm-101-disk-0 1.23T 6.00T 226G -
VMs/vm-102-disk-0 92.3G 5.05T 30.4G -


Destination:
#zfs list
NAME USED AVAIL REFER MOUNTPOINT
VMs 1.32T 13.1T 96K /VMs
VMs/vm-101-disk-0 1.23T 14.1T 226G -
VMs/vm-102-disk-0 92.3G 13.1T 30.4G -

Could you help me?

 

[LnxBil]

out of space

That error is pretty clear ... you don't have enough space on the target available.

Please post the output of zpool status -v on both nodes and please in CODE tags.

 

[pvpaulo]

Source :


zpool status -v
pool: VMs
state: ONLINE
scan: scrub repaired 0B in 14:39:02 with 0 errors on Sun Jul 13 15:03:03 2025
config:

NAME STATE READ WRITE CKSUM
VMs ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
scsi-36842b2b045dde9002feebf750df3dbc1 ONLINE 0 0 0
scsi-36842b2b045dde9002feebfa11096777c ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
scsi-36842b2b045dde9002feebfb411b21fa3 ONLINE 0 0 0
scsi-36842b2b045dde9002feebfc3129db9e4 ONLINE 0 0 0

errors: No known data errors
root@pve7:~#

Destination:


zpool status -v
pool: VMs
state: ONLINE
config:

NAME STATE READ WRITE CKSUM
VMs ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
scsi-36842b2b0772cbf00300815d8109d5e1d ONLINE 0 0 0
scsi-36842b2b0772cbf00300815e611678c29 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
scsi-36842b2b0772cbf00300815f51251e3c5 ONLINE 0 0 0
scsi-36842b2b0772cbf00300816011304e056 ONLINE 0 0 0

errors: No known data errors
root@pve8:~#

 

[LnxBil]

Thank you for using CODE tags.

 

[pvpaulo]

Any suggestions?

 

[VictorSTS]

Literally first search result on Google for "pve zfs replication cannot create snapshot out of space":

Post in thread 'Replication error "out of space"'

Jan 17, 2022

likely your volume is thick-provisioned (has a refreservation set - you can check with zfs get all vmstore/vm-100-disk-0).

in this case, creating a snapshot requires at least as much free space as the volume currently uses:

- a thick provisioned volume has it's full size reserved (to make sure you can always write the full volume)
- creating a snapshot means the currently referenced data lives as long as that snapshot exists
- to ensure you can still fully (over)write the volume, the total amount of space reserved needs to grow by the data referenced by the snapshot (== currently...

• fabian

Please, make the effort to use CODE tags and format you post properly.


Not using them makes posts very hard to read and denotes little effort on your side compared to the amount of expertise required to give proper answers

 

[pvpaulo]

Good afternoon,
I have a Proxmox cluster with 2 nodes.
Each node has a ZFS pool named VMs, and each pool has 4 disks of 8TB, totaling 16TB (with around 8TB usable due to RAID 10).
I have scheduled replication, but there is no space left in the target node’s pool to replicate a VM with 8TB virtual disk size.


The error I get is:
zfs error: cannot create snapshot VMs/vm-100-disk0@_replicate_100-0_ out of space

---

My questions:

1. I want to add 2 more 8TB disks to each pool. What are the best practices to expand a ZFS pool with RAID 10 (mirrored vdevs)?
2. Are there any risks or issues with this process?
3. What is the recommended step-by-step procedure?

 

[VictorSTS]

I want to add 2 more 8TB disks to each pool. What are the best practices to expand a ZFS pool with RAID 10 (mirrored vdevs)?

Create a mirror vdev and add it to the current RAID10 zpool, which will have 3 mirror vdevs instead of the current 2 mirror vdevs. Capacity will increase in ~8TB.

Are there any risks or issues with this process?

No data will be moved to the new disks, so most of your I/O will still hit your current 4 disks and at least initially there won't be a major benefit in iops capacity. Over time data will be stored in the new mirror potentially improving total iops.

What is the recommended step-by-step procedure?

Besides adding the new mirror, given that you use replication, I would suggest that once the zpool has been expanded with the new mirror vdev, you remove replication of VMs and add it again, one by one. The new replication will use all drives, thus increasing the iops capacity of the zpool.

Really suggest that you setup a testing cluster using nested PVE VMs and practice this procedure before conducting this operation in production.

 

[pvpaulo]

Thank you , I will approve.