Replication Failure

[Ugh_noob]

I'm trying to setup replication between two nodes to have a copy on each node as well as backups in PBS and TrueNAS. I have setup PBS and will do TruNAS later.

I currently have both nodes setup with ZFS pools though they do have vastly different configurations. Each node has it own local ZFS and vm share [node#-vms]. These are created from their respective ZFS pools named vm-storage#. I have also added the respective storage's and pools to Datacenter so that they can be seen/recognized by the cluster. When adding vm-storage# and node#-vms I assigned them to there respective node that they are installed on.

Is there something I'm not understanding about replication or do I need to have completely identical nodes and configurations. For now im shutting everything down and going to eat ice cream my head hurts .

Datacenter Storage List:


Storage Config:

Node 1:

Node 2:


Log Output:
2025-04-27 14:18:00 100-0: start replication job
2025-04-27 14:18:00 100-0: guest => VM 100, running => 0
2025-04-27 14:18:00 100-0: volumes => vm-storage2:vm-100-disk-0
2025-04-27 14:18:01 100-0: (remote_prepare_local_job) storage 'vm-storage2' is not available on node 'pve1'
2025-04-27 14:18:01 100-0: end replication job with error: command '/usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=pve1' -o 'UserKnownHostsFile=/etc/pve/nodes/pve1/ssh_known_hosts' -o 'GlobalKnownHostsFile=none' root@192.168.88.20 -- pvesr prepare-local-job 100-0 vm-storage2:vm-100-disk-0 --last_sync 0' failed: exit code 255

 

[fiona]

Hi,
you can have a single storage definition in the datacenter storage configuration for a local storage for multiple nodes, e.g.

zfspool: mypool
    pool mypool
    content images,rootdir
    mountpoint /mypool
    nodes pve8a3,pve8a1,pve8a2

Each node needs an actual local instance of this storage with the same pool name/mountpoint. Replication requires that the used storages are available on each of source and target node.

 

[Ugh_noob]

Hi,
you can have a single storage definition in the datacenter storage configuration for a local storage for multiple nodes, e.g.

zfspool: mypool
    pool mypool
    content images,rootdir
    mountpoint /mypool
    nodes pve8a3,pve8a1,pve8a2

Each node needs an actual local instance of this storage with the same pool name/mountpoint. Replication requires that the used storages are available on each of source and target node.

Thank you,
I will try later when I can.

[still new to Prox]
So it’s creating a pool that references a partition or disk that would have the same name and “file hierarchy”.

[minor thoughts]
So in my case I likely have to add another disk to each node. Or is it possible to reduce a zfs pool from 3 to 2 disks [though not ideal]
Then use that removed disk to create a pool that will be used for replication?

 

[fiona]

[still new to Prox]
So it’s creating a pool that references a partition or disk that would have the same name and “file hierarchy”.

The storage configuration is just configuration, it doesn't actually create the underlying storage. For existing storages, you will need to make sure it matches reality.

That said, you can create new ZFS pools including an associated storage via [your node] > Disks > ZFS > Add.

[minor thoughts]
So in my case I likely have to add another disk to each node. Or is it possible to reduce a zfs pool from 3 to 2 disks [though not ideal]
Then use that removed disk to create a pool that will be used for replication?

You could also import the pool with the same name on both nodes. Then add it as a new storage via Datacenter > Storage > Add > ZFS and remove the old storage configuration later. But you will need to adapt all the guest configurations manually so that the volumes use the new storage name/ID. I'd highly suggest creating backups of all guests, before attempting this.