[SOLVED] Replication in cluster - different ZFS storage on nodes - not available on node

[zaphyre]

I have extended a test cluster with a new machine (called: pmx1885) which has its ZFS pool setup done after the install with the web-gui not with the installer (during installation time). This differs from my other cluster nodes in this case. The other cluster nodes (pmx2b,pmx1b,pmx2a,pmx1a) have a local-zfs Storage "rpool" (with one exception, pmx201, also having a separate storage layout).

The storage is marked with a question mark in the tree view. So I edited storage.cfg as follows and restricted the storage:

root@pmx1885:~# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content backup,vztmpl,iso

zfspool: local-zfs
        pool rpool/data
        content rootdir,images
        nodes pmx2b,pmx1b,pmx2a,pmx1a
        sparse 1

zfspool: local-nvmes-zfs
        pool nvmes
        content rootdir,images
        nodes pmx201
        sparse 1

cifs: dc01.xxx.xxx
        path /mnt/pve/dc01.xxx.xxx
        server xxx.xxx.1.209
        share xxx
        content backup,vztmpl,iso,images
        prune-backups keep-daily=4,keep-last=3,keep-monthly=3,keep-yearly=1
        username xxx

pbs: pmx1d-PBS
        datastore rpool
        server xxx.xxx.1.140
        content backup
        fingerprint xx:c1:xx:7b:xx:xx:cd:a2:f0:xx
        prune-backups keep-all=1
        username backup@pbs

zfspool: rpool
        pool rpool
        content rootdir,images
        mountpoint /rpool
        nodes pmx1885

but when trying to enable replication for a VM to the respective node it fails with:

2022-01-13 17:34:01 101-0: (remote_prepare_local_job) storage 'local-zfs' is not available on node 'pmx1885'

but there is no way of setting the storage to replicate to?

So, question:

• How do I create the exact same ZFS pool setup zfs-local/"rpool" on this new node? There is now need to have another name and I was not able to create it so that PVE mounts it as zfs-local like on the other nodes? (like the installer did)
• Despite restricting the storage, or better, after restricting the storage as above shouldnt the replication work?

As allwas still a bit lost with ZFS....
Thanks for any help

 

[fiona]

Hi,

• How do I create the exact same ZFS pool setup zfs-local/"rpool" on this new node? There is now need to have another name and I was not able to create it so that PVE mounts it as zfs-local like on the other nodes? (like the installer did)

If you already have a Pool named rpool, simply using zfs create rpool/data, and then adding pmx1885 to the local-zfs nodes should work.

• Despite restricting the storage, or better, after restricting the storage as above shouldnt the replication work?

Replication (currently) requires the same storage on both source and target (for each replicated volume).

 

[zaphyre]

Hi,

If you already have a Pool named rpool, simply using zfs create rpool/data, and then adding pmx1885 to the local-zfs nodes should work.

Replication (currently) requires the same storage on both source and target (for each replicated volume).

Hi Fabian! Thanks a lot for your help and clarification! Didn't know that restriction regarding storage for replication yet. Then, the error makes total sense.

Will try to zfs create rpool/data on this machine and then adding it to the local-zfs again to solve this.

Thanks

 

[zaphyre]

That worked just fine! Thanks a ton!

After zfs create rpool/data and editing

root@pmx1885:~# cat /etc/pve/storage.cfg
zfspool: local-zfs
        pool rpool/data
        content images,rootdir
        nodes pmx1b,pmx2b,pmx2a,pmx1a,pmx1885
        sparse 1

adding pmx1885 the storage named "zfs-local" is no longer marked with a "?" and the replication works like a charm!