[SOLVED] ZFS Storage replication network

[mbaldini]

Hello.

In my setup I have a 3 nodes cluster, in each node two network adapters are used as bridge for VMs and containers (vmbr0 and vmbr1) and one network adapter is dedicated to cluster traffic (on a dedicated LAN and switch, as per https://pve.proxmox.com/wiki/Separate_Cluster_Network )

If I use ZFS Storage replication (pvesr) the replication traffic (zfs sed/receive) is using the LAN where vmbr0 is configurated, that is 192.168.2.x in accordance to my hosts file

pve-hs-main[0]:/$ cat /etc/hosts
127.0.0.1 localhost.localdomain localhost

192.168.2.251 pve-hs-main.local     pve-hs-main     pvelocalhost
192.168.2.252 pve-hs-replica.local  pve-hs-replica
192.168.2.253 pve-hs-3.local        pve-hs-3


10.10.10.251 cluster-main.local      cluster-main
10.10.10.252 cluster-replica.local   cluster-replica
10.10.10.253 cluster-3.local         cluster-3

In fact during a replication I see:

pve-hs-main[0]:/$ ps aux | grep zfs

root     13302  0.0  0.0  17948  2680 ?        S    13:05   0:00 /bin/bash -c set -o pipefail && pvesm export local-zfs:vm-101-disk-2 zfs - -with-snapshots 1 -snapshot __replicate_101-0_1506683155__ | /usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=pve-hs-3' root@192.168.2.253 -- pvesm import local-zfs:vm-101-disk-2 zfs - -with-snapshots 1
root     13303  0.0  0.2 294836 63136 ?        S    13:05   0:00 /usr/bin/perl /usr/sbin/pvesm export local-zfs:vm-101-disk-2 zfs - -with-snapshots 1 -snapshot __replicate_101-0_1506683155__
root     13304  2.2  0.0  47956  7736 ?        S    13:05   3:59 /usr/bin/ssh -o BatchMode=yes -o HostKeyAlias=pve-hs-3 root@192.168.2.253 -- pvesm import local-zfs:vm-101-disk-2 zfs - -with-snapshots 1
root     13331  1.1  0.0 107328  2900 ?        Sl   13:05   2:02 zfs send -Rpv -- rpool/data/vm-101-disk-2 __replicate_101-0_1506683155__

And I see the connection betweend nodes of the cluster is on IP 192.168.2.253 (third node pve-hs-3)


This is generating a lot of traffic on LAN used by VMs to communicate with clients, instead the replication could happen on the cluster dedicated LAN. But I can't find a way to change the IP that pvesr is using.

Is there some way to configure this?

Thanks a lot

 

[fabian]

replication uses the migration network configured in datacenter.cfg:

https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_guest_migration

 

[mbaldini]

Great news, thanks for the info!

 

[Mogli]

Sorry to hijack the thread, but I think it's closely related.

I'm currently planning on setting up a small 3-node cluster in my homelab, but the 3rd mainly as quorum helper if one of my compute nodes goes down (just an old Atom Box with single LAN interface). I wanted to setup a Point-to-Point connection as storage network between the two compute nodes and configure that as migration network. What happens in case of migration to the 3rd node outside of that network?

 

[fabian]

Sorry to hijack the thread, but I think it's closely related.

Im currently planning on setting up a small 3-node cluster in my homelab, but the 3rd mainly as quorum helper if one of my compute nodes goes down (just an old Atom Box with single LAN interface). I wanted to setup a Point-to-Point connection as storage network between the two compute nodes and configure that as migration network. What happens in case of migration to the 3rd node outside of that network?

unless you override the migration network using the command line parameter "migration_network", migration will fail.