[SOLVED] ZFS Storage replication network

mbaldini

Well-Known Member
Nov 7, 2015
170
22
58
Hello.

In my setup I have a 3 nodes cluster, in each node two network adapters are used as bridge for VMs and containers (vmbr0 and vmbr1) and one network adapter is dedicated to cluster traffic (on a dedicated LAN and switch, as per https://pve.proxmox.com/wiki/Separate_Cluster_Network )

If I use ZFS Storage replication (pvesr) the replication traffic (zfs sed/receive) is using the LAN where vmbr0 is configurated, that is 192.168.2.x in accordance to my hosts file

Code:
pve-hs-main[0]:/$ cat /etc/hosts
127.0.0.1 localhost.localdomain localhost

192.168.2.251 pve-hs-main.local     pve-hs-main     pvelocalhost
192.168.2.252 pve-hs-replica.local  pve-hs-replica
192.168.2.253 pve-hs-3.local        pve-hs-3


10.10.10.251 cluster-main.local      cluster-main
10.10.10.252 cluster-replica.local   cluster-replica
10.10.10.253 cluster-3.local         cluster-3

In fact during a replication I see:
Code:
pve-hs-main[0]:/$ ps aux | grep zfs

root     13302  0.0  0.0  17948  2680 ?        S    13:05   0:00 /bin/bash -c set -o pipefail && pvesm export local-zfs:vm-101-disk-2 zfs - -with-snapshots 1 -snapshot __replicate_101-0_1506683155__ | /usr/bin/ssh -o 'BatchMode=yes' -o 'HostKeyAlias=pve-hs-3' root@192.168.2.253 -- pvesm import local-zfs:vm-101-disk-2 zfs - -with-snapshots 1
root     13303  0.0  0.2 294836 63136 ?        S    13:05   0:00 /usr/bin/perl /usr/sbin/pvesm export local-zfs:vm-101-disk-2 zfs - -with-snapshots 1 -snapshot __replicate_101-0_1506683155__
root     13304  2.2  0.0  47956  7736 ?        S    13:05   3:59 /usr/bin/ssh -o BatchMode=yes -o HostKeyAlias=pve-hs-3 root@192.168.2.253 -- pvesm import local-zfs:vm-101-disk-2 zfs - -with-snapshots 1
root     13331  1.1  0.0 107328  2900 ?        Sl   13:05   2:02 zfs send -Rpv -- rpool/data/vm-101-disk-2 __replicate_101-0_1506683155__

And I see the connection betweend nodes of the cluster is on IP 192.168.2.253 (third node pve-hs-3)


This is generating a lot of traffic on LAN used by VMs to communicate with clients, instead the replication could happen on the cluster dedicated LAN. But I can't find a way to change the IP that pvesr is using.

Is there some way to configure this?

Thanks a lot
 
Sorry to hijack the thread, but I think it's closely related.

I'm currently planning on setting up a small 3-node cluster in my homelab, but the 3rd mainly as quorum helper if one of my compute nodes goes down (just an old Atom Box with single LAN interface). I wanted to setup a Point-to-Point connection as storage network between the two compute nodes and configure that as migration network. What happens in case of migration to the 3rd node outside of that network?
 
Last edited:
Sorry to hijack the thread, but I think it's closely related.

Im currently planning on setting up a small 3-node cluster in my homelab, but the 3rd mainly as quorum helper if one of my compute nodes goes down (just an old Atom Box with single LAN interface). I wanted to setup a Point-to-Point connection as storage network between the two compute nodes and configure that as migration network. What happens in case of migration to the 3rd node outside of that network?

unless you override the migration network using the command line parameter "migration_network", migration will fail.
 
  • Like
Reactions: Mogli

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!