zfs replication via second LAN?

[randyqx]

i am a proxmox n00b, coming fom ganeti. i am used to drbd-based image replication being on a second LAN, which is pretty much only for that purpose. i am using zfs for image store in proxmox, and zfs-based replication. is there a means to keep the replication traffic off the 'front panel' LAN?

 

[justinclift]

Yeah. When I had the same question I found an older answer that said all replication traffic goes over vmbr0. So people would need to adjust their network config to have vmbr0 be the replication traffic bridge, and the public traffic would then come in over a newly created vmbr1.

So that's what I set up, and it works fine.

Later on I found out there's also an option in the /etc/pve/datacenter.cfg file called `migration`, which controls the network all of your migration traffic goes over:

# cat /etc/pve/datacenter.cfg

crs: ha-rebalance-on-start=1,ha=static
ha: shutdown_policy=migrate
keyboard: en-us
migration: insecure,network=10.200.1.0/24

I'm not sure if the network listed for the migration traffic there also does replication traffic or not, but that entry does seem to work for migration traffic.

The (optional) insecure keyword there tells Proxmox to just do a direct tcp connection (unencrypted) between hosts for replication data (the memory copy part anyway), rather than using ssh. Much, much faster. Perfectly fine in a homelab, but not super suitable for a production environment.

The ha: shutdown_policy=migrate line might be of interest too for clustered setups. With that setting, when you tell a host box to shut down (or reboot) it automatically migrates any VMs on it to other hosts first (then does the shut down or reboot). That's kind of important, as the default setting for Proxmox will just shut down any VMs and the host instead, thereby causing an outage for anyone using those VMs (!!!).

 

[randyqx]

thanks! `/etc/pve/datacenter.cfg` is non-existent on my config, but `/etc/pve/replication.cfg` looks promising although empty. i will experiment.

 

[Falk R.]

thanks! `/etc/pve/datacenter.cfg` is non-existent on my config, but `/etc/pve/replication.cfg` looks promising although empty. i will experiment.

Set the Migration Network in your GUI under Datacenter. This Setting creates the configfile automaticaly.

 

[justinclift]

@randyqx Ahhh. That /etc/pve/replication.cfg file seems to be a different thing.

In my test lab systems it has the list of replication jobs between nodes.

For example, at the moment there's only a single job defined, automatically replicating VM 100 from server1 to server2 using the default schedule (every 15 mins):

local: 100-0
        target server2
        source server1

You can set that kind of thing up in the GUI using the Replication tab in individual VMs. From the command line it's the pvesr command.

If you're using a multi-node setup, it's probably worth taking a look at.

 

[randyqx]

Set the Migration Network in your GUI under Datacenter. This Setting creates the configfile automaticaly.

got it. thanks. i am a bit more focused on replication, as that is a perpetual load and not insignificant. so far, i used the hack of a separate front panel LAN bridge, `br-lan`, for payload data, and configured `vmbr0` to bridge to the back end LAN. seems to work.