Force ZFS replication traffic over separate NIC

[Alain Van Messen]

Hello,

New to Proxmox VE, I' m actually building a two node POC as such :

• Small SSD for OS
• Large SSD for VM storage
• 1Gbps NIC for LAN
• 10Gbps NIC for storage LAN (server to server cable)

Plan is to run the VMs on one node and replicating the VM data to the
other node every 15 mins through ZFS. Have this working just fine but
the replication traffic runs over the 1Gbps link and not the 10Gbps link.
How can I force this traffic over the "internal" high-speed connection ?

Thanks to advise !

 

[LnxBil]

How can I force this traffic over the "internal" high-speed connection ?

use a separate network. If you reach the replication target IP via your 1 GBE connection, it'll used.

I' m actually building a two node POC as such :

• Small SSD for OS
• Large SSD for VM storage

If you POC is planed to be used in production and if you mean only "one" ssd for each then please use two large SSD for everything in a mirrored setup.

 

[Alain Van Messen]

use a separate network. If you reach the replication target IP via your 1 GBE connection, it'll used.



If you POC is planed to be used in production and if you mean only "one" ssd for each then please use two large SSD for everything in a mirrored setup.

You mean Proxmox VE + VM storage directly on a mirrored ZFS pool per node ?
(a POC per definition will not be used in production, thank god ...)

Each machine can see the other over both links (on different networks) so how can
I force the ZFS replication over one link or the other ?

 

[LnxBil]

You mean Proxmox VE + VM storage directly on a mirrored ZFS pool per node ?

If you only want to have two disks, yes. If there is no other limit (except cost), use as much as you can.

(a POC per definition will not be used in production, thank god ...)

From your lips to God's ears - I've seen otherwise.

Each machine can see the other over both links (on different networks) so how can
I force the ZFS replication over one link or the other ?

A host (e.g. IP) can only be reached via one link (if you're not using bonding). Can you share your network configuration?

In a cluster (PVE but also otherwise), you normally have your cluster network with own IPs, own hostnames etc. so that everything is distinct. You configure every cluster-related aspect explicitly only with the IPs and hostnames from this interconnect network. This ensures that the traffic is only possible on that network and stay separated.

 

[Alain Van Messen]

Each node has :

NIC ens192
vmbr0 including ens192 in 192.168.1.0/24 (would be public)
NIC ens224
vmbr1 including ens224 in 172.16.0.1/30 (would be storage)

Have tried setting up the cluster on both IP ranges.

 

[LnxBil]

Have tried setting up the cluster on both IP ranges.

..and reinstalled in between or how did you do it? Please post the output of pvecm status (in CODE tags).

 

[Alain Van Messen]

Testing everything on a VMware ESX server before buying the hardware,
so snapshots are my friends ! Took one before creating the cluster so I' ve
gotten rid of vmbr1 and set the IP on the second NIC. Also modified hosts
file on both servers to reflect the IP addres/FQDN/hostname for both servers
+ reboot. Pinging from pve1 to pve2 works just fine but I cannot see the traffic
on the ESX monitor for those vmnics. Also tried putting the "storage" related
NICs in a separate vSwitch.

root@pve1:~# pvecm status
Quorum information
------------------
Date:             Thu Jul 18 15:22:39 2019
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000001
Ring ID:          1/16
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   2
Highest expected: 2
Total votes:      2
Quorum:           2
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 172.16.0.1 (local)
0x00000002          1 172.16.0.2

 

[tom]

How can I force this traffic over the "internal" high-speed connection ?

See "Datacenter/Options: Migration settings"

 

[Alain Van Messen]

See "Datacenter/Options: Migration settings"

Is the migration parameter also relevant to ZFS replications or only migration traffic ?
(it is the ZFS replication traffic that I' m interrested in)

 

[LnxBil]

Also modified hosts
file on both servers to reflect the IP addres/FQDN/hostname for both servers
+ reboot.

Could you please also share those files?

 

[Alain Van Messen]

Reinstalled everything from scratch in one and the same vSwitch (so all can see each other) :





Cluster status :

root@pve1:~# pvecm status
Quorum information
------------------
Date:             Fri Jul 19 14:18:39 2019
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000001
Ring ID:          1/8
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   2
Highest expected: 2
Total votes:      2
Quorum:           2 
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 192.168.1.151 (local)
0x00000002          1 192.168.1.152

Hosts file :

127.0.0.1 localhost.localdomain localhost
192.168.1.151 pve1.evoluto.be pve1
192.168.1.152 pve2.evoluto.be pve2

Datacenter.cfg :

migration: type=insecure,network=172.16.0.0/30

Cluster is setup on 192.168.1.0/24 which represents public internet in our POC.
Storage network (point-to-point) is setup on 172.16.0.0/30

 

[tom]

Looks ok, but you should consider adding a third node for reliable quorum.

 

[Alain Van Messen]

To be 100% sure I' ve disconnected the "storage" NIC on one node under VMware and
I am now unable to replicate a VM to the other node, meaning the traffic actually runs on
the intended link and the network monitor on our ESXi is not a reliable source !

 

[Alain Van Messen]

Looks ok, but you should consider adding a third node for reliable quorum.

Is a third node a must since we' re not doing HA or bidirectional syncing ?
Node 1 is only replicating storage through ZFS to node 2 in case of failure.
In normal operation we would not run anything on node 2, it' s only purpose
being disaster recovery.

 

[tom]

As long as you two node cluster is running, no issues. If you loose one node, you loose quorum and you cannot manage your remaining Cluster. VMs will stay online, but you cannot manage them unless you manually gain quorum again by starting the missing node again or manually set the needed quorum to 1 vote.

 

[Alain Van Messen]

As long as you two node cluster is running, no issues. If you loose one node, you loose quorum and you cannot manage your remaining Cluster. VMs will stay online, but you cannot manage them unless you manually gain quorum again by starting the missing node again or manually set the needed quorum to 1 vote.

So in our case this would be :

pvecm expected 1
pvecm delnode pve1
mv /etc/pve/node/pve1/qemu-server/*.conf /etc/pve/node/pve2/qemu-server/

 

[tom]

In most situations, just fix the second node and boot again. No need to delete a node from the cluster config in this case.

In the case of software upgrades and reboots, you also loose quorum for the reboot time. So having a third node/vote, is nicer.
If you are aware of the limitations of two node cluster, you can go for it, but keep the quorum issue in your mind.

 

[Alain Van Messen]

In most situations, just fix the second node and boot again. No need to delete a node from the cluster config in this case.

In the case of software upgrades and reboots, you also loose quorum for the reboot time. So having a third node/vote, is nicer.
If you are aware of the limitations of two node cluster, you can go for it, but keep the quorum issue in your mind.

So doing :

pvecm expected 1
mv /etc/pve/node/pve1/qemu-server/*.conf /etc/pve/node/pve2/qemu-server/

Would be enough to get the VMs running on the second node ?

Adding the failed node again would only work if we don' t have to reinstall right ?

 

[tom]

Would be enough to get the VMs running on the second node ?

Depends on the VM config, also see:

https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pvesr

BTW, the thread topic is "Force ZFS replication traffic over separate NIC", so if you have further question do the following:

1. Search this forum for similar thread
2. If you cannot find an answer, please post a new thread for new questions

By following these rules, other can follow much better and the whole community can benefit from your questions and ideas.

 

[Alain Van Messen]

Depends on the VM config, also see:

https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pvesr

BTW, the thread topic is "Force ZFS replication traffic over separate NIC", so if you have further question do the following:

1. Search this forum for similar thread
2. If you cannot find an answer, please post a new thread for new questions

By following these rules, other can follow much better and the whole community can benefit from your questions and ideas.

Agreed, thanks Tom !