Join Cluster on Storage Network

[Raymond Burns]

I have two main networks setup on my Main Proxmox Host

# network interface settingsauto lo
iface lo inet loopback


auto eth0
iface eth0 inet manual


auto eth1
iface eth1 inet manual


iface eth2 inet manual

iface eth3 inet manual


auto bond0
iface bond0 inet manual
        slaves eth2 eth3
        bond_miimon 100
        bond_mode 802.3ad


auto vmbr0
iface vmbr0 inet static
        address  10.1.222.55
        netmask  255.255.255.0
        gateway  10.1.222.1
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0


auto vmbr1
iface vmbr1 inet static
        address  192.168.222.10
        netmask  255.255.255.0
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0


auto vmbr2
iface vmbr2 inet manual
        bridge_ports none
        bridge_stp off
        bridge_fd 0

The other node is

auto bond0
iface bond0 inet manual
     slaves eth0 eth1
     bond_miimon 100
     bond_mode 802.3ad

auto vmbr0
iface vmbr0 inet static
     address 192.168.222.13
     netmask 255.255.255.0
     gateway 192.168.222.3
     bridge_ports bond0
     bridge_stp off
     bridge_fd 0

I tried to do

pvecm add 192.168.222.10

on the secondary node, but it timed out on WAITING FOR QUORUM.
Is there any way to join the node to the cluster through the other network now?

Issue: Secondary node only has 2 ethernet ports and they must be used bonded to connect to the NFS Server.

 

[wahmed]

I think the reason is your 2nd node is on vmbr0 while your first node 192.xxx subnet is on vmbr1. In order to create cluster, all nodes should be on same subnet on same virtual switch.

 

[Raymond Burns]

Does not work.
It times out on waiting on Cluster.
I'm thinking it has something to do with multicast
Master

root@zwtprox1:~# pvecm statusVersion: 6.2.0
Config Version: 4
Cluster Name: ZWT
Cluster Id: 618
Cluster Member: Yes
Cluster Generation: 4
Membership state: Cluster-Member
Nodes: 1
Expected votes: 1
Total votes: 1
Node votes: 1
Quorum: 1
Active subsystems: 5
Flags:
Ports Bound: 0
Node name: zwtprox1
Node ID: 1
Multicast addresses: 239.192.2.108
Node addresses: 10.1.222.55

Secondary

root@zwtprox2:~# pvecm status
Version: 6.2.0
Config Version: 4
Cluster Name: ZWT
Cluster Id: 618
Cluster Member: Yes
Cluster Generation: 8
Membership state: Cluster-Member
Nodes: 1
Expected votes: 2
Total votes: 1
Node votes: 1
Quorum: 2 Activity blocked
Active subsystems: 5
Flags:
Ports Bound: 0
Node name: zwtprox1
Node ID: 1
Multicast addresses: 239.192.2.108
Node addresses: 192.168.222.13

How do I create a cluster on a different network?
IE pvecm create CLUSTER-NAME automatically selects the vmbr0
I need it to be on my vmbr1
How do I do a pvecm create on vmbr1 and will that break anything?

 

[wahmed]

How do I create a cluster on a different network?
IE pvecm create CLUSTER-NAME automatically selects the vmbr0
I need it to be on my vmbr1
How do I do a pvecm create on vmbr1 and will that break anything?

Each node can only be in One cluster. All host nodes must be on same subnet to be member of cluster. pvecm will Auto select vmbr0, most probably because thats the default bridge on all nodes so they can talk to each other for PVE filesystem replication. Since two bridges cannot have same subnet, cluster creation from one subnet to another subnet is not possible. This is just from my understanding of Proxmox mechanics.
Do you have any cluster already setup on any of these nodes?