[SOLVED] Question about creating a Cluster and using the right NIC

[GuiltyNL]

Hi guys,

I'm evaluating Proxmox and so far I'm impressed!

I have a question:

I have two physical machines that will be running Proxmox and I want to setup a Cluster.

They are running several NICs and I want to make the Cluster config right.

SERVER A NIC1 192.168.150.1 Meant for communication to the network
SERVER A NIC2 192.168.155.1 Meant for communication to the other hypervisor. This is a 10GBe NIC

SERVER B NIC1 192.168.150.2 Meant for communication to the network
SERVER B NIC2 192.168.155.2 Meant for communication to the other hypervisor. This is a 10GBe NIC

I want to use replication and so on in the Cluster on NIC2, as this is a fast and dedicated NIC for this purpose.

Do I need to specify the IP of the NIC2 (192.168.155.1) in the Ring 0 field when creating the Cluster?

And attach the 192.168.155.1 directly to the NIC to, without creating a bridge?

Thanks

 

[wolfgang]

Do I need to specify the IP of the NIC2 (192.168.155.1) in the Ring 0 field when creating the Cluster?

Yes, but you should also use ring1 on NIC2 because corosync is very latency sensible and if the replication runs the latency will increase.

And attach the 192.168.155.1 directly to the NICE to, without creating a bridge?

Yes, use only the nic no need for a bridge. Bridges are only needed for VM/CT.

 

[GuiltyNL]

I don't understand the ring1 recommendation on NIC2? @wolfgang

In my setup I would attach NIC2 to ring0 and according to your documentation the ring1 adapter should be physically seperated from each other?

Did you mean I have to attach ring1 to NIC1?

I'm able to add another NIC to this config, like:

SERVER A NIC1 192.168.150.1 Meant for communication to the network
SERVER A NIC2 192.168.155.1 Meant for communication to the other hypervisor. This is a 10GBe NIC
SERVER A NIC3 192.168.160.1 For Ring 1 ?

 

[wolfgang]

Yes I meant NIC1 for the ring1 and NIC2 for ring0

A dedicated network for corosync would be better.
So NIC2 for ring0 and NIC3 for ring1 would be good.

Also see
https://pve.proxmox.com/wiki/Separate_Cluster_Network

 

[GuiltyNL]

I'm puzzled at this moment. I'm doing some tests now with replication.

I've specified the IP address (192.168.155.1) of NIC2 as Ring0
The other node joined the cluster with 192.168.155.2 as Ring0 IP

Both IP addresses are bound to a 10GBe NIC adapter.

I start a replication and all data goes via the other 1GB adapter (NIC1) via 192.168.150.1 to 192.168.150.2

When I click the 'Join Information' I see at IP address 192.168.150.2

What am I missing here?

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: kermit
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 192.168.155.2
  }
  node {
    name: piggy
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 192.168.155.1
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: Muppets
  config_version: 2
  interface {
    bindnetaddr: 192.168.155.2
    ringnumber: 0
  }
  ip_version: ipv4
  secauth: on
  version: 2
}

 

[wolfgang]

You have to change the migration network in the datacenter.conf.
see

man datacenter.cfg

 

[GuiltyNL]

Aha, I missed that somehow. I was expecting he would choose the Ring0 as default route.

But wouldn't it be better to have the Ring0 on a NIC without (heavy) traffic?

So:

NIC1 Use for GUI / SSH Purposes (and maybe use as Ring1?)
NIC2 Use for LAN traffic to the VM's
NIC3 Use for DATA/Migration/Replication traffic in the cluster
NIC4 Use as Ring0 for the cluster

Would that make sense?

 

[GuiltyNL]

I solved this one:

My added settings in the datacenter.cfg don't get applied. I restarted the pve-cluster service on both nodes, don't get any parser errors in the syslogs on node 1 but it seems it throws and error on node 2:

keyboard: en-us
migration: type=insecure, network=192.168.155.0/24

Jul 06 09:49:00 pvesr[39345]: parse error in '/etc/pve/datacenter.cfg' - 'migration': invalid format - format error
Jul 06 09:49:00 pvesr[39345]: migration. network: property is not defined in schema and the schema does not allow additional properties

Solution: remove space after the comma, so:

keyboard: en-us
migration: type=insecure,network=192.168.155.0/24

 

[wolfgang]

NIC1 Use for GUI / SSH Purposes (and maybe use as Ring1?)
NIC2 Use for LAN traffic to the VM's
NIC3 Use for DATA/Migration/Replication traffic in the cluster
NIC4 Use as Ring0 for the cluster

Would that make sense?

Yes, this would be the correct way.