Cluster with dedicated network, using public IP (?

[carles89]

Hi,

I've set up a two node cluster using a dedicated network, as seen on Proxmox Wiki: https://pve.proxmox.com/wiki/Cluster_Manager

Dedicated network range: 10.17.14.0/24
Node 1 IP: 10.17.14.1
Node 2 IP: 10.17.14.2

When creating the cluster, on the first node I did:

pvecm create mycluster --ring0_addr 10.17.14.1 --bindnet0_addr 10.17.14.0

And for joining the second node:

pvecm add 10.17.14.1 -ring0_addr 10.17.14.2

And that's the output of /etc/pve/corosync.conf

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: node1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 10.17.14.1
  }

  node {
    name: node2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 10.17.14.2
  }

}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: mycluster
  config_version: 4
  ip_version: ipv4
  secauth: on
  version: 2
  interface {
    bindnetaddr: 10.17.14.0
    ringnumber: 0
  }

}

And that's the output of /etc/pve/.members:

{
"nodename": "node1",
"version": 4,
"cluster": { "name": "mycluster", "version": 4, "nodes": 2, "quorate": 1 },
"nodelist": {
  "node1": { "id": 1, "online": 1, "ip": "37.xx.xx.xx"},
  "node2": { "id": 2, "online": 1, "ip": "5.xx.xx.xx"}
  }
}

As you can see, on .members, public IPs are listed instead of the dedicated cluster network ones.

Is that normal?

I've had to add a firewall rule on both nodes allowing to pass all traffic between them on public IPs.

If I only set a rule allowing all traffic on 10.17.14.0/24 range between nodes, I cannot view or config VMs running on the other node. For example, I cannot manage a VM stored on node 2 from node 1.

Thank you all

Regards

 

[wolfgang]

Hi,

only the cluster communication goes over this network.
What you mean is the api.

You have to edit on all nodes the
/etc/hosts

There is the public ip and pve is searching for 'pvelocalhost'

 

[carles89]

Hi Wolfgang,

OK, I understand that's the expected behaviour. I'll leave the api on the public interfaces, since I want to acces to the api from other machines.

Thank you!

 

[wolfgang]

It listening on all ip addresses but the hosts file is a lookup for the cluster.

 

[carles89]

Hi,

What should I change? Should I add the private IP address too on every hosts file?

Here is the /etc/hosts file on node0:

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1       localhost.localdomain localhost
37.xx.xx.xx    node0      node0
# The following lines are desirable for IPv6 capable hosts
#(added automatically by netbase upgrade)
::1     ip6-localhost ip6-loopback
feo0::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

And on node1:

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1       localhost.localdomain localhost
5.xx.xx.xx   node1.domain        node1
# The following lines are desirable for IPv6 capable hosts
#(added automatically by netbase upgrade)
::1     ip6-localhost ip6-loopback
feo0::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Regards,

 

[wolfgang]

change the ip in this file and to be sure add pvelocalhost

10.17.14.1 node0.domain node0 pvelocalhost

 

[carles89]

Hi,

I've done that, but I still cannot access to a VM's console from other node unless I allow communication between public interfaces.

Is it ok?

Thanks

 

[wolfgang]

You have to restart the pvedaemon.service.

 

[carles89]

It still does not work. After restarting services, if I disable the FW rule allowing communication between public interfaces, I cannot acces the console.

Thank you

 

[carles89]

Hi,

I'm still having the issue. I've noticed /etc/pve/.members version is different on both nodes. corosync.conf version is the same.

Is it related to the issue?

Thank you