[SOLVED] Adding sencond node to cluster faled

aasami · Jan 30, 2018

Ihave tried to add a node2 to newly created cluster on node1 which failed:

Code:

[root@node2]# pvecm add node1
The authenticity of host 'node1 (10.0.0.1)' can't be established.
ECDSA key fingerprint is SHA256:aRIk8xO64eUL5RVX7uxy5BlIeg6EjtqKc7KWnKLruQE.
Are you sure you want to continue connecting (yes/no)? yes
root@node1's password:
unable to copy ssh ID: cat: write error: Permission denied
[root@node2]#

Now it says the quorum is lost and I can't do anything:

Code:

[root@node1]# pvecm status
Quorum information
------------------
Date:             Tue Jan 30 12:10:48 2018
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000001
Ring ID:          1/16
Quorate:          No

Votequorum information
----------------------
Expected votes:   2
Highest expected: 2
Total votes:      1
Quorum:           2 Activity blocked
Flags:           

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 10.0.0.1 (local)
[root@node1]#

Code:

[root@node1]# pvecm nodes

Membership information
----------------------
    Nodeid      Votes Name
         1          1 node1 (local)
[root@node1]#

Code:

[root@node1]# cat /etc/pve/corosync.conf
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: node2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: node2
  }
  node {
    name: node1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: node1
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: cluster
  config_version: 2
  interface {
    bindnetaddr: 10.0.0.1
    ringnumber: 0
  }
  ip_version: ipv4
  secauth: on
  version: 2
}
[root@node1]#

What can I do?

wolfgang · Jan 31, 2018

Hi,

what you get as error message if you try to execute from both nodes

Code:

ssh-copy-id -o "HostKeyAlias=<RemoteNode> <RemoteNode IP>"

aasami · Jan 31, 2018

Code:

[root@node1]# ssh-copy-id -o "HostKeyAlias=node2" 10.0.0.2
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh -o 'HostKeyAlias=node2' '10.0.0.2'"
and check to make sure that only the key(s) you wanted were added.

[root@node1]# ssh -o "HostKeyAlias=node2" 10.0.0.2
Linux node2 4.13.13-5-pve #1 SMP PVE 4.13.13-38 (Fri, 26 Jan 2018 10:47:09 +0100) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Jan 30 12:36:10 2018 from 10.0.0.100
[root@node2]# ssh-copy-id -o "HostKeyAlias=node1" 10.0.0.1
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
cat: write error: Permission denied
[root@node2]#

aasami · Jan 31, 2018

Thank you for your interest Wolfgang.
I have managed to copy the key this way:

Code:

[root@node1]# ls -l /root/.ssh/authorized_keys
lrwxrwxrwx 1 root root 29 júl 19  2017 /root/.ssh/authorized_keys -> /etc/pve/priv/authorized_keys
[root@node1]# ls -l /etc/pve/priv/authorized_keys
-r-------- 1 root www-data 1570 jan 30 12:03 /etc/pve/priv/authorized_keys
[root@node1]# systemctl stop corosync
[root@node1]# pmxcfs -l
[main] notice: unable to aquire pmxcfs lock - trying again
[main] crit: unable to aquire pmxcfs lock: Resource temporarily unavailable
[main] notice: exit proxmox configuration filesystem (-1)
[root@node1]# rm -f /var/lib/pve-cluster/.pmxcfs.lockfile
[root@node1]# pmxcfs -l
[main] notice: forcing local mode (although corosync.conf exists)
[root@node1]# ls -l /etc/pve/priv/authorized_keys
-rw------- 1 root www-data 1570 jan 30 12:03 /etc/pve/priv/authorized_keys
[root@node1]#

Code:

[root@node2]# ssh-copy-id -o "HostKeyAlias=node1" 10.0.0.1
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh -o 'HostKeyAlias=node1' '10.0.0.1'"
and check to make sure that only the key(s) you wanted were added.

[root@node2]#

Code:

[root@node1]# rm -f /var/lib/pve-cluster/.pmxcfs.lockfile
[root@node1]# systemctl start corosync
[root@node1]# systemctl start pve-cluster
[root@node1]# systemctl restart pvedaemon
[root@node1]# systemctl restart pveproxy
[root@node1]#

What's the next step?

wolfgang · Jan 31, 2018

What is the current status of your cluster on node 2.
Did you try to restart also all services on the node2?

aasami · Jan 31, 2018

I have tired to reboot whole node. Corosync.conf doesn't exist on node2.

Code:

[root@node2]# pvecm status
Corosync config '/etc/pve/corosync.conf' does not exist - is this node part of a cluster?
Cannot initialize CMAP service
[root@node2]#

aasami · Feb 1, 2018

I am out of ideas now. On node2, there is no sign of it beeing in a cluster anywhere.
Is there any way how to fool node1 for it to think that it is the only node in it's cluster?

Code:

[root@node1]# cat corosync.conf
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: node2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: node2
  }
  node {
    name: node1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: node1
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: cluster
  config_version: 2
  interface {
    bindnetaddr: 10.0.0.1
    ringnumber: 0
  }
  ip_version: ipv4
  secauth: on
  version: 2
}

[root@node1]# cat .members
{
"nodename": "node1",
"version": 3,
"cluster": { "name": "cluster", "version": 2, "nodes": 2, "quorate": 0 },
"nodelist": {
  "hp6": { "id": 2, "online": 0},
  "hp7": { "id": 1, "online": 1, "ip": "10.0.0.1"}
  }
}
[root@node1]

wolfgang · Feb 1, 2018

set on the first node the expected votes to one.

Code:

pvecm expected 1

Add the second node with a force parameter again to the cluster

aasami · Feb 1, 2018

Success!!! Thank you a lot. You saved me.

Code:

[root@node2]# pvecm add --force 1 node1
trying to aquire cfs lock 'file-corosync_conf' ...
trying to aquire cfs lock 'file-corosync_conf' ...
trying to aquire cfs lock 'file-corosync_conf' ...
trying to aquire cfs lock 'file-corosync_conf' ...
trying to aquire cfs lock 'file-corosync_conf' ...
trying to aquire cfs lock 'file-corosync_conf' ...
trying to aquire cfs lock 'file-corosync_conf' ...
trying to aquire cfs lock 'file-corosync_conf' ...
trying to aquire cfs lock 'file-corosync_conf' ...
copy corosync auth key
stopping pve-cluster service
backup old database
waiting for quorum...OK
generating node certificates
merge known_hosts file
restart services
successfully added node 'node2' to cluster.
[root@node2]#

Code:

[root@node1]# pvecm status
Quorum information
------------------
Date:             Thu Feb  1 12:40:02 2018
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000001
Ring ID:          2/44
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   2
Highest expected: 2
Total votes:      2
Quorum:           2  
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000002          1 10.0.0.2
0x00000001          1 10.0.0.1 (local)
[root@node1]# pvecm nodes

Membership information
----------------------
    Nodeid      Votes Name
         2          1 node2
         1          1 node1 (local)
[root@node1]#

Search

Search

[SOLVED] Adding sencond node to cluster faled

aasami

Renowned Member

wolfgang

Proxmox Retired Staff

aasami

Renowned Member

aasami

Renowned Member

wolfgang

Proxmox Retired Staff

aasami

Renowned Member

aasami

Renowned Member

wolfgang

Proxmox Retired Staff

aasami

Renowned Member