[SOLVED] Adding sencond node to cluster faled

aasami

aasami

Renowned Member
Proxmox Subscriber
Mar 9, 2016
61
16
73
Ihave tried to add a node2 to newly created cluster on node1 which failed:

Code: 
[root@node2]# pvecm add node1
The authenticity of host 'node1 (10.0.0.1)' can't be established.
ECDSA key fingerprint is SHA256:aRIk8xO64eUL5RVX7uxy5BlIeg6EjtqKc7KWnKLruQE.
Are you sure you want to continue connecting (yes/no)? yes
root@node1's password:
unable to copy ssh ID: cat: write error: Permission denied
[root@node2]#

Now it says the quorum is lost and I can't do anything:

Code: 
[root@node1]# pvecm status
Quorum information
------------------
Date:             Tue Jan 30 12:10:48 2018
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000001
Ring ID:          1/16
Quorate:          No

Votequorum information
----------------------
Expected votes:   2
Highest expected: 2
Total votes:      1
Quorum:           2 Activity blocked
Flags:           

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 10.0.0.1 (local)
[root@node1]#

Code: 
[root@node1]# pvecm nodes

Membership information
----------------------
    Nodeid      Votes Name
         1          1 node1 (local)
[root@node1]#

Code: 
[root@node1]# cat /etc/pve/corosync.conf
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: node2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: node2
  }
  node {
    name: node1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: node1
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: cluster
  config_version: 2
  interface {
    bindnetaddr: 10.0.0.1
    ringnumber: 0
  }
  ip_version: ipv4
  secauth: on
  version: 2
}
[root@node1]#

What can I do?
 
Hi,

what you get as error message if you try to execute from both nodes

Code: 
ssh-copy-id -o "HostKeyAlias=<RemoteNode> <RemoteNode IP>"
 
Code: 
[root@node1]# ssh-copy-id -o "HostKeyAlias=node2" 10.0.0.2
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh -o 'HostKeyAlias=node2' '10.0.0.2'"
and check to make sure that only the key(s) you wanted were added.

[root@node1]# ssh -o "HostKeyAlias=node2" 10.0.0.2
Linux node2 4.13.13-5-pve #1 SMP PVE 4.13.13-38 (Fri, 26 Jan 2018 10:47:09 +0100) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Jan 30 12:36:10 2018 from 10.0.0.100
[root@node2]# ssh-copy-id -o "HostKeyAlias=node1" 10.0.0.1
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
cat: write error: Permission denied
[root@node2]#
 
Thank you for your interest Wolfgang.
I have managed to copy the key this way:
Code: 
[root@node1]# ls -l /root/.ssh/authorized_keys
lrwxrwxrwx 1 root root 29 júl 19  2017 /root/.ssh/authorized_keys -> /etc/pve/priv/authorized_keys
[root@node1]# ls -l /etc/pve/priv/authorized_keys
-r-------- 1 root www-data 1570 jan 30 12:03 /etc/pve/priv/authorized_keys
[root@node1]# systemctl stop corosync
[root@node1]# pmxcfs -l
[main] notice: unable to aquire pmxcfs lock - trying again
[main] crit: unable to aquire pmxcfs lock: Resource temporarily unavailable
[main] notice: exit proxmox configuration filesystem (-1)
[root@node1]# rm -f /var/lib/pve-cluster/.pmxcfs.lockfile
[root@node1]# pmxcfs -l
[main] notice: forcing local mode (although corosync.conf exists)
[root@node1]# ls -l /etc/pve/priv/authorized_keys
-rw------- 1 root www-data 1570 jan 30 12:03 /etc/pve/priv/authorized_keys
[root@node1]#
Code: 
[root@node2]# ssh-copy-id -o "HostKeyAlias=node1" 10.0.0.1
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh -o 'HostKeyAlias=node1' '10.0.0.1'"
and check to make sure that only the key(s) you wanted were added.

[root@node2]#
Code: 
[root@node1]# rm -f /var/lib/pve-cluster/.pmxcfs.lockfile
[root@node1]# systemctl start corosync
[root@node1]# systemctl start pve-cluster
[root@node1]# systemctl restart pvedaemon
[root@node1]# systemctl restart pveproxy
[root@node1]#
What's the next step?
 
What is the current status of your cluster on node 2.
Did you try to restart also all services on the node2?
 
I have tired to reboot whole node. Corosync.conf doesn't exist on node2.
Code: 
[root@node2]# pvecm status
Corosync config '/etc/pve/corosync.conf' does not exist - is this node part of a cluster?
Cannot initialize CMAP service
[root@node2]#
 
I am out of ideas now. On node2, there is no sign of it beeing in a cluster anywhere.
Is there any way how to fool node1 for it to think that it is the only node in it's cluster?

Code: 
[root@node1]# cat corosync.conf
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: node2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: node2
  }
  node {
    name: node1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: node1
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: cluster
  config_version: 2
  interface {
    bindnetaddr: 10.0.0.1
    ringnumber: 0
  }
  ip_version: ipv4
  secauth: on
  version: 2
}

[root@node1]# cat .members
{
"nodename": "node1",
"version": 3,
"cluster": { "name": "cluster", "version": 2, "nodes": 2, "quorate": 0 },
"nodelist": {
  "hp6": { "id": 2, "online": 0},
  "hp7": { "id": 1, "online": 1, "ip": "10.0.0.1"}
  }
}
[root@node1]
 
set on the first node the expected votes to one.
Code: 
pvecm expected 1

Add the second node with a force parameter again to the cluster
 
Last edited:
  • Like
Reactions: aasami
Success!!! Thank you a lot. You saved me.

Code: 
[root@node2]# pvecm add --force 1 node1
trying to aquire cfs lock 'file-corosync_conf' ...
trying to aquire cfs lock 'file-corosync_conf' ...
trying to aquire cfs lock 'file-corosync_conf' ...
trying to aquire cfs lock 'file-corosync_conf' ...
trying to aquire cfs lock 'file-corosync_conf' ...
trying to aquire cfs lock 'file-corosync_conf' ...
trying to aquire cfs lock 'file-corosync_conf' ...
trying to aquire cfs lock 'file-corosync_conf' ...
trying to aquire cfs lock 'file-corosync_conf' ...
copy corosync auth key
stopping pve-cluster service
backup old database
waiting for quorum...OK
generating node certificates
merge known_hosts file
restart services
successfully added node 'node2' to cluster.
[root@node2]#
Code: 
[root@node1]# pvecm status
Quorum information
------------------
Date:             Thu Feb  1 12:40:02 2018
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000001
Ring ID:          2/44
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   2
Highest expected: 2
Total votes:      2
Quorum:           2  
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000002          1 10.0.0.2
0x00000001          1 10.0.0.1 (local)
[root@node1]# pvecm nodes

Membership information
----------------------
    Nodeid      Votes Name
         2          1 node2
         1          1 node1 (local)
[root@node1]#
 
You must log in or register to reply here.
Top Bottom