Do not add the node to the cluster DRBD (PVE-4.3)

K

kolombo

New Member
Feb 18, 2013
8
0
1
.ru
Good time of day.
There is a cluster consisting of three nodes (pve1, pve3, pve4). PVE-4.2 was installed. And it was made common space DRBD. There is a need to add new node (pve2). On adding a node, I set PVE-4.3, then turn updated the existing three-node (pve1, pve3, pve4) and then added a new node (pve2) to the cluster. When you try to add a section DRBD get the error.

on the first node, I do
Code: 
root@pve1:~# drbdmanage add-node -q pve2 172.20.1.2
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
Operation completed successfully
Operation completed successfully

Executing join command using ssh.
IMPORTANT: The output you see comes from pve2
IMPORTANT: Your input is executed on pve2
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
Error: Operation not allowed on satellite node

Code: 
root@pve1:~# drbdmanage list-nodes
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
+------------------------------------------------------------------------------------------------------------+
| Name | Pool Size | Pool Free |                                   |                               State |
|------------------------------------------------------------------------------------------------------------|
| pve1 |     65536 |     65234 |                                   |                                  ok |
| pve2 |   unknown |   unknown |                                   | pending actions: adjust connections |
| pve3 |     65536 |     65234 |                                   |                                  ok |
| pve4 |     65536 |     65234 |                                   |                                  ok |
+------------------------------------------------------------------------------------------------------------+

Since I have all written
Code: 
post-up drbdadm adjust all
on all nodes in /etc/network/interfaces, I picked up one by one and restart all nodes. The output after the restart
Code: 
root@pve1:~# drbdmanage list-nodes
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
+------------------------------------------------------------------------------------------------------------+
| Name | Pool Size | Pool Free |                          |                                        State |
|------------------------------------------------------------------------------------------------------------|
| pve1 |     65536 |     65234 |                          |                                           ok |
| pve2 |   unknown |   unknown |                          | OFFLINE, pending actions: adjust connections |
| pve3 |     65536 |     65234 |                          |                                           ok |
| pve4 |     65536 |     65234 |                          |                                           ok |
+------------------------------------------------------------------------------------------------------------+

Tell me where and what went wrong and how to fix it?
 
Code: 
root@pve1:~# pvecm status
Quorum information
------------------
Date:             Thu Sep 29 09:28:52 2016
Quorum provider:  corosync_votequorum
Nodes:            4
Node ID:          0x00000001
Ring ID:          1/248
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   4
Highest expected: 4
Total votes:      4
Quorum:           3 
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 172.20.1.1 (local)
0x00000004          1 172.20.1.2
0x00000002          1 172.20.1.3
0x00000003          1 172.20.1.4

If I understand correctly, the section "Votequorum information" should be displayed "Quorum: 4" ?
 
no - that just says that for 4 nodes with 1 vote each, a minimum of 3 votes means you have a quorum. also, the PVE/corosync cluster is not related to the drbd cluster/synchronization. my guess is, something went wrong when adding node pve2, and you should probably remove that one from the drbd cluster, drbdmanage uninit it, and then try to add it again.
 
I did:
deleted node of DRBD
Code: 
root@pve1:~# drbdmanage delete-node -q pve2

I removed the node from the cluster
Code: 
root@pve1:~# pvecm delnode pve2
Killing node 4

He added the node to the cluster
Code: 
root@pve2:~# pvecm add 172.20.1.1 -ring0_addr pve2cl -force
copy corosync auth key
stopping pve-cluster service
backup old database
waiting for quorum...OK
generating node certificates
merge known_hosts file
restart services
successfully added node 'pve2' to cluster.

Update certs
Code: 
root@pve2:~# pvecm updatecerts

I add a node in the DRBD
Code: 
root@pve1:~# drbdmanage add-node -q pve2 172.20.1.2
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
Operation completed successfully
Operation completed successfully

Executing join command using ssh.
IMPORTANT: The output you see comes from pve2
IMPORTANT: Your input is executed on pve2
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
Error: Operation not allowed on satellite node

and nothing has changed:
Code: 
root@pve1:~# drbdmanage list-nodes
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
+------------------------------------------------------------------------------------------------------------+
| Name | Pool Size | Pool Free |                                   |                               State |
|------------------------------------------------------------------------------------------------------------|
| pve1 |     65536 |     65234 |                                   |                                  ok |
| pve2 |   unknown |   unknown |                                   | pending actions: adjust connections |
| pve3 |     65536 |     65234 |                                   |                                  ok |
| pve4 |     65536 |     65234 |                                   |                                  ok |
+------------------------------------------------------------------------------------------------------------+
:(
 
you did not uninit the node after removing (in order to completely reset the DRBD config on that node).
 
"drbdmanage uninit" on pve2 should do the trick (note: if you already have production data on that drbd cluster, back it up before playing around with it like this!)
 
fabian said:
"drbdmanage uninit" on pve2
Click to expand...

To no avail.

On pve1:
Code: 
root@pve1:~# drbdmanage list-nodes
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
+------------------------------------------------------------------------------------------------------------+
| Name | Pool Size | Pool Free |                                   |                               State |
|------------------------------------------------------------------------------------------------------------|
| pve1 |     65536 |     65234 |                                   |                                  ok |
| pve2 |   unknown |   unknown |                                   | pending actions: adjust connections |
| pve3 |     65536 |     65234 |                                   |                                  ok |
| pve4 |     65536 |     65234 |                                   |                                  ok |
+------------------------------------------------------------------------------------------------------------+

On pve2:
Code: 
root@pve2:~# drbdmanage uninit
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'

You are going to remove the drbdmanage server from this node.
CAUTION! Note that:
  * All temporary configuration files for resources managed by drbdmanage
    will be removed
  * Any remaining resources managed by this drbdmanage installation
    that still exist on this system will no longer be managed by drbdmanage

Confirm:

  yes/no: yes
root@pve2:~#
root@pve2:~# drbdadm status
# No currently configured DRBD found.
root@pve2:~#
root@pve2:~# drbdmanage list-nodes
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'

On pve1:
Code: 
root@pve1:~# drbdmanage list-nodes
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
+------------------------------------------------------------------------------------------------------------+
| Name | Pool Size | Pool Free |                          |                                        State |
|------------------------------------------------------------------------------------------------------------|
| pve1 |     65536 |     65234 |                          |                                           ok |
| pve2 |   unknown |   unknown |                          | OFFLINE, pending actions: adjust connections |
| pve3 |     65536 |     65234 |                          |                                           ok |
| pve4 |     65536 |     65234 |                          |                                           ok |
+------------------------------------------------------------------------------------------------------------+

On pve1
Code: 
root@pve1:~# drbdmanage delete-node -q pve2
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
Operation completed successfully
Code: 
root@pve1:~# drbdmanage add-node -q pve2 172.20.1.2
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
Operation completed successfully
Operation completed successfully

Executing join command using ssh.
IMPORTANT: The output you see comes from pve2
IMPORTANT: Your input is executed on pve2
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
Error: Operation not allowed on satellite node


But that was not the case!
For some reason it was not on pve2 /etc/drbd.d/drbdctrl.res file!

I copied it from the node pve1
Code: 
root@pve1:~# scp /etc/drbd.d/drbdctrl.res 172.20.1.2:/etc/drbd.d

removed node
Code: 
root@pve1:~# drbdmanage delete-node -q pve2
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
Operation completed successfully

added node
Code: 
root@pve1:~# drbdmanage add-node -q pve2 172.20.1.2
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
Operation completed successfully
Operation completed successfully

Executing join command using ssh.
IMPORTANT: The output you see comes from pve2
IMPORTANT: Your input is executed on pve2
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
Operation completed successfully

and
Code: 
root@pve1:~# drbdmanage list-nodes
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
+------------------------------------------------------------------------------------------------------------+
| Name | Pool Size | Pool Free |                                                                 | State |
|------------------------------------------------------------------------------------------------------------|
| pve1 |     65536 |     65234 |                                                                 |    ok |
| pve2 |     65536 |     65536 |                                                                 |    ok |
| pve3 |     65536 |     65234 |                                                                 |    ok |
| pve4 |     65536 |     65234 |                                                                 |    ok |
+------------------------------------------------------------------------------------------------------------+

But I am embarrassed that all node "Pool Free" = 65234, while pve2 - 65536.

In all four nodes were created sections:
Code: 
lvcreate -L 64G -n drbdthinpool -T drbdpool

Can we assume that the problem is resolved, I do not know yet. I checked again.
 
kolombo said:
But that was not the case!
For some reason it was not on pve2 /etc/drbd.d/drbdctrl.res file!

I copied it from the node pve1
Click to expand...

Unfortunately I'm experiencing the same issue. I've tryed to reconfigure DRBD9 for uncountably many times and also copied this file, but as soon as I do that, and after that I add the second node, the res file gets cleared on the second server! Of course after that I always get this infamous "
Error: Operation not allowed on satellite node". I am offering 30 euros to whoever comes up with a solution to always avoid this problem, because sometimes I can add the second node and sometimes I get this error. It feels totally random.
 
Hi, I tried a fresh Proxmox Setup with DRBD9 on two servers and running in the same problem:
As I try to add the second node with drbdmange:

Code: 
root@tehhrxpmx4:~# drbdmanage init -q 172.16.5.4
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
Empty drbdmanage control volume initialized on '/dev/drbd0'.
Empty drbdmanage control volume initialized on '/dev/drbd1'.
Operation completed successfully

root@tehhrxpmx4:~# drbdmanage add-node -q tehhrxpmx5 172.16.5.5
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
Operation completed successfully
Operation completed successfully

Executing join command using ssh.
IMPORTANT: The output you see comes from tehhrxpmx5
IMPORTANT: Your input is executed on tehhrxpmx5
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
Error: Operation not allowed on satellite node

Code: 
root@tehhrxpmx4:~# drbdmanage list-nodes
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
+------------------------------------------------------------------------------------------------------------+
| Name       | Pool Size | Pool Free |                                 |                               State |
|------------------------------------------------------------------------------------------------------------|
| tehhrxpmx4 |   1047552 |   1047552 |                                 |                                  ok |
| tehhrxpmx5 |   unknown |   unknown |                                 | pending actions: adjust connections |
+------------------------------------------------------------------------------------------------------------+

On the second node I see, that /etc/drbd.d/drbdctrl.res is missing.

I tried to remove the node and did the add-node several times. Allways the same problem :-)

Please can anybody help?
 
crodde said:
Please can anybody help?
Click to expand...

A solution that I found should be working is to execute:

1) drbdmanage init -q 172.16.5.4 (on master server)
2) drbdmanage init (on the other server) --> this is key
3) drbdmanage add-node -q tehhrxpmx5 172.16.5.5 (on master server)

Please let us know if it works.
 
Last edited:
You must log in or register to reply here.
Top Bottom