PVE-4.2 DRBD9 didn't create

[Black Knight MHT]

Hi.
Some monthes age I installed PVE4.2 with DRBD9 without any problems using wiki.
But today I have a problem.
After drbdmanage init -q I recieved such lay out:

root@pve4:~# drbdmanage init -q 100.100.0.4
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
Empty drbdmanage control volume initialized on '/dev/drbd0'.
Empty drbdmanage control volume initialized on '/dev/drbd1'.
Operation completed successfully
root@pve4:~# drbdadm status
.drbdctrl role:Secondary
volume:0 disk:UpToDate
volume:1 disk:UpToDate

root@pve4:~# drbdmanage list-nodes
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
+------------------------------------------------------------------------------------------------------------+
| Name | Pool Size | Pool Free | | State |
|------------------------------------------------------------------------------------------------------------|
| pve4 | 563200 | 563200 | | ok |
+------------------------------------------------------------------------------------------------------------+

And second node cann't connect.

root@pve4:~# drbdmanage add-node -q pve2 100.100.0.2
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
Operation completed successfully
Operation completed successfully

Executing join command using ssh.
IMPORTANT: The output you see comes from pve2
IMPORTANT: Your input is executed on pve2
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
Error: Operation not allowed on satellite node

root@pve4:~# drbdmanage list-nodes
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
+------------------------------------------------------------------------------------------------------------+
| Name | Pool Size | Pool Free | | State |
|------------------------------------------------------------------------------------------------------------|
| pve2 | unknown | unknown | | pending actions: adjust connections |
| pve4 | 563200 | 563200 | | ok |
+------------------------------------------------------------------------------------------------------------

 

[N9br]

Hi there,

I get exactly the same output, trying to set up a drbd9 cluster with 3 proxmox4-3 cluster nodes. Only 2 of them will do drbd.

Its a freshly installed setup on shiny new hardware. we have a 10G interconnect for drbd interfaces.

Setup is as follows (identical on all nodes)

proxmox-ve: 4.3-66 (running kernel: 4.4.19-1-pve)
pve-manager: 4.3-1 (running version: 4.3-1/e7cdc165)
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.4.16-1-pve: 4.4.16-64
pve-kernel-4.4.19-1-pve: 4.4.19-66
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-46
qemu-server: 4.0-88
pve-firmware: 1.1-9
libpve-common-perl: 4.0-73
libpve-access-control: 4.0-19
libpve-storage-perl: 4.0-61
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-qemu-kvm: 2.6.1-6
pve-container: 1.0-75
pve-firewall: 2.0-29
pve-ha-manager: 1.0-35
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 2.0.4-1
lxcfs: 2.0.3-pve1
criu: 1.6.0-1
novnc-pve: 0.5-8
zfsutils: 0.6.5.7-pve10~bpo80
drbdmanage: 0.97.3-1

I get the exact same error . I followed http://pve.proxmox.com/wiki/DRBD9 .
Not sure wether this is proxmox or drbd9 related

root@node1:~# drbdmanage add-node -q node2 172.16.100.102
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
Operation completed successfully
Operation completed successfully

Executing join command using ssh.
IMPORTANT: The output you see comes from prodnode2
IMPORTANT: Your input is executed on prodnode2
WARNING:root:Could not read configuration file '/etc/drbdmanaged.cfg'
Error: Operation not allowed on satellite node

Thanks!
Andre

 

[Black Knight MHT]

Hi.
It looks like I found 2 ways to solve this problem:
1. Install drbdmanager another version:
apt-get install drbdmanage=0.97-1 -y
and continue man
2. Install the latest version and do init on BOTH nodes. (drbdmanage init -q IP)
and after that add-node with overwriting control volumes.

 

[N9br]

Hi
we followed method 2. Could add the second node, but 'pool size' and 'pool free' are both unknown.
initially, we had an LV created on the PV with (almost) max size, as suggested by the docs.
That got me thinking, because DRBD9 is supposed to assign an LV to each machine when creating the machine in Proxmox.

Any experience with that ?
thanks!

 

[Black Knight MHT]

Hi!
Try method 1......but del drbdmanager first

 

[N9br]

No success yet :/

deleted drbdmanage, installed drbdmanage=0.97-1 .
same result
did delete-node(s), uninit
re-created the huge LV, then init and add-node(s) again.
still

root@node2:~# drbdmanage list-nodes
+--------------------------------------------------------------------------------------------------+
| Name      | Pool Size | Pool Free |                                                      | State |
|----------------------------------------------------------------------------------------------------|
| prodnode1 |   unknown |   unknown |                                                   |    ok |
| prodnode2 |   unknown |   unknown |                                                   |    ok |
+--------------------------------------------------------------+------------------------------------+

will read some more docs ..

 

[Black Knight MHT]

Well.
I have success with:
1. install version 0.97-1
2. init (already 3 nodes) all nodes
3. add-node from master.
4. update to the latest version

root@pve4:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 279.4G 0 disk
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 127M 0 part
└─sda3 8:3 0 279.2G 0 part
├─pve-root 251:0 0 69.8G 0 lvm /
├─pve-swap 251:1 0 8G 0 lvm [SWAP]
├─pve-data_tmeta 251:2 0 96M 0 lvm
│ └─pve-data 251:4 0 185.5G 0 lvm
└─pve-data_tdata 251:3 0 185.5G 0 lvm
└─pve-data 251:4 0 185.5G 0 lvm
sdb 8:16 0 558.7G 0 disk

root@pve4:~# parted /dev/sdb mktable gpt
root@pve4:~# parted /dev/sdb mkpart drbd 1 100%
root@pve4:~# vgcreate drbdpool /dev/sdb1
root@pve4:~# lvcreate -L 550G -n drbdthinpool -T drbdpool

root@pve4:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 279.4G 0 disk
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 127M 0 part
└─sda3 8:3 0 279.2G 0 part
├─pve-root 251:0 0 69.8G 0 lvm /
├─pve-swap 251:1 0 8G 0 lvm [SWAP]
├─pve-data_tmeta 251:2 0 96M 0 lvm
│ └─pve-data 251:4 0 185.5G 0 lvm
└─pve-data_tdata 251:3 0 185.5G 0 lvm
└─pve-data 251:4 0 185.5G 0 lvm
sdb 8:16 0 558.7G 0 disk
└─sdb1 8:17 0 558.7G 0 part
├─drbdpool-drbdthinpool_tmeta 251:5 0 72M 0 lvm
│ └─drbdpool-drbdthinpool 251:7 0 550G 0 lvm
└─drbdpool-drbdthinpool_tdata 251:6 0 550G 0 lvm
└─drbdpool-drbdthinpool 251:7 0 550G 0 lvm

root@pve4:~# pvecm create PVE
root@pve4:~# reboot

root@pve4:~# apt-get install drbdmanage=0.97-1 -y
root@pve4:~# drbdmanage init -q 100.100.0.4
Empty drbdmanage control volume initialized on '/dev/drbd0'.
Empty drbdmanage control volume initialized on '/dev/drbd1'.
Operation completed successfully

root@pve5:~# drbdmanage init -q 100.100.0.5
Empty drbdmanage control volume initialized on '/dev/drbd0'.
Empty drbdmanage control volume initialized on '/dev/drbd1'.
Operation completed successfully

root@pve4:~# drbdmanage add-node -q pve5 100.100.0.5
Operation completed successfully
Operation completed successfully

Executing join command using ssh.
IMPORTANT: The output you see comes from pve5
IMPORTANT: Your input is executed on pve5
Operation completed successfully
root@pve4:~# drbdmanage list-nodes
+------------------------------------------------------------------------------------------------------------+
| Name | Pool Size | Pool Free | | State |
|------------------------------------------------------------------------------------------------------------|
| pve4 | 563200 | 563200 | | ok |
| pve5 | 563200 | 563200 | | ok |
+------------------------------------------------------------------------------------------------------------+

in that way add third node

 

[N9br]

Hi Black Knight MHT,

thanks a lot for your detailed posting.
Looks like what I did - except we do not want a thinpool, thats why I also changed the vol-name. Which should be allowed

In order to find out more I started reading the drbd guidebook
https://www.drbd.org/en/doc/users-guide-90/ch-admin-drbdmanage#s-dm-init-cluster
and one of the first steps is verifying e correct /etc/drbdmanaged.cfg file.
This file is not present on neither of my nodes, and it should come with drbdmanage package.

root@node1:~# ll /etc/drbd*
-rw-r--r-- 1 root root 133 Jul 16 17:14 /etc/drbd.conf

/etc/drbd.d:
total 12
-rw-r--r-- 1 root root 541 Jul 16 17:00 drbdctrl.res_template
-rw-r--r-- 1 root root 211 Jul 16 17:00 drbdmanage-resources.res
-rw-r--r-- 1 root root 2062 Jul 16 17:14 global_common.conf

dpkg-reconfigure is not able to do the magic here.

at least in version 0.95, its supposed to come with the drbdmanage package
https://build.opensuse.org/package/...clustering:Factory/drbdmanage/drbdmanage.spec

drbdmanage 0.97-1 semms to not have it
# dpkg -s drbdmanage
...
Conffiles:
/etc/bash_completion.d/drbdmanage ce667172492feac4ead9f277641c2df7
/etc/dbus-1/system.d/org.drbd.drbdmanaged.conf cb5d4d6d288fe7d6888a2b61f72b2764
/etc/drbd.d/drbdctrl.res_template 4e75122e9b2f7dfaeba78539c360e548
/etc/drbd.d/drbdmanage-resources.res 0bf0e348c51f27e69815cca6e692ead6
...

I was thinking, maybe you could provide a sample drbdmange.cfg file ?
that could help

edit: of course, as with many other configs, /etc/drbd.conf has been split into any files that reside in /etc/drbd.d/ , having *.conf extension, or being included by directive.
Any directive that the docs are talking about will be found in any of the includes files.

thanks
Andre

 

[Black Knight MHT]

I have seen this warning...BUT!....it works without drbdmanage.cfg)))))) I don't know how?!?!
It was some troubles with changing name of lvs. Try default schema.

 

[N9br]

ok now, we'll go for a new, straight setup of the DRBD9 config.
meaning, I did delete-node for all of them, and uninit (on both main nodes, as I had initialized them earlier).
My plan is now to stick to the DRBD9 docs, and try to work with it, and with the Linbit team, and with you, too
Reason is, I need a 100% reliable setup, its not just in lab. Will be for production.

We are going to use ThickLVM.

Now I had some success already. Here is what I did (after uninitializing):

1. make sure I have only an LVM volgroup named 'drbdpool'
2. not create / delete any LVs other than the Proxmox created vols
3. update drbdmange to latest version 0.97.3
4. drbdmanage init -q <Storage Interface>
5. drbdmanage modify-config ; storage-plugin = drbdmanage.storage.lvm.Lvm // LVM Thick if desired
6. drbdmanage list-nodes again showed 'unknown pool size' ; reboot ; now pool size shows correctly. possibly 'drbdmange restart' helps, too.
7. drbdmanage add-node -q node2 <Storage Interface> : Error: Operation not allowed on satellite node
8. drbdmanage init -q <Storage Interface> (on node2 !!)
9. now 'drbdmanage add-node -q node2 <Storage Interface>' works ok
10. Pool Size / Pool Free are shown correctly by 'drbdmanage list-nodes'
11. Creating a Machine in Proxmox on storage drbd1 works . Logs say drbd-sync works. Machine migration possible

Now I only have to find out, how I can add a second pool, bc we want to use 2 volgroups .

Edit: 2 volgroups not possible, says LinBit .

 

[Black Knight MHT]

ОК....And what about adding 3 node after creating VM in PVE? Did You do failover test with HDD?

 

[Black Knight MHT]

Well. I've done tests myself.

a) Fail of hdd (in my situation it was raid0 of 2*300Gb sas 15K drives) on pve5 and vm id 550
On working node:
1. drbdmanage unassign vm-550-disk-1 pve5
2. drbdmanage delete-node -f pve5

On fail node:
1. replace fail drives
2. parted /dev/sda mktable gpt
3. parted /dev/sda mkpart drbd 1 100%
4. vgcreate drbdpool /dev/sda1
5. drbdmanage init -q 100.100.0.5
6. drbdadm create-md all
7. reboot

On working node:
3. drbdmanage add-node pve5 100.100.0.5
4. drbdmanage assign vm-550-disk-1 pve5

Wait for sync.......

root@pve2:~# drbdadm status
.drbdctrl role:Secondary
volume:0 disk:UpToDate
volume:1 disk:UpToDate
pve4 role:Secondary
volume:0 peer-disk:UpToDate
volume:1 peer-disk:UpToDate
pve5 role:Secondary
volume:0 peer-disk:UpToDate
volume:1 peer-disk:UpToDate

vm-550-disk-1 role:Secondary
disk:UpToDate
pve4 rolerimary
peer-disk:UpToDate
pve5 role:Secondary
replication:SyncSource peer-disk:Inconsistent done:73.62

b) Fail of the hole node:
1. reinstall new node.
2. add it to the pve cluster
3. drbdmanage add-node pve5 100.100.0.5
4. drbdmanage assign vm-550-disk-1 pve5

 

[N9br]

I'll be on it.

There is on thing I want to understand once and for all.

Everybody is talking about primary / secondary node. Do I have to - at any point - specify Primary somewhere ?

As far as I can tell from playing with my cluster, Proxmox seems to take care of the primary/secondary bit.
As soon as I move a machine to the other node, this node will become primary for this LV (drbdadm status)

But, at the same time, I see the control Volume '.drbdctrl' is a secondary volume all the time on both nodes.

Is this correct behaviour ?

Will continue with HA now
We are using only 2-Node-DRBD, but 3-Node Proxmox.
see http://pve.proxmox.com/wiki/DRBD9#Alternative_Two_storage_node_setup

thanks