DRBD online verifiy all does not work

[Jospeh Huber]

Hi

we have a fresh 3 Node DRBD9 Cluster setup with Proxmox 4.3 up and runnig like it is described in the Wiki article here https://pve.proxmox.com/wiki/DRBD9.
MyVersions: 4.3-66 (running kernel: 4.4.19-1-pve),drbdmanage: 0.97.3-1

We want to configure some periodical integrity checking as suggested in the wiki article. A cronjob with "drbdadm verify all". (https://pve.proxmox.com/wiki/DRBD9#Integrity_checking)

As described in the documentation here (https://www.drbd.org/en/doc/users-guide-90/s-use-online-verify#s-online-verify-enable, https://www.drbd.org/en/doc/users-guide-90/s-online-verify)
I have defined a verification algorithm for all resources in "/etc/drbd.d/global_common.conf".

If you have enabled on-line verification for all your resources (for example, by adding verify-alg <algorithm> to the common section in /etc/drbd.d/global_common.conf), you may also use...

lsmod | grep crc32c
libcrc32c              16384  2 drbd,dm_persistent_data

/etc/drbd.d/global_common.conf:

  net {
                # protocol timeout max-epoch-size max-buffers unplug-watermark
                # connect-int ping-int sndbuf-size rcvbuf-size ko-count
                # allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
                # after-sb-1pri after-sb-2pri always-asbp rr-conflict
                # ping-timeout data-integrity-alg tcp-cork on-congestion
                # congestion-fill congestion-extents csums-alg verify-alg
                # use-rle
                verify-alg crc32c;

        }

But even after rebooting the system (i read somewhere one has to reboot if the parameter verify-alg changes) the online verification fails.
I also tried the other algorithms "sha1" and "md5", the reason is always the same.
Any ideas?

/usr/sbin/drbdadm verify all
vm-107-disk-1: State change failed: (-14) Need a verify algorithm to start online verify
Command '/usr/sbin/drbdsetup verify vm-107-disk-1 0 0' terminated with exit code 11
vm-132-disk-1: State change failed: (-14) Need a verify algorithm to start online verify
Command '/usr/sbin/drbdsetup verify vm-132-disk-1 1 0' terminated with exit code 11

dmesg:
[58989.267552] drbd .drbdctrl/0 drbd0 vmhost5: repl( Established -> VerifyS )
[58989.267569] drbd .drbdctrl/0 drbd0 vmhost5: Starting Online Verify from sector 0
[58989.268032] drbd .drbdctrl: Preparing cluster-wide state change 2941080310 (1->0 496/288)
[58989.295453] drbd .drbdctrl: State change 2941080310: primary_nodes=0, weak_nodes=0
[58989.295454] drbd .drbdctrl: Committing cluster-wide state change 2941080310 (24ms)
[58989.295462] drbd .drbdctrl/1 drbd1 vmhost1: repl( Established -> VerifyS )
[58989.295466] drbd .drbdctrl/1 drbd1 vmhost1: Starting Online Verify from sector 0
[58989.295942] drbd .drbdctrl: Preparing cluster-wide state change 1522830319 (1->2 496/288)
[58989.303656] drbd .drbdctrl: State change 1522830319: primary_nodes=0, weak_nodes=0
[58989.303658] drbd .drbdctrl: Committing cluster-wide state change 1522830319 (4ms)
[58989.303664] drbd .drbdctrl/1 drbd1 vmhost5: repl( Established -> VerifyS )
[58989.303667] drbd .drbdctrl/1 drbd1 vmhost5: Starting Online Verify from sector 0
[58989.304064] drbd vm-107-disk-1: State change failed: Need a verify algorithm to start online verify
[58989.304296] drbd vm-107-disk-1/0 drbd102 vmhost1: Failed: repl( Established -> VerifyS )
[58989.304618] drbd vm-132-disk-1: State change failed: Need a verify algorithm to start online verify
[58989.304844] drbd vm-132-disk-1/0 drbd101 vmhost1: Failed: repl( Established -> VerifyS )

 

[Dubard]

Hi Joseph,

I've the same problem. We have a fresh 2 Nodes DRBD Cluster setup With Proxmox 4-2-23 (https://pve.proxmox.com/wiki/Two-Node_High_Availability_Cluster)

Proxmox node information:

Virtual Environment 4.2-23/d94f9458
Nœud 'node2'
Utilisation CPU
0.35% of 16 CPU(s)
Délai E/S
0.00%
Charge système
0.08,0.06,0.01
Utilisation Mémoire Vive (RAM)
18.43% (2.86 GiB of 15.54 GiB)
KSM sharing
0 B
Espace disque(root)
23.08% (6.27 GiB of 27.19 GiB)
Utilisation Swap
0.00% (0 B of 8.00 GiB)
CPUs
16 x Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (1 Socket)
Kernel Version
Linux 4.4.19-1-pve #1 SMP Wed Sep 14 14:33:50 CEST 2016
PVE Manager Version
pve-manager/4.2-23/d94f9458
Logs
()
proxmox-ve: 4.2-66 (running kernel: 4.4.19-1-pve)
pve-manager: 4.2-23 (running version: 4.2-23/d94f9458)
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.4.19-1-pve: 4.4.19-66
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-46
qemu-server: 4.0-88
pve-firmware: 1.1-9
libpve-common-perl: 4.0-73
libpve-access-control: 4.0-19
libpve-storage-perl: 4.0-61
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-qemu-kvm: 2.6.1-6
pve-container: 1.0-75
pve-firewall: 2.0-29
pve-ha-manager: 1.0-35
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 2.0.4-1
lxcfs: 2.0.3-pve1
criu: 1.6.0-1
novnc-pve: 0.5-8
zfsutils: 0.6.5.7-pve10~bpo80
drbdmanage: 0.97.3-1

root@node2:~# drbdadm verify all
.drbdctrl: State change failed: (-14) Need a verify algorithm to start online verify
Command 'drbdsetup verify .drbdctrl 0 0' terminated with exit code 11
vm-100-disk-1: State change failed: (-14) Need a verify algorithm to start online verify
Command 'drbdsetup verify vm-100-disk-1 0 0' terminated with exit code 11
vm-101-disk-1: State change failed: (-14) Need a verify algorithm to start online verify
Command 'drbdsetup verify vm-101-disk-1 0 0' terminated with exit code 11
vm-102-disk-1: State change failed: (-14) Need a verify algorithm to start online verify
Command 'drbdsetup verify vm-102-disk-1 0 0' terminated with exit code 11
root@node2:~#

all ideas are welcome ;-)

 

[Jospeh Huber]

It seems that there is no solution for this...
To the Moderators: can you please submit a bug for this?

 

[tom]

It seems that there is no solution for this...
To the Moderators: can you please submit a bug for this?

Seems you used the wrong doc.

For Proxmox VE 4.x, you need to go for this one:

https://pve.proxmox.com/wiki/DRBD9

 

[laci_ur]

hi,

i guess you are using drbd veersion 9+ and more important you configured it using drbdmanage. Try setting the verify algo using

drbdmanage net-options --verify-alg crc32c --common

It gets replicated among drbd nodes.

 

[Jospeh Huber]

@tom: No, as I have posted in my question I used the Wiki Topic for drbd9 and also the documentation for drbd9 located on drbd9.org... I have configured it as described here: https://www.drbd.org/en/doc/users-g...rbd.org/en/doc/users-guide-90/s-online-verify)

@laci_ur:
In the man drbd.conf "NOTES ON DATAINTEGRITY":
"DRBD supports two different mechanisms for data integrity checking: first, the data-integrity-alg network parameter allows to
add a checksum to the data sent over the network. Second, the online verification mechanism (drbdadm verify and the verify-alg
parameter) allows to check for differences in the on-disk data..."

Does the command "drbdmanage net-options --verify-alg crc32c --common"
a) activate the data integrity checking over the network or (that ist NOT what i want)
b) activate online verification mechanism (that is what i want)
?

I cannot find anything in the docs "man drbdmanage-net-options"...

 

[laci_ur]

@tom: No, as I have posted in my question I used the Wiki Topic for drbd9 and also the documentation for drbd9 located on drbd9.org... I have configured it as described here: https://www.drbd.org/en/doc/users-g...rbd.org/en/doc/users-guide-90/s-online-verify)

@laci_ur:
In the man drbd.conf "NOTES ON DATAINTEGRITY":
"DRBD supports two different mechanisms for data integrity checking: first, the data-integrity-alg network parameter allows to
add a checksum to the data sent over the network. Second, the online verification mechanism (drbdadm verify and the verify-alg
parameter) allows to check for differences in the on-disk data..."

Does the command "drbdmanage net-options --verify-alg crc32c --common"
a) activate the data integrity checking over the network or (that ist NOT what i want)
b) activate online verification mechanism (that is what i want)
?

I cannot find anything in the docs "man drbdmanage-net-options"...

According here :https://www.drbd.org/en/doc/users-guide-90/s-dm-setupopts it should the verify-alg option, and the entry is also visible in the config file. You can run #drbdadm verify all and the online verification will start.

 

[Jospeh Huber]

@laci_ur: Thanks a lot, the verification is running now.
I set the verification alg before every verification, because after a reboot the setting gets lost.
drbdmanage net-options --verify-alg crc32c --common
drbdadm verify all


Has anyone experience which verify alg is the fastest?
In the post here is written that crc32c should be the fastest on modern cpus...
http://www.gossamer-threads.com/lists/drbd/users/20277

 

[laci_ur]

On my particular setup i have sha1 set up, and i never tried crc32c. I will give it a shot sometime, but since thespeed of the verification is affected by the utilization of the resources, i can't give you a baseline for this algo. I can tell that the progress shows 100% from the start(meh), and it takes a very long time.. I have a 2 node drbd setup, with a linux bond consisting of 3 gbe connections (in RR mode). verifying 4 resources consisting of ~50gb each is usually done within a day (dual xeon x5550, 32G RAM)