1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

DRBD online verifiy all does not work

Discussion in 'Proxmox VE: Installation and configuration' started by Jospeh Huber, Oct 5, 2016.

Tags:
  1. Jospeh Huber

    Jospeh Huber New Member

    Joined:
    Apr 18, 2016
    Messages:
    27
    Likes Received:
    2
    Hi

    we have a fresh 3 Node DRBD9 Cluster setup with Proxmox 4.3 up and runnig like it is described in the Wiki article here https://pve.proxmox.com/wiki/DRBD9.
    MyVersions: 4.3-66 (running kernel: 4.4.19-1-pve),drbdmanage: 0.97.3-1

    We want to configure some periodical integrity checking as suggested in the wiki article. A cronjob with "drbdadm verify all". (https://pve.proxmox.com/wiki/DRBD9#Integrity_checking)

    As described in the documentation here (https://www.drbd.org/en/doc/users-guide-90/s-use-online-verify#s-online-verify-enable, https://www.drbd.org/en/doc/users-guide-90/s-online-verify)
    I have defined a verification algorithm for all resources in "/etc/drbd.d/global_common.conf".

    Code:
    lsmod | grep crc32c
    libcrc32c              16384  2 drbd,dm_persistent_data
    
    /etc/drbd.d/global_common.conf:
    
      net {
                    # protocol timeout max-epoch-size max-buffers unplug-watermark
                    # connect-int ping-int sndbuf-size rcvbuf-size ko-count
                    # allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
                    # after-sb-1pri after-sb-2pri always-asbp rr-conflict
                    # ping-timeout data-integrity-alg tcp-cork on-congestion
                    # congestion-fill congestion-extents csums-alg verify-alg
                    # use-rle
                    verify-alg crc32c;
    
            }
    
    
    But even after rebooting the system (i read somewhere one has to reboot if the parameter verify-alg changes) the online verification fails.
    I also tried the other algorithms "sha1" and "md5", the reason is always the same.
    Any ideas?

    Code:
    /usr/sbin/drbdadm verify all
    vm-107-disk-1: State change failed: (-14) Need a verify algorithm to start online verify
    Command '/usr/sbin/drbdsetup verify vm-107-disk-1 0 0' terminated with exit code 11
    vm-132-disk-1: State change failed: (-14) Need a verify algorithm to start online verify
    Command '/usr/sbin/drbdsetup verify vm-132-disk-1 1 0' terminated with exit code 11
    
    dmesg:
    [58989.267552] drbd .drbdctrl/0 drbd0 vmhost5: repl( Established -> VerifyS )
    [58989.267569] drbd .drbdctrl/0 drbd0 vmhost5: Starting Online Verify from sector 0
    [58989.268032] drbd .drbdctrl: Preparing cluster-wide state change 2941080310 (1->0 496/288)
    [58989.295453] drbd .drbdctrl: State change 2941080310: primary_nodes=0, weak_nodes=0
    [58989.295454] drbd .drbdctrl: Committing cluster-wide state change 2941080310 (24ms)
    [58989.295462] drbd .drbdctrl/1 drbd1 vmhost1: repl( Established -> VerifyS )
    [58989.295466] drbd .drbdctrl/1 drbd1 vmhost1: Starting Online Verify from sector 0
    [58989.295942] drbd .drbdctrl: Preparing cluster-wide state change 1522830319 (1->2 496/288)
    [58989.303656] drbd .drbdctrl: State change 1522830319: primary_nodes=0, weak_nodes=0
    [58989.303658] drbd .drbdctrl: Committing cluster-wide state change 1522830319 (4ms)
    [58989.303664] drbd .drbdctrl/1 drbd1 vmhost5: repl( Established -> VerifyS )
    [58989.303667] drbd .drbdctrl/1 drbd1 vmhost5: Starting Online Verify from sector 0
    [58989.304064] drbd vm-107-disk-1: State change failed: Need a verify algorithm to start online verify
    [58989.304296] drbd vm-107-disk-1/0 drbd102 vmhost1: Failed: repl( Established -> VerifyS )
    [58989.304618] drbd vm-132-disk-1: State change failed: Need a verify algorithm to start online verify
    [58989.304844] drbd vm-132-disk-1/0 drbd101 vmhost1: Failed: repl( Established -> VerifyS )
    
     
    #1 Jospeh Huber, Oct 5, 2016
    Last edited: Oct 5, 2016
  2. Dubard

    Dubard New Member

    Joined:
    Oct 5, 2016
    Messages:
    2
    Likes Received:
    0
    Hi Joseph,

    I've the same problem. We have a fresh 2 Nodes DRBD Cluster setup With Proxmox 4-2-23 (https://pve.proxmox.com/wiki/Two-Node_High_Availability_Cluster)

    Proxmox node information:

    Code:
    Virtual Environment 4.2-23/d94f9458
    Nœud 'node2'
    Utilisation CPU
    0.35% of 16 CPU(s)
    Délai E/S
    0.00%
    Charge système
    0.08,0.06,0.01
    Utilisation Mémoire Vive (RAM)
    18.43% (2.86 GiB of 15.54 GiB)
    KSM sharing
    0 B
    Espace disque(root)
    23.08% (6.27 GiB of 27.19 GiB)
    Utilisation Swap
    0.00% (0 B of 8.00 GiB)
    CPUs
    16 x Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz (1 Socket)
    Kernel Version
    Linux 4.4.19-1-pve #1 SMP Wed Sep 14 14:33:50 CEST 2016
    PVE Manager Version
    pve-manager/4.2-23/d94f9458
    Logs
    ()
    proxmox-ve: 4.2-66 (running kernel: 4.4.19-1-pve)
    pve-manager: 4.2-23 (running version: 4.2-23/d94f9458)
    pve-kernel-4.4.6-1-pve: 4.4.6-48
    pve-kernel-4.4.19-1-pve: 4.4.19-66
    lvm2: 2.02.116-pve3
    corosync-pve: 2.4.0-1
    libqb0: 1.0-1
    pve-cluster: 4.0-46
    qemu-server: 4.0-88
    pve-firmware: 1.1-9
    libpve-common-perl: 4.0-73
    libpve-access-control: 4.0-19
    libpve-storage-perl: 4.0-61
    pve-libspice-server1: 0.12.8-1
    vncterm: 1.2-1
    pve-qemu-kvm: 2.6.1-6
    pve-container: 1.0-75
    pve-firewall: 2.0-29
    pve-ha-manager: 1.0-35
    ksm-control-daemon: 1.2-1
    glusterfs-client: 3.5.2-2+deb8u2
    lxc-pve: 2.0.4-1
    lxcfs: 2.0.3-pve1
    criu: 1.6.0-1
    novnc-pve: 0.5-8
    zfsutils: 0.6.5.7-pve10~bpo80
    drbdmanage: 0.97.3-1
    
    Code:
    root@node2:~# drbdadm verify all
    .drbdctrl: State change failed: (-14) Need a verify algorithm to start online verify
    Command 'drbdsetup verify .drbdctrl 0 0' terminated with exit code 11
    vm-100-disk-1: State change failed: (-14) Need a verify algorithm to start online verify
    Command 'drbdsetup verify vm-100-disk-1 0 0' terminated with exit code 11
    vm-101-disk-1: State change failed: (-14) Need a verify algorithm to start online verify
    Command 'drbdsetup verify vm-101-disk-1 0 0' terminated with exit code 11
    vm-102-disk-1: State change failed: (-14) Need a verify algorithm to start online verify
    Command 'drbdsetup verify vm-102-disk-1 0 0' terminated with exit code 11
    root@node2:~#
    
    all ideas are welcome ;-)
     
    #2 Dubard, Oct 5, 2016
    Last edited: Oct 5, 2016
  3. Jospeh Huber

    Jospeh Huber New Member

    Joined:
    Apr 18, 2016
    Messages:
    27
    Likes Received:
    2
    It seems that there is no solution for this...
    To the Moderators: can you please submit a bug for this?
     
  4. tom

    tom Proxmox Staff Member
    Staff Member

    Joined:
    Aug 29, 2006
    Messages:
    11,964
    Likes Received:
    190
    Seems you used the wrong doc.

    For Proxmox VE 4.x, you need to go for this one:

    https://pve.proxmox.com/wiki/DRBD9
     
  5. laci_ur

    laci_ur New Member

    Joined:
    Oct 25, 2016
    Messages:
    3
    Likes Received:
    2
    hi,

    i guess you are using drbd veersion 9+ and more important you configured it using drbdmanage. Try setting the verify algo using
    Code:
    drbdmanage net-options --verify-alg crc32c --common
    
    It gets replicated among drbd nodes.
     
  6. Jospeh Huber

    Jospeh Huber New Member

    Joined:
    Apr 18, 2016
    Messages:
    27
    Likes Received:
    2
    @tom: No, as I have posted in my question I used the Wiki Topic for drbd9 and also the documentation for drbd9 located on drbd9.org... I have configured it as described here: https://www.drbd.org/en/doc/users-g...rbd.org/en/doc/users-guide-90/s-online-verify)

    @laci_ur:
    In the man drbd.conf "NOTES ON DATAINTEGRITY":
    "DRBD supports two different mechanisms for data integrity checking: first, the data-integrity-alg network parameter allows to
    add a checksum to the data sent over the network. Second, the online verification mechanism (drbdadm verify and the verify-alg
    parameter) allows to check for differences in the on-disk data..."


    Does the command "drbdmanage net-options --verify-alg crc32c --common"
    a) activate the data integrity checking over the network or (that ist NOT what i want)
    b) activate online verification mechanism (that is what i want)
    ?

    I cannot find anything in the docs "man drbdmanage-net-options"...
     
  7. laci_ur

    laci_ur New Member

    Joined:
    Oct 25, 2016
    Messages:
    3
    Likes Received:
    2
    According here :https://www.drbd.org/en/doc/users-guide-90/s-dm-setupopts it should the verify-alg option, and the entry is also visible in the config file. You can run #drbdadm verify all and the online verification will start.
     
    Jospeh Huber likes this.
  8. Jospeh Huber

    Jospeh Huber New Member

    Joined:
    Apr 18, 2016
    Messages:
    27
    Likes Received:
    2
    @laci_ur: Thanks a lot, the verification is running now.
    I set the verification alg before every verification, because after a reboot the setting gets lost.
    drbdmanage net-options --verify-alg crc32c --common
    drbdadm verify all


    Has anyone experience which verify alg is the fastest?
    In the post here is written that crc32c should be the fastest on modern cpus...
    http://www.gossamer-threads.com/lists/drbd/users/20277
     
  9. laci_ur

    laci_ur New Member

    Joined:
    Oct 25, 2016
    Messages:
    3
    Likes Received:
    2
    On my particular setup i have sha1 set up, and i never tried crc32c. I will give it a shot sometime, but since thespeed of the verification is affected by the utilization of the resources, i can't give you a baseline for this algo. I can tell that the progress shows 100% from the start(meh), and it takes a very long time.. I have a 2 node drbd setup, with a linux bond consisting of 3 gbe connections (in RR mode). verifying 4 resources consisting of ~50gb each is usually done within a day (dual xeon x5550, 32G RAM)
     
    Jospeh Huber likes this.

Share This Page