DRBD8 to DRBD9 upgrade

[hjjg]

Hi,


after upgrading to Proxmox 4.0, my DRBD volume is missing. It was set up with the DRBD howto from the Proxmox wiki (dual primary, but the two resources were never primary on both machines)

This particular cluster is not in production but I also plan to upgrade another pair of servers and I don't want to loose the DRBD volumes.
How to upgrade without loosing the volumes and how do I get them back?

I also installed drbdmanage, but did not issue any commands. drbd-overview says:

0:r0/0  Connected(2*) Secondary(2*) Diskle/Diskle
1:r1/0  Connected(2*) Secondary(2*) Diskle/Diskle

I am not able to fix this. I tried the upgrade guide from Linbit (drbd.linbit.com /users-guide-9.0/s-upgrading-drbd.html#s-upgrade-convert)
drbdadm create-md r0 upgraded the meta-data as described but this did not help. This is the output from dmesg:

[12698.189819] drbd r0: Preparing cluster-wide state change 3099231762 (0->1 499/146)
[12698.189975] drbd r0: State change 3099231762: primary_nodes=0, weak_nodes=0
[12698.189977] drbd r0: Committing cluster-wide state change 3099231762 (0ms)
[12698.189985] drbd r0 virt02: conn( Connecting -> Connected ) peer( Unknown -> Secondary )
[12698.197844] drbd r0/0 drbd0 virt02: pdsk( DUnknown -> Diskless ) repl( Off -> Established )
[12698.233780] drbd r1: Preparing cluster-wide state change 1831514047 (0->1 499/146)
[12698.234038] drbd r1: State change 1831514047: primary_nodes=0, weak_nodes=0
[12698.234040] drbd r1: Committing cluster-wide state change 1831514047 (0ms)
[12698.234050] drbd r1 virt02: conn( Connecting -> Connected ) peer( Unknown -> Secondary )
[12698.241823] drbd r1/0 drbd1 virt02: pdsk( DUnknown -> Diskless ) repl( Off -> Established )

Thanks in advance!

 

[dietmar]

You cannot upgrade DRBD8 system - the new DRBD9 code is not compatible. I would use backup/restore instead.

 

[hjjg]

What a pity. Thanks for your reply!

 

[psuter]

maybe things have changed since the original post. i just did an upgrade from drbd 8 to 9 by upgrading from proxmox 3.4 to 4.3 and it worked. it also states in the drbd 9 documentation, that it is compatible with drbd 8.4.

in order to prepare i first upgraded my proxmox 3.4 to the latest 3.10.x kernel which installes drbd 8.4. i then had to manually compile the drbd tools for 8.4. all this according to the drbd manual, so no big deal. i then converted the configs by using drbdadm dump so they did not contain any obsolete options anymore.

after that i did the upgrade of proxmox to version 4.3 according to the instructions in the proxmox wiki. I did all this on my first node, while the second node kept all the vm's running. after the upgrade i converted the metadata format of my drbd disk. i then started drbd and got the same diskless state as the OP described.

what helped in my case was, that i re-generated the config files using the drbdadm dump command. i then had to swap the node-id's between the two servers, so that my first server which was also the one i upgraded first, had node-id 0 and the other had node-id 1. after that the disk was recognized again and drbd started to sync between the other server and the newly upgraded server.

since the sync between one drbd 8.4 and one drbd 9 machine worked flawlessly, the diskless state could probably also be resolved by runnig the
drbdadm create-md command a second time and saying yes to erasing the metadata. this would void all data on the disk and it will need a full re-sync. but again, this should probably work jsut fine, as syncing between an old and new drbd seems to be no problem at all.

once i had my drbd back in sync i had to shut down the various vm's and then manually move the config files for the vm's on my updated machine. i could then simply power on all the vm's and they ran just fine.

after that i updated the second host and there i also ra into the diskless state.. i could not resolve it this time, even though i copied the drbd files that where working on the first host to my second. i then decided to simply erase the meatadata as described above and run a re-sync.
this is taking very long. it is still syncing at the moment but looks good so far.

just wanted to let you know, that obviously drbd 8 and 9 are compatible after all and will sync with one another during an upgrade. i am pretty sure that the upgrade to drbd 8.4 before the move to drbd 9 is a good idea though or even mandatory

regards
Pascal

 

[psuter]

okay this needs a self-reply now.. I might have bee slightly over enthusiastic turns out I've missed an important note or warning in the drbd9 documentation:
"In DRBD 9.0.x Dual-Primary mode is not supported. It might work "by accident", but don’t rely on it. Because of that, changes in the configuration are very likely"

unfortunately drbd 9.1 isn't released yet and so proxmox uses 9.0.x as well. which means, the "old" setup with drbd in dual primary mode and LVM on top of it is just simply not possible with drbd 9.0
It worked a short moment for me, but all of a sudden, one of the two nodes took over as primary and the other went to secondary mode and all vm's on that machine chrashed.. as a matter of fact, the entire host server crashed with a kernel panic of some sort.

From what I understand, proxmox 4 uses drbd on top of lvm, so it creates a drbd volume for every vm. The setup I (and probably mos people who did a pre proxmox 4 setup with drbd) have is the opposite: we have a single drbd volume and on top of that multiple lvm volumes, hence the "old" setup needs dual primary mode whereas the "new" setup doesn't.

i'll downgrade my drbd to 8.4 again and will keep this setup until either drbd 9.1 is released or i finally make the move to ceph
Here are some instructions that might help for that:
http://coolsoft.altervista.org/it/b...rnel-panic-downgrade-drbd-resources-drbd-9-84