DRBD9: both nodes outdated

[metaplop]

Hello,

I have a problem on two nodes with PVE4 and DRBD9 as both thinks that disks are outdated and I do not manage to fix things.

Both nodes are up to date with pve-no-subscription:

# cat /proc/drbd
version: 9.0.3-1 (api:2/proto:86-111)

ii  drbd-utils  8.9.7-1  amd64  RAID 1 over TCP/IP for Linux (use
ii  drbdmanage  0.97-1  amd64  Distributed configuration managem

on first node edfu:

root@edfu:~# drbd-overview
  0:.drbdctrl/0  Connected(2*) Secondary(2*) UpToDa/UpToDa
  1:.drbdctrl/1  Connected(2*) Secondary(2*) UpToDa/UpToDa
102:vm-221-disk-1/0  Connec/C'ting Second/Unknow Outdat/Incons
105:vm-221-disk-2/0  Connected(2*) Secondary(2*) UpToDa/UpToDa

root@edfu:~# drbdsetup status --verbose --statistics
vm-221-disk-1 node-id:1 role:Secondary suspended:no
  write-ordering:flush
  volume:0 minor:102 disk:Outdated
  size:33554432 read:0 written:0 al-writes:0 bm-writes:0 upper-pending:0
  lower-pending:0 al-suspended:no blocked:no
  tuat node-id:0 connection:Connecting role:Unknown congested:no
  volume:0 replication:Off peer-disk:Inconsistent resync-suspended:no
  received:0 sent:0 out-of-sync:106496 pending:0 unacked:0

vm-221-disk-2 node-id:1 role:Secondary suspended:no
  write-ordering:flush
  volume:0 minor:105 disk:UpToDate
  size:67108864 read:59016 written:0 al-writes:0 bm-writes:0 upper-pending:0
  lower-pending:0 al-suspended:no blocked:no
  tuat node-id:0 connection:Connected role:Secondary congested:no
  volume:0 replication:Established peer-disk:UpToDate
  resync-suspended:no
  received:0 sent:0 out-of-sync:0 pending:0 unacked:0

on second node tuat:

root@tuat:~# drbd-overview
  0:.drbdctrl/0  Connected(2*) Secondary(2*) UpToDa/UpToDa
  1:.drbdctrl/1  Connected(2*) Secondary(2*) UpToDa/UpToDa
102:vm-221-disk-1/0  Connec/StaAlo Second/Unknow Outdat/Outdat
105:vm-221-disk-2/0  Connected(2*) Secondary(2*) UpToDa/UpToDa

root@tuat:~# drbdsetup status --verbose --statistics
vm-221-disk-1 node-id:0 role:Secondary suspended:no
  write-ordering:flush
  volume:0 minor:102 disk:Outdated
  size:33554432 read:0 written:0 al-writes:0 bm-writes:0 upper-pending:0
  lower-pending:0 al-suspended:no blocked:no
  edfu node-id:1 connection:StandAlone role:Unknown congested:no
  volume:0 replication:Off peer-disk:Outdated resync-suspended:no
  received:0 sent:0 out-of-sync:106496 pending:0 unacked:0

vm-221-disk-2 node-id:0 role:Secondary suspended:no
  write-ordering:flush
  volume:0 minor:105 disk:UpToDate
  size:67108864 read:60436 written:0 al-writes:0 bm-writes:0 upper-pending:0
  lower-pending:0 al-suspended:no blocked:no
  edfu node-id:1 connection:Connected role:Secondary congested:no
  volume:0 replication:Established peer-disk:UpToDate
  resync-suspended:no
  received:0 sent:0 out-of-sync:0 pending:0 unacked:0

I think that tuat has the most up-to-date disk but I do not manage to set it as consistent event with discard-my-data... Any help would be appreciated

 

[metaplop]

I rebooted both nodes simultaneously. A sync occured and now resources are connected, consistent but outdated on both side:

root@edfu:~# drbdsetup status --verbose --statistics vm-221-disk-1
vm-221-disk-1 node-id:1 role:Secondary suspended:no
  write-ordering:flush
  volume:0 minor:102 disk:Outdated
  size:33554432 read:106496 written:0 al-writes:0 bm-writes:0
  upper-pending:0 lower-pending:0 al-suspended:no blocked:no
  tuat node-id:0 connection:Connected role:Secondary congested:no
  volume:0 replication:Established peer-disk:Outdated
  resync-suspended:no
  received:0 sent:106496 out-of-sync:0 pending:0 unacked:0

root@tuat:~# drbdsetup status --verbose --statistics vm-221-disk-1
vm-221-disk-1 node-id:0 role:Secondary suspended:no
  write-ordering:flush
  volume:0 minor:102 disk:Outdated
  size:33554432 read:0 written:106496 al-writes:0 bm-writes:0
  upper-pending:0 lower-pending:0 al-suspended:no blocked:no
  edfu node-id:1 connection:Connected role:Secondary congested:no
  volume:0 replication:Established peer-disk:Outdated
  resync-suspended:no
  received:106496 sent:0 out-of-sync:0 pending:0 unacked:0

How can I fix that ? I tried to overwrite-data-of-peer without succcess:

root@edfu:~# drbdadm -- --overwrite-data-of-peer primary vm-221-disk-1
vm-221-disk-1: State change failed: (-2) Need access to UpToDate data
Command 'drbdsetup primary vm-221-disk-1' terminated with exit code 17

 

[resoli]

Try with:

drbdsetup primary --force  vm-221-disk-1

rob

 

[metaplop]

Thanks that did it ! I was able to backup/dump my last vm to migrate to zfs storage...

 

[albert_a]

drbdadm -- --overwrite-data-of-peer primary vm-221-disk-1
drbdsetup primary --force  vm-221-disk-1

None of this commands worked for me until I brought down the resource on the other nodes. After that, first one succeeded, I didn't try the second;