Error: I/O error while accessing persistent configuration storage

C

cdukes

Renowned Member
Sep 11, 2015
89
10
73
Raleigh, NC
www.logzilla.net
I keep getting this error when trying to create my first drbd disk(s). Can anyone please explain how I can fix it?

drbdmanage add-node ucs 10.0.28.201
Operation completed successfully
Operation completed successfully


Executing join command using ssh.
IMPORTANT: The output you see comes from ucs
IMPORTANT: Your input is executed on ucs
You are going to join an existing drbdmanage cluster.
CAUTION! Note that:
* Any previous drbdmanage cluster information may be removed
* Any remaining resources managed by a previous drbdmanage installation
that still exist on this system will no longer be managed by drbdmanage


Confirm:


yes/no: yes
Logical volume ".drbdctrl" successfully removed
Logical volume ".drbdctrl" created.
You want me to create a v09 style flexible-size internal meta data block.
There appears to be a v09 flexible-size internal meta data block
already in place on /dev/drbdpool/.drbdctrl at byte offset 4190208


Do you really want to overwrite the existing meta-data?
*** confirmation forced via --force option ***
NOT initializing bitmap
initializing activity log
Writing meta data...
New drbd meta data block successfully created.
Error: I/O error while accessing persistent configuration storage
 
Last edited:
I just noticed the following in the 2nd server's log after I run drbdmanage new-node ucs 10.0.28.201

Sep 21 20:56:53 ucs kernel: [890029.220380] drbd .drbdctrl bfs: Terminating sender thread
Sep 21 20:56:53 ucs kernel: [890029.253037] drbd .drbdctrl/0 drbd0: disk( Inconsistent -> Detaching )
Sep 21 20:56:53 ucs kernel: [890029.253070] drbd .drbdctrl/0 drbd0: disk( Detaching -> Diskless )
Sep 21 20:56:53 ucs kernel: [890029.253082] drbd .drbdctrl/0 drbd0: drbd_bm_resize called with capacity == 0
Sep 21 20:56:53 ucs kernel: [890029.324997] drbd .drbdctrl: Terminating worker thread
Sep 21 20:56:54 ucs kernel: [890030.184384] drbd .drbdctrl: Starting worker thread (from drbdsetup [13403])
Sep 21 20:56:55 ucs kernel: [890030.388995] drbd .drbdctrl/0 drbd0: disk( Diskless -> Attaching )
Sep 21 20:56:55 ucs kernel: [890030.389011] drbd .drbdctrl/0 drbd0: Maximum number of peer devices = 31
Sep 21 20:56:55 ucs kernel: [890030.389110] drbd .drbdctrl: Method to ensure write ordering: flush
Sep 21 20:56:55 ucs kernel: [890030.389116] drbd .drbdctrl/0 drbd0: drbd_bm_resize called with capacity == 8112
Sep 21 20:56:55 ucs kernel: [890030.389125] drbd .drbdctrl/0 drbd0: resync bitmap: bits=1014 words=496 pages=1
Sep 21 20:56:55 ucs kernel: [890030.389261] drbd .drbdctrl/0 drbd0: recounting of set bits took additional 0ms
Sep 21 20:56:55 ucs kernel: [890030.389274] drbd .drbdctrl/0 drbd0: disk( Attaching -> Inconsistent )
Sep 21 20:56:55 ucs kernel: [890030.389278] drbd .drbdctrl/0 drbd0: attached to current UUID: 0000000000000004
Sep 21 20:56:55 ucs kernel: [890030.395554] drbd .drbdctrl bfs: Starting sender thread (from drbdsetup [13417])
Sep 21 20:56:55 ucs kernel: [890030.406618] drbd .drbdctrl bfs: conn( StandAlone -> Unconnected )
Sep 21 20:56:55 ucs kernel: [890030.407761] drbd .drbdctrl bfs: Starting receiver thread (from drbd_w_.drbdctr [13405])
Sep 21 20:56:55 ucs kernel: [890030.407900] drbd .drbdctrl bfs: conn( Unconnected -> Connecting )
Sep 21 20:56:55 ucs kernel: [890030.409998] drbd .drbdctrl: State change failed: Need access to UpToDate data
Sep 21 20:56:55 ucs kernel: [890030.410082] drbd .drbdctrl: Failed: role( Secondary -> Primary )
Sep 21 20:56:55 ucs kernel: [890030.410085] drbd .drbdctrl: Auto-promote failed: Need access to UpToDate data
Sep 21 20:56:55 ucs kernel: [890030.906404] drbd .drbdctrl bfs: Handshake successful: Agreed network protocol version 110
Sep 21 20:56:55 ucs kernel: [890030.906408] drbd .drbdctrl bfs: Agreed to support TRIM on protocol level
Sep 21 20:56:55 ucs kernel: [890030.906600] drbd .drbdctrl bfs: Peer authenticated using 32 bytes HMAC
Sep 21 20:56:55 ucs kernel: [890030.906619] drbd .drbdctrl bfs: Starting ack_recv thread (from drbd_r_.drbdctr [13423])
Sep 21 20:56:55 ucs kernel: [890030.916877] drbd .drbdctrl bfs: incompatible after-sb-0pri settings
Sep 21 20:56:55 ucs kernel: [890030.916950] drbd .drbdctrl bfs: conn( Connecting -> Disconnecting )
Sep 21 20:56:55 ucs kernel: [890030.916979] drbd .drbdctrl bfs: error receiving P_PROTOCOL, e: -5 l: 1!
Sep 21 20:56:55 ucs kernel: [890030.917056] drbd .drbdctrl bfs: ack_receiver terminated
Sep 21 20:56:55 ucs kernel: [890030.917058] drbd .drbdctrl bfs: Terminating ack_recv thread
Sep 21 20:56:55 ucs kernel: [890030.964934] drbd .drbdctrl bfs: Connection closed
Sep 21 20:56:55 ucs kernel: [890030.964982] drbd .drbdctrl bfs: conn( Disconnecting -> StandAlone )
Sep 21 20:56:55 ucs kernel: [890030.965017] drbd .drbdctrl bfs: Terminating receiver thread
Sep 21 20:56:56 ucs kernel: [890031.451268] drbd .drbdctrl: State change failed: Need access to UpToDate data
Sep 21 20:56:56 ucs kernel: [890031.451355] drbd .drbdctrl: Failed: role( Secondary -> Primary )
Sep 21 20:56:56 ucs kernel: [890031.451359] drbd .drbdctrl: Auto-promote failed: Need access to UpToDate data
Sep 21 20:56:57 ucs kernel: [890032.432546] drbd .drbdctrl: State change failed: Need access to UpToDate data
Sep 21 20:56:57 ucs kernel: [890032.432633] drbd .drbdctrl: Failed: role( Secondary -> Primary )
Sep 21 20:56:57 ucs kernel: [890032.432637] drbd .drbdctrl: Auto-promote failed: Need access to UpToDate data
Sep 21 20:56:58 ucs kernel: [890033.824265] drbd .drbdctrl: State change failed: Need access to UpToDate data
Sep 21 20:56:58 ucs kernel: [890033.824353] drbd .drbdctrl: Failed: role( Secondary -> Primary )
Sep 21 20:56:58 ucs kernel: [890033.824356] drbd .drbdctrl: Auto-promote failed: Need access to UpToDate data
Sep 21 20:57:01 ucs kernel: [890036.587404] drbd .drbdctrl: State change failed: Need access to UpToDate data
Sep 21 20:57:01 ucs kernel: [890036.587493] drbd .drbdctrl: Failed: role( Secondary -> Primary )
Sep 21 20:57:01 ucs kernel: [890036.587497] drbd .drbdctrl: Auto-promote failed: Need access to UpToDate data
Sep 21 20:57:02 ucs kernel: [890038.148296] drbd .drbdctrl: State change failed: Need access to UpToDate data
Sep 21 20:57:02 ucs kernel: [890038.148384] drbd .drbdctrl: Failed: role( Secondary -> Primary )
Sep 21 20:57:02 ucs kernel: [890038.148388] drbd .drbdctrl: Auto-promote failed: Need access to UpToDate data
Sep 21 20:57:03 ucs kernel: [890038.689072] drbd .drbdctrl: State change failed: Need access to UpToDate data
Sep 21 20:57:03 ucs kernel: [890038.689155] drbd .drbdctrl: Failed: role( Secondary -> Primary )
Sep 21 20:57:03 ucs kernel: [890038.689172] drbd .drbdctrl: Auto-promote failed: Need access to UpToDate data
Sep 21 20:57:05 ucs kernel: [890040.461145] drbd .drbdctrl: State change failed: Need access to UpToDate data
Sep 21 20:57:05 ucs kernel: [890040.461234] drbd .drbdctrl: Failed: role( Secondary -> Primary )
Sep 21 20:57:05 ucs kernel: [890040.461252] drbd .drbdctrl: Auto-promote failed: Need access to UpToDate data
Sep 21 20:57:06 ucs kernel: [890041.662641] drbd .drbdctrl: State change failed: Need access to UpToDate data
Sep 21 20:57:06 ucs kernel: [890041.662729] drbd .drbdctrl: Failed: role( Secondary -> Primary )
Sep 21 20:57:06 ucs kernel: [890041.662733] drbd .drbdctrl: Auto-promote failed: Need access to UpToDate data
Sep 21 20:57:07 ucs kernel: [890042.613894] drbd .drbdctrl: State change failed: Need access to UpToDate data
Sep 21 20:57:07 ucs kernel: [890042.614012] drbd .drbdctrl: Failed: role( Secondary -> Primary )
Sep 21 20:57:07 ucs kernel: [890042.614016] drbd .drbdctrl: Auto-promote failed: Need access to UpToDate data
Sep 21 20:57:08 ucs drbdmanaged[11690]: ERROR cannot open control volume '/dev/drbd0' (10 failed attempts)

So it seems the problem is about not being able to open /dev/drbd0. I checked and it is there, but I *thought* the control volume is supposed to be that .drbdctrl lvm that is created?
 
Last edited:
On the remote host, the syslog shows:
Code: 
Sep 21 21:47:44 ucs kernel: [893079.620321] drbd .drbdctrl: State change failed: Need access to UpToDate data
Code: 
Sep 21 21:47:44 ucs kernel: [893079.620409] drbd .drbdctrl: Failed: role( Secondary -> Primary )
Code: 
Sep 21 21:47:44 ucs kernel: [893079.620413] drbd .drbdctrl: Auto-promote failed: Need access to UpToDate data
Code: 
Sep 21 21:47:45 ucs kernel: [893080.821808] drbd .drbdctrl: State change failed: Need access to UpToDate data
Code: 
Sep 21 21:47:45 ucs kernel: [893080.821896] drbd .drbdctrl: Failed: role( Secondary -> Primary )
Code: 
Sep 21 21:47:45 ucs kernel: [893080.821915] drbd .drbdctrl: Auto-promote failed: Need access to UpToDate data
Code: 
Sep 21 21:47:47 ucs kernel: [893082.944299] drbd .drbdctrl: State change failed: Need access to UpToDate data
Code: 
Sep 21 21:47:47 ucs kernel: [893082.944388] drbd .drbdctrl: Failed: role( Secondary -> Primary )
Code: 
Sep 21 21:47:47 ucs kernel: [893082.944392] drbd .drbdctrl: Auto-promote failed: Need access to UpToDate data
Code: 
Sep 21 21:47:48 ucs drbdmanaged[11690]: ERROR      cannot open control volume '/dev/drbd0' (10 failed attempts)
So it seems the problem is about not being able to open /dev/drbd0. I checked and it is there, but I *thought* the control volume is supposed to be that .drbdctrl lvm that is created? P.S., I had to put [code tags on every line because I guess this forum software doesn't allow line breaks inside the code tags? odd...
 
Hi,

I've got the same " Error: I/O error while accessing persistent configuration storage" - plain PVE 4.0 with testing repository following the proxmon drbd9 howto.

did you found a solution?

Regards,
Tom
 
I've alredy installed 0.50.2. :(
Code: 
root@px1:~# dpkg -l | grep '^ii' | grep drbdii  drbd-utils                     8.9.4-1                        amd64        RAID 1 over TCP/IP for Linux (user utilities)ii  drbdmanage                     0.50-2                         amd64        Distributed configuration management for DRBD
 
I do this after package installation - I've installed only this package, because drbdmanage was no tavailable before. Thats why I have made my first post here. I added the test repository and do "apt-get install drbdmanage" - I follow step-by-step the drbd9 howto.
 
plepps said:
I do this after package installation - I've installed only this package, because drbdmanage was no tavailable before.
Click to expand...

That is strange, because the output you posted is from an old version (0.49):

Code: 
[COLOR=#333333][FONT=courier new]Logical volume ".drbdctrl" successfully removed[/FONT][/COLOR]
[COLOR=#333333][FONT=courier new]Logical volume ".drbdctrl" created.
[/FONT][/COLOR]


new 0.50 creates 2 different drbdctrl volumes instead.
The output from new version looks like:
Code: 
Logical volume ".drbdctrl_0" created.          
Logical volume ".drbdctrl_1" created.
 
plepps said:
thats not my output, I've got only the same error message as the thread starter.
Click to expand...


It was my output, but I have upgraded the servers today and still get the same error:

Code: 
root@bfs:~# drbdmanage delete-node ucs
Code: 
You are going to remove the node 'ucs' from the cluster. This will remove all resources from the node.
Please confirm:
  yes/no: yes
Code: 
Operation completed successfully
root@bfs:~# drbdmanage new-node ucs 10.0.28.201
Operation completed successfully
Operation completed successfully
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified

Executing join command using ssh.
IMPORTANT: The output you see comes from ucs
IMPORTANT: Your input is executed on ucs
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
You are going to join an existing drbdmanage cluster.
CAUTION! Note that:
  * Any previous drbdmanage cluster information may be removed
  * Any remaining resources managed by a previous drbdmanage installation
    that still exist on this system will no longer be managed by drbdmanage

Confirm:

  yes/no: yes
  Logical volume ".drbdctrl" successfully removed
  Logical volume ".drbdctrl" created.
You want me to create a v09 style flexible-size internal meta data block.
There appears to be a v09 flexible-size internal meta data block
already in place on /dev/drbdpool/.drbdctrl at byte offset 4190208

Do you really want to overwrite the existing meta-data?
*** confirmation forced via --force option ***
NOT initializing bitmap
initializing activity log
Writing meta data...
New drbd meta data block successfully created.
Error: I/O error while accessing persistent configuration storage
root@bfs:~#
 
Hi,

my problem seems to be solved:
however on node 1 was the public address and not ip from storrage interface in /etc/drbd.d/drbdctrl.res.
After made the changes I rebooted all nodes and drbd creation works.

cheers,
tom
 
You must log in or register to reply here.
Top Bottom