pmxcfs: memdb_open failed - unable to open database '/var/lib/pve-cluster/conf

[jcheger]

From this week (Dec 24), I've got several machine which got this issue while upgrading:

# /etc/init.d/pve-cluster restart

Syslog:
Dec 24 08:56:00 elrond pmxcfs[14809]: [database] parent is not a directory (inode = 0000000000000261, parent = 0000000000000313, name = '1038.conf') (database.c:401:bdb_backend_load_index)
Dec 24 08:56:00 elrond pmxcfs[14809]: [database] DB load failed (database.c:445:bdb_backend_load_index)
Dec 24 08:56:00 elrond pmxcfs[14809]: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db' (pmxcfs.c:763:main)
Dec 24 08:56:00 elrond pmxcfs[14809]: exit proxmox configuration filesystem (-1)

The database is OK:
# echo "pragma integrity_check;" | sqlite3 /var/lib/pve-cluster/config.db
ok

But the only way to get out of this that I've found is to delete the database. If this happens only to one node in the cluster, it it rebuilt. But on yesterday, it did happen on both of my nodes while upgrading one (the working one hanged up, and mount died on reboot). This also happened on another cluster yesterday.


I can start over, but any help welcome.

By the way, thanks for the great job you're doing.

Jean-Christophe


Revelant infos:
Debian Squeeze + Proxmox 2.0 beta
PVE Packages:
ii clvm 2.02.86-1pve2 Cluster LVM Daemon for lvm2
ii corosync-pve 1.4.1-1 Standards-based cluster framework (daemon and modules)
ii dmsetup 2:1.02.65-1pve2 Linux Kernel Device Mapper userspace library
ii fuse-utils 2.8.4-1.1 Filesystem in USErspace (utilities)
ii libaprutil1-dbd-sqlite3 1.3.9+dfsg-5 The Apache Portable Runtime Utility Library - SQLite3 Driver
ii libcorosync4-pve 1.4.1-1 Standards-based cluster framework (libraries)
ii libdevmapper-dev 2:1.02.65-1pve2 Linux Kernel Device Mapper header files
ii libdevmapper1.02.1 2:1.02.65-1pve2 Linux Kernel Device Mapper userspace library
ii libfuse2 2.8.4-1.1 Filesystem in USErspace library
ii libopenais3-pve 1.1.4-1 Standards-based cluster framework (libraries)
ii libpve-access-control 1.0-3 Proxmox VE access control library
ii libpve-common-perl 1.0-10 Proxmox VE base library
ii libpve-storage-perl 2.0-9 Proxmox VE storage management library
ii libsqlite3-0 3.7.3-1 SQLite 3 shared library
ii lvm2 2.02.86-1pve2 Linux Logical Volume Manager
ii openais-pve 1.1.4-1 Standards-based cluster framework (daemon and modules)
ii pve-cluster 1.0-12 Cluster Infrastructure for Proxmox Virtual Environment
ii pve-firmware 1.0-13 Binary firmware code for the pve-kernel
ii pve-headers-2.6.32-6-pve 2.6.32-54 The Proxmox PVE Kernel Headers
ii pve-kernel-2.6.32-6-pve 2.6.32-54 The Proxmox PVE Kernel Image
ii pve-manager 2.0-14 The Proxmox Virtual Environment
ii pve-qemu-kvm 1.0-1 Full virtualization on x86 hardware
ii redhat-cluster-pve 3.1.7-1 Red Hat cluster suite
ii sqlite3 3.7.3-1 A command line interface for SQLite 3
ii vzctl 3.0.29-3pve7 OpenVZ - server virtualization solution - control tools

 

[dietmar]

Dec 24 08:56:00 elrond pmxcfs[14809]: [database] parent is not a directory (inode = 0000000000000261, parent = 0000000000000313, name = '1038.conf') (database.c:401:bdb_backend_load_index)

It would be great if you find out a way to reproduce that. Do you remember what you have done before that happened? Did you copied something into /etc/pve/ manually?

If you delete the database, it is automatically copied from other nodes when you restart the filesystem - so it should be quite fast to recover.

 

[Faye]

Bumping this as I think it's related to my issue. I believe I've root caused it but you'd need to look at bdb_backend_load_index to see if my assertion that it loads the nodes in order is true. Certainly the effects would seem to bear it out.

http://www.ordinoscope.net/index.php/Informatique/Softwares/Proxmox/PVE_2.x_-_Programmes_et_services

Also here, in French, I know, but suggests nuking the database as a 'fix' In all these cases, the child is a lower id than the parent.

 

[jcheger]

You're right, I've done some changes manually. I did not have time to test it seriously yet, but I think that this issue did appear when I've changed something in the /etc/pve/openvz or /etc/pve/qemu-server symlinks, instead of working in /etc/pve/nodes. I still need to spend a couple of hours to understand, because this issue did kill 2 clusters already.

About the link above, Ordinoscope is my web site actually. This page contains my notes until now, but is not complete yet.