Module 'volumes' has failed dependency: /lib/python3/dist-packages/cephfs.cpython-37m-x86_64-linux-gnu.so: undefined symbol: ceph_abort_conn

F

FelixJ

Well-Known Member
Mar 1, 2019
66
3
48
44
Hi, after Upgrade to 6.4.13 on a 3-node cluster I get a health warning:

Module 'volumes' has failed dependency: /lib/python3/dist-packages/cephfs.cpython-37m-x86_64-linux-gnu.so: undefined symbol: ceph_abort_conn

What am I missing?

I have already checked, that python-cephfs as well as python3-cephfs are installed. In doubt, I ran apt reinstall those 2 packages.
Any help appreciated!

regards,
Felix
 
Code: 
dpkg --list | grep ceph
ii  ceph                                 15.2.14-pve1~bpo10            amd64        distributed storage and file system
ii  ceph-base                            15.2.14-pve1~bpo10            amd64        common ceph daemon libraries and management tools
ii  ceph-common                          15.2.14-pve1~bpo10            amd64        common utilities to mount and interact with a ceph storage cluster
ii  ceph-fuse                            15.2.14-pve1~bpo10            amd64        FUSE-based client for the Ceph distributed file system
ii  ceph-mds                             15.2.14-pve1~bpo10            amd64        metadata server for the ceph distributed file system
ii  ceph-mgr                             15.2.14-pve1~bpo10            amd64        manager for the ceph distributed storage system
ii  ceph-mgr-modules-core                15.2.14-pve1~bpo10            all          ceph manager modules which are always enabled
ii  ceph-mon                             15.2.14-pve1~bpo10            amd64        monitor server for the ceph storage system
ii  ceph-osd                             15.2.14-pve1~bpo10            amd64        OSD server for the ceph storage system
ii  libcephfs2                           15.2.14-pve1~bpo10            amd64        Ceph distributed file system client library
ii  python-cephfs                        12.2.11+dfsg1-2.1+b1          amd64        Python libraries for the Ceph libcephfs library
ii  python3-ceph-argparse                15.2.14-pve1~bpo10            all          Python 3 utility libraries for Ceph CLI
ii  python3-ceph-common                  15.2.14-pve1~bpo10            all          Python 3 utility libraries for Ceph
ii  python3-cephfs                       15.2.14-pve1~bpo10            amd64        Python 3 libraries for the Ceph libcephfs library
 
what does ldd /lib/python3/dist-packages/cephfs.cpython-37m-x86_64-linux-gnu.so output? what about objdump --dynamic-syms /usr/lib/libcephfs.so.2.0.0 | grep ceph_abort_conn
 
fabian said:
what does ldd /lib/python3/dist-packages/cephfs.cpython-37m-x86_64-linux-gnu.so output? what about objdump --dynamic-syms /usr/lib/libcephfs.so.2.0.0 | grep ceph_abort_conn
Click to expand...
Code: 
ldd /lib/python3/dist-packages/cephfs.cpython-37m-x86_64-linux-gnu.so
    linux-vdso.so.1 (0x00007ffd6fd2b000)
    libcephfs.so.2 => /lib/libcephfs.so.2 (0x00007f0878016000)
    libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f0877fdc000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0877fbb000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f0877fb6000)
    libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f0877fb1000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0877e2e000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0877c6b000)
    libceph-common.so.2 => /usr/lib/ceph/libceph-common.so.2 (0x00007f086efc8000)
    libcrypto.so.1.1 => /lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f086ecdf000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f086ecd5000)
    libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f086ecbb000)
    libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 (0x00007f086ec66000)
    libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f086ec3e000)
    libibverbs.so.1 => /lib/x86_64-linux-gnu/libibverbs.so.1 (0x00007f086ec23000)
    librdmacm.so.1 => /lib/x86_64-linux-gnu/librdmacm.so.1 (0x00007f086ec06000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f086e9e8000)
    libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f086e864000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f086e84a000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f08781ca000)
    libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f086e83f000)
    libnl-route-3.so.200 => /lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x00007f086e5c6000)
    libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007f086e3a5000)

objdump --dynamic-syms /usr/lib/libcephfs.so.2.0.0 | grep ceph_abort_conn
0000000000028820 g    DF .text    0000000000000025  Base        ceph_abort_conn
 
that looks okay.. any other python2 ceph packages still around? e.g., python-rbd
 
fabian said:
that looks okay.. any other python2 ceph packages still around? e.g., python-rbd
Click to expand...
Here's a list of all ceph and python related packages:

Code: 
dpkg --list | grep -i python
ii  libpython-stdlib:amd64               2.7.16-1                      amd64        interactive high-level object-oriented language (Python2)
ii  libpython2-stdlib:amd64              2.7.16-1                      amd64        interactive high-level object-oriented language (Python2)
ii  libpython2.7:amd64                   2.7.16-2+deb10u1              amd64        Shared Python runtime library (version 2.7)
ii  libpython2.7-minimal:amd64           2.7.16-2+deb10u1              amd64        Minimal subset of the Python language (version 2.7)
ii  libpython2.7-stdlib:amd64            2.7.16-2+deb10u1              amd64        Interactive high-level object-oriented language (standard library, version 2.7)
ii  libpython3-stdlib:amd64              3.7.3-1                       amd64        interactive high-level object-oriented language (default python3 version)
ii  libpython3.7:amd64                   3.7.3-2+deb10u3               amd64        Shared Python runtime library (version 3.7)
ii  libpython3.7-minimal:amd64           3.7.3-2+deb10u3               amd64        Minimal subset of the Python language (version 3.7)
ii  libpython3.7-stdlib:amd64            3.7.3-2+deb10u3               amd64        Interactive high-level object-oriented language (standard library, version 3.7)
ii  python                               2.7.16-1                      amd64        interactive high-level object-oriented language (Python2 version)
ii  python-apt-common                    1.8.4.3                       all          Python interface to libapt-pkg (locales)
ii  python-certifi                       2018.8.24-1                   all          root certificates for validating SSL certs and verifying TLS hosts
ii  python-chardet                       3.0.4-3                       all          universal character encoding detector for Python2
ii  python-configparser                  3.5.0b2-1                     all          backport of the enhanced config parser introduced in Python 3.2
ii  python-future                        0.16.0-1                      all          single-source support for Python 3 and 2 - Python 2.x
ii  python-idna                          2.6-1                         all          Python IDNA2008 (RFC 5891) handling (Python 2)
ii  python-ipaddr                        2.2.0-2                       all          Python module for working with IP addresses, both IPv4 and IPv6
ii  python-minimal                       2.7.16-1                      amd64        minimal subset of the Python2 language
ii  python-pastedeploy-tpl               2.0.1-1                       all          load, configure, and compose WSGI applications and servers - templates
ii  python-pkg-resources                 40.8.0-1                      all          Package Discovery and Resource Access using pkg_resources
ii  python-prettytable                   0.7.2-4                       all          library to represent tabular data in visually appealing ASCII tables
ii  python-protobuf                      3.6.1.3-2                     amd64        Python bindings for protocol buffers
ii  python-requests                      2.21.0-1                      all          elegant and simple HTTP library for Python2, built for human beings
ii  python-six                           1.12.0-1                      all          Python 2 and 3 compatibility library (Python 2 interface)
ii  python-talloc:amd64                  2.1.14-2                      amd64        hierarchical pool based memory allocator - Python bindings
ii  python-urllib3                       1.24.1-1                      all          HTTP library with thread-safe connection pooling for Python
ii  python2                              2.7.16-1                      amd64        interactive high-level object-oriented language (Python2 version)
ii  python2-minimal                      2.7.16-1                      amd64        minimal subset of the Python2 language
ii  python2.7                            2.7.16-2+deb10u1              amd64        Interactive high-level object-oriented language (version 2.7)
ii  python2.7-minimal                    2.7.16-2+deb10u1              amd64        Minimal subset of the Python language (version 2.7)
ii  python3                              3.7.3-1                       amd64        interactive high-level object-oriented language (default python3 version)
ii  python3-apt                          1.8.4.3                       amd64        Python 3 interface to libapt-pkg
ii  python3-asn1crypto                   0.24.0-1                      all          Fast ASN.1 parser and serializer (Python 3)
ii  python3-bcrypt                       3.1.6-1                       amd64        password hashing library for Python 3
ii  python3-bs4                          4.7.1-1                       all          error-tolerant HTML parser for Python 3
ii  python3-ceph-argparse                15.2.14-pve1~bpo10            all          Python 3 utility libraries for Ceph CLI
ii  python3-ceph-common                  15.2.14-pve1~bpo10            all          Python 3 utility libraries for Ceph
ii  python3-cephfs                       15.2.14-pve1~bpo10            amd64        Python 3 libraries for the Ceph libcephfs library
ii  python3-certifi                      2018.8.24-1                   all          root certificates for validating SSL certs and verifying TLS hosts (python3)
ii  python3-cffi-backend                 1.12.2-1                      amd64        Foreign Function Interface for Python 3 calling C code - runtime
ii  python3-chardet                      3.0.4-3                       all          universal character encoding detector for Python3
ii  python3-cherrypy3                    8.9.1-2                       all          Python web development framework - Python 3 module
ii  python3-cryptography                 2.6.1-3+deb10u2               amd64        Python library exposing cryptographic recipes and primitives (Python 3)
ii  python3-dateutil                     2.7.3-3                       all          powerful extensions to the standard Python 3 datetime module
ii  python3-debconf                      1.5.71                        all          interact with debconf from Python 3
ii  python3-debian                       0.1.35                        all          Python 3 modules to work with Debian-related data formats
ii  python3-debianbts                    2.8.2                         all          Python interface to Debian's Bug Tracking System
ii  python3-distutils                    3.7.3-1                       all          distutils package for Python 3.x
ii  python3-html5lib                     1.0.1-1                       all          HTML parser/tokenizer based on the WHATWG HTML5 specification
ii  python3-httplib2                     0.11.3-2                      all          comprehensive HTTP client library written for Python3
ii  python3-idna                         2.6-1                         all          Python IDNA2008 (RFC 5891) handling (Python 3)
ii  python3-jwt                          1.7.0-2                       all          Python 3 implementation of JSON Web Token
ii  python3-lib2to3                      3.7.3-1                       all          Interactive high-level object-oriented language (2to3, version 3.6)
ii  python3-logutils                     0.3.3-5                       all          handlers for the Python standard library's logging package - Python 3.x
ii  python3-lxml:amd64                   4.3.2-1+deb10u3               amd64        pythonic binding for the libxml2 and libxslt libraries
ii  python3-mako                         1.0.7+ds1-1                   all          fast and lightweight templating for the Python 3 platform
ii  python3-markupsafe                   1.1.0-1                       amd64        HTML/XHTML/XML string library for Python 3
ii  python3-minimal                      3.7.3-1                       amd64        minimal subset of the Python language (default python3 version)
ii  python3-openssl                      19.0.0-1                      all          Python 3 wrapper around the OpenSSL library
ii  python3-paste                        3.0.6+dfsg-1                  all          tools for using a Web Server Gateway Interface stack - Python 3.x
ii  python3-pastedeploy                  2.0.1-1                       all          load, configure, and compose WSGI applications and servers - Python 3.x
ii  python3-pastescript                  2.0.2-2                       all          create file layouts for Python3 packages and serve webapps
ii  python3-pecan                        1.3.2-2                       all          WSGI object-dispatching web framework - Python 3.x
ii  python3-pkg-resources                40.8.0-1                      all          Package Discovery and Resource Access using pkg_resources
ii  python3-prettytable                  0.7.2-4                       all          library to represent tabular data in visually appealing ASCII tables (Python3)
ii  python3-pycurl                       7.43.0.2-0.1                  amd64        Python bindings to libcurl (Python 3)
ii  python3-pyinotify                    0.9.6-1                       all          simple Linux inotify Python bindings
ii  python3-pysimplesoap                 1.16.2-1                      all          simple and lightweight SOAP Library (Python 3)
ii  python3-rados                        15.2.14-pve1~bpo10            amd64        Python 3 libraries for the Ceph librados library
ii  python3-rbd                          15.2.14-pve1~bpo10            amd64        Python 3 libraries for the Ceph librbd library
ii  python3-reportbug                    7.5.3~deb10u1                 all          Python modules for interacting with bug tracking systems
ii  python3-repoze.lru                   0.7-1                         all          tiny LRU cache implementation and decorator for Python 3
ii  python3-requests                     2.21.0-1                      all          elegant and simple HTTP library for Python3, built for human beings
ii  python3-rgw                          15.2.14-pve1~bpo10            amd64        Python 3 libraries for the Ceph librgw library
ii  python3-routes                       2.4.1-1                       all          Routing Recognition and Generation Tools for Python 3
ii  python3-setuptools                   40.8.0-1                      all          Python3 Distutils Enhancements
ii  python3-simplegeneric                0.8.1-2                       all          simple generic functions for Python3
ii  python3-simplejson                   3.16.0-1                      amd64        simple, fast, extensible JSON encoder/decoder for Python 3.x
ii  python3-singledispatch               3.4.0.3-2                     all          single-dispatch generic functions for Python
ii  python3-six                          1.12.0-1                      all          Python 2 and 3 compatibility library (Python 3 interface)
ii  python3-soupsieve                    1.8+dfsg-1                    all          modern CSS selector implementation for BeautifulSoup (Python 3)
ii  python3-tempita                      0.5.2-2                       all          very small text templating language
ii  python3-urllib3                      1.24.1-1                      all          HTTP library with thread-safe connection pooling for Python3
ii  python3-waitress                     1.2.0~b2-2                    all          production-quality pure-Python WSGI server (Python 3)
ii  python3-webencodings                 0.5.1-1                       all          Python implementation of the WHATWG Encoding standard
ii  python3-webob                        1:1.8.5-1                     all          Python module providing WSGI request and response objects (Python 3)
ii  python3-webtest                      2.0.32-1                      all          wraps any WSGI application and makes it easy to test
ii  python3-werkzeug                     0.14.1+dfsg1-4+deb10u1        all          collection of utilities for WSGI applications (Python 3.x)
ii  python3-yaml                         3.13-2                        amd64        YAML parser and emitter for Python3
ii  python3.7                            3.7.3-2+deb10u3               amd64        Interactive high-level object-oriented language (version 3.7)
ii  python3.7-minimal                    3.7.3-2+deb10u3               amd64        Minimal subset of the Python language (version 3.7)
dpkg --list | grep -i ceph
ii  ceph                                 15.2.14-pve1~bpo10            amd64        distributed storage and file system
ii  ceph-base                            15.2.14-pve1~bpo10            amd64        common ceph daemon libraries and management tools
ii  ceph-common                          15.2.14-pve1~bpo10            amd64        common utilities to mount and interact with a ceph storage cluster
ii  ceph-fuse                            15.2.14-pve1~bpo10            amd64        FUSE-based client for the Ceph distributed file system
ii  ceph-mds                             15.2.14-pve1~bpo10            amd64        metadata server for the ceph distributed file system
ii  ceph-mgr                             15.2.14-pve1~bpo10            amd64        manager for the ceph distributed storage system
ii  ceph-mgr-modules-core                15.2.14-pve1~bpo10            all          ceph manager modules which are always enabled
ii  ceph-mon                             15.2.14-pve1~bpo10            amd64        monitor server for the ceph storage system
ii  ceph-osd                             15.2.14-pve1~bpo10            amd64        OSD server for the ceph storage system
ii  libcephfs2                           15.2.14-pve1~bpo10            amd64        Ceph distributed file system client library
ii  python3-ceph-argparse                15.2.14-pve1~bpo10            all          Python 3 utility libraries for Ceph CLI
ii  python3-ceph-common                  15.2.14-pve1~bpo10            all          Python 3 utility libraries for Ceph
ii  python3-cephfs                       15.2.14-pve1~bpo10            amd64        Python 3 libraries for the Ceph libcephfs library
ii  python3-rados                        15.2.14-pve1~bpo10            amd64        Python 3 libraries for the Ceph librados library
ii  python3-rbd                          15.2.14-pve1~bpo10            amd64        Python 3 libraries for the Ceph librbd library
ii  python3-rgw                          15.2.14-pve1~bpo10            amd64        Python 3 libraries for the Ceph librgw library
 
that also looks okay. what about ceph versions?
 
Code: 
ceph versions
{
    "mon": {
        "ceph version 15.2.14 (e3964f3ac2e00e5baacdec60b36e45d1c2e8571b) octopus (stable)": 3
    },
    "mgr": {
        "ceph version 15.2.14 (e3964f3ac2e00e5baacdec60b36e45d1c2e8571b) octopus (stable)": 2
    },
    "osd": {
        "ceph version 15.2.14 (e3964f3ac2e00e5baacdec60b36e45d1c2e8571b) octopus (stable)": 20
    },
    "mds": {
        "ceph version 15.2.14 (e3964f3ac2e00e5baacdec60b36e45d1c2e8571b) octopus (stable)": 2
    },
    "overall": {
        "ceph version 15.2.14 (e3964f3ac2e00e5baacdec60b36e45d1c2e8571b) octopus (stable)": 27
    }
}
 
possibly stupid question - did you restart ceph-mgr after cleaning out the python packages?
 
fabian said:
possibly stupid question - did you restart ceph-mgr after cleaning out the python packages?
Click to expand...
This is not a stupid question at all, but I did via the gui. Your question, how ever inspired me, to restart the daemons on each host via the systemctl restart ceph-mgr@hostname.service

Out of an old sysadmin habit, I always check systemctl status daemon-name afterwards, which I did this time as well.
Then I looked in /var/log/daemon.log and found this:
Code: 
Sep 22 14:05:56 nodeX ceph-mgr[3954763]: 2021-09-22T14:05:56.949+0200 7fca0c6ae0c0 -1 mgr[py] Module not found: 'volumes'
Sep 22 14:05:56 nodeX ceph-mgr[3954763]: 2021-09-22T14:05:56.949+0200 7fca0c6ae0c0 -1 mgr[py] Traceback (most recent call last):
Sep 22 14:05:56 nodeX ceph-mgr[3954763]:   File "/usr/share/ceph/mgr/volumes/__init__.py", line 2, in <module>
Sep 22 14:05:56 nodeX ceph-mgr[3954763]:     from .module import Module
Sep 22 14:05:56 nodeX ceph-mgr[3954763]:   File "/usr/share/ceph/mgr/volumes/module.py", line 10, in <module>
Sep 22 14:05:56 nodeX ceph-mgr[3954763]:     from .fs.volume import VolumeClient
Sep 22 14:05:56 nodeX ceph-mgr[3954763]:   File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 6, in <module>
Sep 22 14:05:56 nodeX ceph-mgr[3954763]:     import cephfs
Sep 22 14:05:56 nodeX ceph-mgr[3954763]: ImportError: /lib/python3/dist-packages/cephfs.cpython-37m-x86_64-linux-gnu.so: undefined symbol: ceph_abort_conn
Sep 22 14:05:56 nodeX ceph-mgr[3954763]: 2021-09-22T14:05:56.949+0200 7fca0c6ae0c0 -1 mgr[py] Class not found in module 'volumes'
Sep 22 14:05:56 nodeX ceph-mgr[3954763]: 2021-09-22T14:05:56.949+0200 7fca0c6ae0c0 -1 mgr[py] Error loading module 'volumes': (2) No such file or directory
Sep 22 14:05:57 remote3 ceph-mgr[3954763]: 2021-09-22T14:05:57.093+0200 7fca0c6ae0c0 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: volumes

Now, at least, I know, which node produces that error... hope, this is helpful.
 
Found something else:
This is from the broken host: ldd /lib/python3/dist-packages/cephfs.cpython-37m-x86_64-linux-gnu.so
Code: 
    linux-vdso.so.1 (0x00007ffcba889000)
    libcephfs.so.2 => /lib/x86_64-linux-gnu/libcephfs.so.2 (0x00007f5c06ec7000)
    libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f5c06e8d000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5c06e6c000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5c06e67000)
    libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f5c06e62000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5c06cdf000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c06b1c000)
    libceph-common.so.0 => /usr/lib/x86_64-linux-gnu/ceph/libceph-common.so.0 (0x00007f5bfe054000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5bfe04a000)
    libboost_thread.so.1.67.0 => /lib/x86_64-linux-gnu/libboost_thread.so.1.67.0 (0x00007f5bfe01e000)
    libboost_system.so.1.67.0 => /lib/x86_64-linux-gnu/libboost_system.so.1.67.0 (0x00007f5bfe017000)
    libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5bfde93000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5bfde77000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f5c07062000)
    libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f5bfde5d000)
    libboost_regex.so.1.67.0 => /lib/x86_64-linux-gnu/libboost_regex.so.1.67.0 (0x00007f5bfdd48000)
    libboost_iostreams.so.1.67.0 => /lib/x86_64-linux-gnu/libboost_iostreams.so.1.67.0 (0x00007f5bfdd2a000)
    libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 (0x00007f5bfdcd5000)
    libsmime3.so => /lib/x86_64-linux-gnu/libsmime3.so (0x00007f5bfdca4000)
    libnss3.so => /lib/x86_64-linux-gnu/libnss3.so (0x00007f5bfdb56000)
    libnspr4.so => /lib/x86_64-linux-gnu/libnspr4.so (0x00007f5bfdb15000)
    libibverbs.so.1 => /lib/x86_64-linux-gnu/libibverbs.so.1 (0x00007f5bfdafa000)
    libboost_atomic.so.1.67.0 => /lib/x86_64-linux-gnu/libboost_atomic.so.1.67.0 (0x00007f5bfdaf5000)
    libicudata.so.63 => /lib/x86_64-linux-gnu/libicudata.so.63 (0x00007f5bfc103000)
    libicui18n.so.63 => /lib/x86_64-linux-gnu/libicui18n.so.63 (0x00007f5bfbe28000)
    libicuuc.so.63 => /lib/x86_64-linux-gnu/libicuuc.so.63 (0x00007f5bfbc59000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5bfba3b000)
    libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f5bfba28000)
    libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f5bfba1f000)
    libnssutil3.so => /lib/x86_64-linux-gnu/libnssutil3.so (0x00007f5bfb9eb000)
    libplc4.so => /lib/x86_64-linux-gnu/libplc4.so (0x00007f5bfb9e4000)
    libplds4.so => /lib/x86_64-linux-gnu/libplds4.so (0x00007f5bfb9df000)
    libnl-route-3.so.200 => /lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x00007f5bfb766000)
    libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007f5bfb545000)

2 Things, I can tell compared to the output from the working host, I sent you earlier:
  • There are far more libraries linked
  • the Path is different
 
is 'ceph-mgr-modules-core' installed on that node?
 
Good Morning!
I think, I could nail down the problem, but I don't know, how to solve it:

On the good node:
ldd /lib/python3/dist-packages/cephfs.cpython-37m-x86_64-linux-gnu.so| grep libceph-common libceph-common.so.2 => /usr/lib/ceph/libceph-common.so.2 (0x00007fe0502ef000)

On the bad node:
ldd /lib/python3/dist-packages/cephfs.cpython-37m-x86_64-linux-gnu.so|grep libceph-common libceph-common.so.0 => /usr/lib/x86_64-linux-gnu/ceph/libceph-common.so.0 (0x00007fe8dbd04000)

This library comes from the deb package librados2, which is installed in the same version on both machines.
It seams, that cephfs.cpython-37m-x86_64-linux-gnu.so is linked to the wrong Library on the bad node. I tried to reinstall both packages (librados2 and python3-cephfs) but the problem is still here.
How can one library coming from the same source package be linked to to different depended libraries?

I also checked the ld.conf.so settings on both nodes, they are identical.
Thankfully for any Ideas!
 
which version of librados2 is installed on both nodes (dpkg -l librados2 and apt-cache policy librados2)? what does dpkg -L librados2 say on both nodes?
 
Good node:

dpkg -l librados2 Gewünscht=Unbekannt/Installieren/R=Entfernen/P=Vollständig Löschen/Halten | Status=Nicht/Installiert/Config/U=Entpackt/halb konFiguriert/ Halb installiert/Trigger erWartet/Trigger anhängig |/ Fehler?=(kein)/R=Neuinstallation notwendig (Status, Fehler: GROSS=schlecht) ||/ Name Version Architektur Beschreibung +++-==============-==================-============-============================================= ii librados2 15.2.14-pve1~bpo10 amd64 RADOS distributed object store client library
apt-cache policy librados2 librados2: Installiert: 15.2.14-pve1~bpo10 Installationskandidat: 15.2.14-pve1~bpo10 Versionstabelle: *** 15.2.14-pve1~bpo10 500 500 http://download.proxmox.com/debian/ceph-octopus buster/main amd64 Packages 100 /var/lib/dpkg/status 15.2.13-pve1~bpo10 500 500 http://download.proxmox.com/debian/ceph-octopus buster/main amd64 Packages 15.2.12-pve1 500 500 http://download.proxmox.com/debian/ceph-octopus buster/main amd64 Packages 15.2.11-pve1 500 500 http://download.proxmox.com/debian/ceph-octopus buster/main amd64 Packages 15.2.10-pve1 500 500 http://download.proxmox.com/debian/ceph-octopus buster/main amd64 Packages 15.2.9-pve1 500 500 http://download.proxmox.com/debian/ceph-octopus buster/main amd64 Packages 15.2.8-pve2 500 500 http://download.proxmox.com/debian/ceph-octopus buster/main amd64 Packages 15.2.6-pve1 500 500 http://download.proxmox.com/debian/ceph-octopus buster/main amd64 Packages 15.2.5-pve1 500 500 http://download.proxmox.com/debian/ceph-octopus buster/main amd64 Packages 15.2.4-pve1 500 500 http://download.proxmox.com/debian/ceph-octopus buster/main amd64 Packages 15.2.3-pve1 500 500 http://download.proxmox.com/debian/ceph-octopus buster/main amd64 Packages 12.2.11+dfsg1-2.1+b1 500 500 http://ftp.at.debian.org/debian buster/main amd64 Packages

dpkg -L librados2 /. /usr /usr/lib /usr/lib/ceph /usr/lib/ceph/libceph-common.so.2 /usr/lib/librados.so.2.0.0 /usr/lib/librados_tp.so.2.0.0 /usr/share /usr/share/doc /usr/share/doc/librados2 /usr/share/doc/librados2/changelog.Debian.gz /usr/share/doc/librados2/changelog.gz /usr/lib/librados.so.2 /usr/lib/librados_tp.so.2

Bad node:

dpkg -l librados2 Gewünscht=Unbekannt/Installieren/R=Entfernen/P=Vollständig Löschen/Halten | Status=Nicht/Installiert/Config/U=Entpackt/halb konFiguriert/ Halb installiert/Trigger erWartet/Trigger anhängig |/ Fehler?=(kein)/R=Neuinstallation notwendig (Status, Fehler: GROSS=schlecht) ||/ Name Version Architektur Beschreibung +++-==============-==================-============-============================================= ii librados2 15.2.14-pve1~bpo10 amd64 RADOS distributed object store client library

apt-cache policy librados2 librados2: Installiert: 15.2.14-pve1~bpo10 Installationskandidat: 15.2.14-pve1~bpo10 Versionstabelle: *** 15.2.14-pve1~bpo10 500 500 http://download.proxmox.com/debian/ceph-octopus buster/main amd64 Packages 100 /var/lib/dpkg/status 15.2.13-pve1~bpo10 500 500 http://download.proxmox.com/debian/ceph-octopus buster/main amd64 Packages 15.2.12-pve1 500 500 http://download.proxmox.com/debian/ceph-octopus buster/main amd64 Packages 15.2.11-pve1 500 500 http://download.proxmox.com/debian/ceph-octopus buster/main amd64 Packages 15.2.10-pve1 500 500 http://download.proxmox.com/debian/ceph-octopus buster/main amd64 Packages 15.2.9-pve1 500 500 http://download.proxmox.com/debian/ceph-octopus buster/main amd64 Packages 15.2.8-pve2 500 500 http://download.proxmox.com/debian/ceph-octopus buster/main amd64 Packages 15.2.6-pve1 500 500 http://download.proxmox.com/debian/ceph-octopus buster/main amd64 Packages 15.2.5-pve1 500 500 http://download.proxmox.com/debian/ceph-octopus buster/main amd64 Packages 15.2.4-pve1 500 500 http://download.proxmox.com/debian/ceph-octopus buster/main amd64 Packages 15.2.3-pve1 500 500 http://download.proxmox.com/debian/ceph-octopus buster/main amd64 Packages 12.2.11+dfsg1-2.1+b1 500 500 http://ftp.at.debian.org/debian buster/main amd64 Packages

dpkg -L librados2 /. /usr /usr/lib /usr/lib/ceph /usr/lib/ceph/libceph-common.so.2 /usr/lib/librados.so.2.0.0 /usr/lib/librados_tp.so.2.0.0 /usr/share /usr/share/doc /usr/share/doc/librados2 /usr/share/doc/librados2/changelog.Debian.gz /usr/share/doc/librados2/changelog.gz /usr/lib/librados.so.2 /usr/lib/librados_tp.so.2

They are identical, I ran a diff across the entire output...
 
You must log in or register to reply here.
Top Bottom