[SOLVED] undefined symbol: rados_mgr_command_target

hoffmn01 · May 17, 2021

Hello @Moayad,

I guess the wrong/missing shared libraries cause the trouble here, is there a way to correct these?

Thank you and best regards,

Nico

Moayad · May 17, 2021

not to easy but you can try to link the missing libraries like in the other nodes

hoffmn01 · May 17, 2021

Hello @Moayad,

thank you for your answer, but how would you recommend to proceed? How to link the libraries back to the correct values? I checked the ld.conf-parameters and they seem to look ok.

Thank you and best regards,

Nico

RokaKen · May 17, 2021

hoffmn01 said:

Hello @Moayad ,

here comes the output of dpkg --list | grep ceph:

Bash:

ii  ceph                                 15.2.11-pve1                  amd64        distributed storage and file system
ii  ceph-base                            15.2.11-pve1                  amd64        common ceph daemon libraries and management tools
ii  ceph-common                          15.2.11-pve1                  amd64        common utilities to mount and interact with a ceph storage cluster
ii  ceph-fuse                            15.2.11-pve1                  amd64        FUSE-based client for the Ceph distributed file system
ii  ceph-mds                             15.2.11-pve1                  amd64        metadata server for the ceph distributed file system
ii  ceph-mgr                             15.2.11-pve1                  amd64        manager for the ceph distributed storage system
ii  ceph-mgr-modules-core                15.2.11-pve1                  all          ceph manager modules which are always enabled
ii  ceph-mon                             15.2.11-pve1                  amd64        monitor server for the ceph storage system
ii  ceph-osd                             15.2.11-pve1                  amd64        OSD server for the ceph storage system
ii  libcephfs2                           15.2.11-pve1                  amd64        Ceph distributed file system client library
ii  python-cephfs                        12.2.11+dfsg1-2.1+b1          amd64        Python libraries for the Ceph libcephfs library

<snip>

I'd say that last line above is your problem. You have installed 'python-cephfs' for Luminous from the Debian repository. That explains the library mismatches.

Try apt install --reinstall python-cephfs to install the correct PVE package. If that fails, post the output of the install command and the output of apt policy python-cephfs.

[EDIT for format -- I'll get it right eventually]

hoffmn01 · May 17, 2021

RokaKen said:
<snip>

I'd say that last line above is your problem. You have installed 'python-cephfs' for Luminous from the Debian repository. That explains the library mismatches.

Try apt install --reinstall python-cephfs to install the correct PVE package. If that fails, post the output of the install command and the output of apt policy python-cephfs.

[EDIT for format -- I'll get it right eventually]

Hello @RokaKen ,

I did and now the output of the dpkg --list is equal, nevertheless the ldd output remains broken. Any idea how to fix that? I already rebooted the machine, no change.

Thank you and best regards,

Nico

RokaKen · May 17, 2021

You will have to compare package versions from your working node and broken node until you find and replace all the incorrect packages.

hoffmn01 · May 17, 2021

RokaKen said:
You will have to compare package versions from your working node and broken node until you find and replace all the incorrect packages.

I did as requested, here is the output of the compare:

Bash:

root@pve3:~/install# ./compare.pl compare-pm2.txt compare-pm3.txt | less
No differences were found.

I used this tool: https://github.com/lepe/scripts/blob/master/compare_ubuntu_apt.pl

The situation did not change... :-(

RokaKen · May 17, 2021

Then you will have to compare package components. For instance, compare ldconfig -v between the working and broken nodes, determine the package that installed the differing library and reinstall the appropriate package and so forth.

hoffmn01 · May 17, 2021

Hello @Moayad , @t.lamprecht , @RokaKen ,

I just found the missing piece on the other working node:

Bash:

root@pve1:~# ceph -s
  cluster:
    id:     73703df0-b8ae-4aca-8269-1fb68da2142d
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum delbgpm02,delbgpm03,delbgpm01 (age 67s)
    mgr: delbgpm02(active, since 5d), standbys: delbgpm01, delbgpm03
    mds: cephfs:1 {0=delbgpm01=up:active} 2 up:standby
    osd: 13 osds: 13 up (since 28m), 13 in (since 7d)

  data:
    pools:   4 pools, 193 pgs
    objects: 2.69M objects, 10 TiB
    usage:   30 TiB used, 58 TiB / 88 TiB avail
    pgs:     193 active+clean

  io:
    client:   412 KiB/s wr, 0 op/s rd, 49 op/s wr

root@pve1:~# apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  libpq5
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 170 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://security.debian.org buster/updates/main amd64 libpq5 amd64 11.12-0+deb10u1 [170 kB]
Fetched 170 kB in 0s (345 kB/s)
Reading changelogs... Done
(Reading database ... 74307 files and directories currently installed.)
Preparing to unpack .../libpq5_11.12-0+deb10u1_amd64.deb ...
Unpacking libpq5:amd64 (11.12-0+deb10u1) over (11.11-0+deb10u1) ...
Setting up libpq5:amd64 (11.12-0+deb10u1) ...
Processing triggers for libc-bin (2.28-10) ...
root@delbgpm01:~# ceph -s
Traceback (most recent call last):
  File "/usr/bin/ceph", line 140, in <module>
    import rados
ImportError: /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so: undefined symbol: rados_mgr_command_target

I will try to revert the version of libpq5 now.

hoffmn01 · May 17, 2021

Downgrade did not help...

Code:

root@pve1:~# apt install libpq5=11.11-0+deb10u1
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be DOWNGRADED:
  libpq5
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 0 not upgraded.
Need to get 169 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://ftp.de.debian.org/debian buster/main amd64 libpq5 amd64 11.11-0+deb10u1 [169 kB]
Fetched 169 kB in 0s (725 kB/s)
dpkg: warning: downgrading libpq5:amd64 from 11.12-0+deb10u1 to 11.11-0+deb10u1
(Reading database ... 74307 files and directories currently installed.)
Preparing to unpack .../libpq5_11.11-0+deb10u1_amd64.deb ...
Unpacking libpq5:amd64 (11.11-0+deb10u1) over (11.12-0+deb10u1) ...
Setting up libpq5:amd64 (11.11-0+deb10u1) ...
Processing triggers for libc-bin (2.28-10) ...
root@pve1:~# ceph -s
Traceback (most recent call last):
  File "/usr/bin/ceph", line 140, in <module>
    import rados
ImportError: /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so: undefined symbol: rados_mgr_command_target

RokaKen · May 17, 2021

hoffmn01 said:

Downgrade did not help...

Code:

root@pve1:~# apt install libpq5=11.11-0+deb10u1
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be DOWNGRADED:
  libpq5
<snip>

Well, at least you found how you broke it. No, you are just downgrading a Debian package

Code:

# apt policy libpq5
libpq5:
  Installed: (none)
  Candidate: 11.12-0+deb10u1
  Version table:
     11.12-0+deb10u1 500
        500 http://security.debian.org/debian-security buster/updates/main amd64 Packages
     11.11-0+deb10u1 500
        500 http://deb.debian.org/debian buster/main amd64 Packages

Try just removing libpq5

hoffmn01 · May 17, 2021

Hello @RokaKen ,

thank you for your message, but I did not break it at all. It was apt upgrade which I was supposed to run earlier in the threat. I guess there is something wrong with the packages since the last update of the systems. I do have another cluster we run, but I will not update it until this is cleared.

After removing the package the error stays the same.

Any other idea or hint?

Best regards,

Nico

RokaKen · May 17, 2021

In general, never, Never, NEVER run apt upgrade on a PVE node. Always use apt full-upgrade || apt-get dist-upgrade || pveupgrade in order to install updated packages.

hoffmn01 · May 18, 2021

RokaKen said:
In general, never, Never, NEVER run apt upgrade on a PVE node. Always use apt full-upgrade || apt-get dist-upgrade || pveupgrade in order to install updated packages.

Hello @RokaKen ,

so the recommendation is to wait for the next dist-upgrade to possibly fix things?

Best regards,

Nico

RokaKen · May 18, 2021

I apologize for the lack of context, so let me try to clarify (somewhat in reverse order). So, apt upgrade somewhat blindly updates installed packages to "newer" versions, but doesn't check dependencies and won't add new or remove packages where as apt full-upgrade || apt-get dist-upgrade will check dependencies and perform conflict resolution so that things "just work" (usually). pveupgrade is just a script around apt-get dist-upgrade so you don't have to remember. Thus, running an 'apt upgrade' is usually how PVE users get conflicting packages and related problems.

The comments with respect to 'libpq5' just meant to convey that it wasn't your problem. It isn't needed/related to PVE, Ceph, or even Python as far as I can tell -- only PostgreSQL. You'll need to look elsewhere. But, running apt upgrade did break your formerly working 'ceph' command as you reported.

Background: /usr/bin/ceph is a Python script. On my (Nautilus) system it is python 2.7 and a system can have more than one version of Python. I have:

Code:

# ls -lh /usr/bin/python*
lrwxrwxrwx 1 root root    7 Mar  4  2019 /usr/bin/python -> python2
lrwxrwxrwx 1 root root    9 Mar  4  2019 /usr/bin/python2 -> python2.7
-rwxr-xr-x 1 root root 3.6M Oct 10  2019 /usr/bin/python2.7
-rwxr-xr-x 1 root root  385 Jan 30  2019 /usr/bin/python2-futurize
-rwxr-xr-x 1 root root  392 Sep  3  2018 /usr/bin/python2-gunicorn_pecan
-rwxr-xr-x 1 root root  389 Jan 30  2019 /usr/bin/python2-pasteurize
-rwxr-xr-x 1 root root  374 Sep  3  2018 /usr/bin/python2-pecan
lrwxrwxrwx 1 root root    9 Mar 26  2019 /usr/bin/python3 -> python3.7
-rwxr-xr-x 2 root root 4.7M Jan 22 15:04 /usr/bin/python3.7
-rwxr-xr-x 2 root root 4.7M Jan 22 15:04 /usr/bin/python3.7m
lrwxrwxrwx 1 root root   10 Mar 26  2019 /usr/bin/python3m -> python3.7m

So, when you (or I) execute 'python -c' we are actually running the python2.7 interpreter. Therefore, I'm not surprised when you ran:

root@pve3:~# python -c 'import rados;print(rados.__file__)'
/usr/lib/python2.7/dist-packages/rados.x86_64-linux-gnu.so

That 'rados' module came from the 2.7 package modules. To be clear, Nautilus is the same way and it does NOT contain the symbol 'rados_mgr_command_target' which I assume was added at some point in Octopus.

Code:

# strings /usr/lib/librados.so.2 | grep rados_mgr_command
rados_mgr_command
librados:rados_mgr_command_exit
librados:rados_mgr_command_cmd
librados:rados_mgr_command_enter

However, your invocation of 'ceph' seems to be invoking python3.7[m] based on the module error. Your posting of the dynamic linking also indicates that it is looking at the wrong 'librados2' library. To reiterate, you had this on the working system:

Code:

ldd /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so
        <snip>
        librados.so.2 => /lib/librados.so.2 (0x00007fc2262d5000)
        <snip>

Your library is in '/lib' while mine is in '/usr/lib', but so be it. However, on the broken system(s), you have:

Code:

ldd /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so: /lib/x86_64-linux-gnu/librados.so.2: no version information available (required by /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so)
        <snip>
        librados.so.2 => /lib/x86_64-linux-gnu/librados.so.2 (0x00007f5bd19cc000)

That is a completely different version of 'librados2' in a different directory and the message "no version information available" usually means the major.minor.patch version of the library does not meet the minimum requirements. So, what installed/replaced that library over the original? Does the original still exist? Did the python module actually change?

That is what you'll need to resolve to "fix things" -- it may happen in time with proper package "dist-upgrades" or it may not. I just don't have enough information to know.

hoffmn01 · May 19, 2021

Hello @RokaKen,

I fixed the issue with the following statements:

Bash:

  unlink /lib/x86_64-linux-gnu/librados.so.2 && ln -s /lib/librados.so.2 /lib/x86_64-linux-gnu/librados.so.2
  unlink /lib/x86_64-linux-gnu/libradosstriper.so.1 && ln -s /lib/libradosstriper.so.1 /lib/x86_64-linux-gnu/libradosstriper.so.1
  unlink /lib/x86_64-linux-gnu/librbd.so.1 && ln -s /lib/librbd.so.1 /lib/x86_64-linux-gnu/librbd.so.1

Thank you and best regards,

Nico Hoffmann

Moayad · May 19, 2021

Hello,

Glad to solve your issue by yourself

Thank you for sharing your solution and thanks @RokaKen

I'll make this thread as solved to help other people who have the same issue.

Have a good day.

Search

Search

[SOLVED] undefined symbol: rados_mgr_command_target

hoffmn01

Active Member

Moayad

Proxmox Staff Member

hoffmn01

Active Member

RokaKen

Active Member

hoffmn01

Active Member

RokaKen

Active Member

hoffmn01

Active Member

RokaKen

Active Member

hoffmn01

Active Member

hoffmn01

Active Member

RokaKen

Active Member

hoffmn01

Active Member

RokaKen

Active Member

hoffmn01

Active Member

RokaKen

Active Member

hoffmn01

Active Member

Moayad

Proxmox Staff Member