[SOLVED] undefined symbol: rados_mgr_command_target

hoffmn01 · May 10, 2021

Hello,

We have a three node cluster that has been working for some month now. Today I rebooted one node because of updates and now we are no longer able to access ceph:

root@pve3:~# ceph -s
Traceback (most recent call last):
File "/usr/bin/ceph", line 140, in <module>
import rados
ImportError: /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so: undefined symbol: rados_mgr_command_target

Do you have any recommendation how to solve this? We already tried to reinstall the packages for python rados.

Thank you and best regards,

Nico

Moayad · May 11, 2021

Hi,

How was the upgrade? Was it any message error?

What is your sources in /etc/apt/sources.list and /etc/apt/sources.list.d/*

can you send us the output of pveversion -v

hoffmn01 · May 11, 2021

Hello @Moayad ,

thank you for your reply. I checked with all servers the files and versions (pve3 is the one with the issue):

Bash:

root@pve1:~# pveversion
pve-manager/6.4-5/6c7bf5de (running kernel: 5.4.106-1-pve)

---

root@pve2:~# pveversion
pve-manager/6.4-5/6c7bf5de (running kernel: 5.4.106-1-pve)

---

root@pve3:~# pveversion
pve-manager/6.4-5/6c7bf5de (running kernel: 5.4.106-1-pve)

The content of /etc/apt/sources.list:

Bash:

root@pve1:~# cat /etc/apt/sources.list
deb http://ftp.de.debian.org/debian buster main contrib non-free

deb http://ftp.de.debian.org/debian buster-updates main contrib non-free

# security updates
deb http://security.debian.org buster/updates main contrib non-free

---

root@pve2:~# cat /etc/apt/sources.list
deb http://ftp.de.debian.org/debian buster main contrib

deb http://ftp.de.debian.org/debian buster-updates main contrib

# security updates
deb http://security.debian.org buster/updates main contrib

---

root@pve3:~# cat /etc/apt/sources.list
deb http://ftp.de.debian.org/debian buster main contrib

deb http://ftp.de.debian.org/debian buster-updates main contrib

# security updates
deb http://security.debian.org buster/updates main contrib

Bash:

root@pve1:~# ls -lart /etc/apt/sources.list.d/*
-rw-r--r-- 1 root root 68 May 10  2020 /etc/apt/sources.list.d/pve-enterprise.list
-rw-r--r-- 1 root root 64 Feb 16 16:53 /etc/apt/sources.list.d/ceph.list

---

root@pve2:~# ls -lart /etc/apt/sources.list.d/*
-rw-r--r-- 1 root root 68 Nov 25 16:20 /etc/apt/sources.list.d/pve-enterprise.list
-rw-r--r-- 1 root root 64 Feb 16 15:47 /etc/apt/sources.list.d/ceph.list

---

root@pve3:~# ls -lart /etc/apt/sources.list.d/*
-rw-r--r-- 1 root root 68 Nov 25 16:20 /etc/apt/sources.list.d/pve-enterprise.list
-rw-r--r-- 1 root root 64 May 10 16:31 /etc/apt/sources.list.d/ceph.list

I also checked the versions of the packages, and they are quite identical.

Do you have another hint?

Best regards,
Nico

Moayad · May 11, 2021

THank you for the output.

Please provide us the full output of pveversion -v on the affected node also the output of cat /etc/apt/sources.list.d/ceph.list command

hoffmn01 · May 11, 2021

Hello @Moayad ,

thank you for the quick reply, here comes the output of pveversion -v:

Bash:

root@delbgpm03:~# pveversion -v
proxmox-ve: 6.4-1 (running kernel: 5.4.106-1-pve)
pve-manager: 6.4-5 (running version: 6.4-5/6c7bf5de)
pve-kernel-5.4: 6.4-1
pve-kernel-helper: 6.4-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph: 15.2.11-pve1
ceph-fuse: 15.2.11-pve1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.8
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.4-1
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-2
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-1
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.5-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-3
pve-cluster: 6.4-1
pve-container: 3.3-5
pve-docs: 6.4-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1

And cat /etc/apt/sources.list.d/ceph.list:

Bash:

deb http://download.proxmox.com/debian/ceph-octopus buster main

Thank you and best regards,
Nico

Moayad · May 12, 2021

Hello again,

Thank you for the output

Can you see what's on line 140 in /usr/bin/ceph file? it's a python script you can just see what is at line 140 using the following command:

Bash:

head -n 140 /usr/bin/ceph | tail -n 1

In my workstation the line 140 at ceph is 140- import rados, maybe you don't have the rados in your server, you can check if rados installed using python -c 'import rados;print(rados.__file__)' command, or you can open python and import rados to see if exist.

Moayad · May 12, 2021

Also check the os library in your server if installed:

Bash:

dpkg -S /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so

If not you can install it using:

Bash:

apt install python3-rados

hoffmn01 · May 12, 2021

Hello @Moayad ,

thank you for your help. Here comes the information for the ceph file:

Bash:

root@pve3:~# ceph -s
Traceback (most recent call last):
  File "/usr/bin/ceph", line 140, in <module>
    import rados
ImportError: /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so: undefined symbol: rados_mgr_command_target
root@pve3:~# head -n 140 /usr/bin/ceph | tail -n 1
import rados
root@pve3:~# python -c 'import rados;print(rados.__file__)'
/usr/lib/python2.7/dist-packages/rados.x86_64-linux-gnu.so

It seems to be installed:

Bash:

root@delbgpm03:~# dpkg -S /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so
python3-rados: /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so

I also did the commands on the other (working) hosts, and they react similar.

Thank you and best regards

Nico

Moayad · May 12, 2021

How did you the upgrade, have you used just apt upgrade or apt dist-upgrade

Please try to re-install the rados packet on the affected node:

Bash:

apt install --reinstall python3-rados

hope that helps

hoffmn01 · May 12, 2021

Hi @Moayad,

I did as you requested, but this does not change the scene:

Bash:

root@pve3:~# apt install --reinstall python3-rados
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 0 B/316 kB of archives.
After this operation, 0 B of additional disk space will be used.
(Reading database ... 69938 files and directories currently installed.)
Preparing to unpack .../python3-rados_15.2.11-pve1_amd64.deb ...
Unpacking python3-rados (15.2.11-pve1) over (15.2.11-pve1) ...
Setting up python3-rados (15.2.11-pve1) ...
root@pve3:~# ceph -s
Traceback (most recent call last):
  File "/usr/bin/ceph", line 140, in <module>
    import rados
ImportError: /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so: undefined symbol: rados_mgr_command_target

Do you have any other idea?

Thank you and best regards,

Nico

t.lamprecht · May 12, 2021

It rather seems that librados2 is not installed correctly, as that provides the symbol python3-rados only uses it...

Can you try:

Bash:

apt update
apt install --reinstall librados2

If that doesn't do the trick I'd think that more packages are not correctly installed, then it'd be interesting to know from which PVE version did you upgrade now and which was the first this cluster was setup with?

hoffmn01 · May 12, 2021

Hi @t.lamprecht,

I did as requested, but the scene is unchanged:

Bash:

root@pve3:~# apt update
Get:1 http://security.debian.org buster/updates InRelease [65.4 kB]
Hit:2 http://ftp.de.debian.org/debian buster InRelease
0% [Connecting to download.proxmox.com] [Connecting to enterprise.proxmox.com]
Get:3 http://ftp.de.debian.org/debian buster-updates InRelease [51.9 kB]
Hit:4 http://download.proxmox.com/debian/ceph-octopus buster InRelease
Hit:5 https://enterprise.proxmox.com/debian/pve buster InRelease
Fetched 117 kB in 1s (101 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
root@pve3:~# apt install --reinstall librados2
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 0 B/3,132 kB of archives.
After this operation, 0 B of additional disk space will be used.
(Reading database ... 69938 files and directories currently installed.)
Preparing to unpack .../librados2_15.2.11-pve1_amd64.deb ...
Unpacking librados2 (15.2.11-pve1) over (15.2.11-pve1) ...
Setting up librados2 (15.2.11-pve1) ...
Processing triggers for libc-bin (2.28-10) ...
root@pve3:~# ceph -s
Traceback (most recent call last):
  File "/usr/bin/ceph", line 140, in <module>
    import rados
ImportError: /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so: undefined symbol: rados_mgr_command_target

Do you have any other idea that I might follow?

Thank you and best regards,

Nico

t.lamprecht · May 12, 2021

t.lamprecht said:
If that doesn't do the trick I'd think that more packages are not correctly installed, then it'd be interesting to know from which PVE version did you upgrade now and which was the first this cluster was setup with?

hoffmn01 · May 12, 2021

Hello @t.lamprecht ,

initially this three node cluster was set up with PVE 6.3 and was updated the regular way to the current version with all updates in between. We update each cluster node one by one and reboot them afterwards. We never had this trouble before.

Best regards,

Nico

Moayad · May 12, 2021

hoffmn01 said:
We update each cluster node one by one and reboot them afterwards. We never had this trouble before.

How exactly did the upgrade? have you used dist-upgrade or just apt upgrade

hoffmn01 · May 12, 2021

Hello @Moayad,

we used the function within the GUI, I am not sure which backend function is called from within.

Best regards,

Nico

hoffmn01 · May 12, 2021

Hello @t.lamprecht , @Moayad ,

would you recommend to reinstall this specific node? Is there a documented way what needs to be backed up prior to reinstallation? Is there a way to preserve configuration (certificates, settings and all other configuration) to simply drop that in afterwards?

Thank you and best regards,

Nico

hoffmn01 · May 12, 2021

Hello @t.lamprecht , @Moayad ,

I just started python3 with the -vvv switch from command line and tried to import the rados module first from a working node:

Python:

...
[12.05.2021, 3:30:21,563 PM] >>> import rados
[12.05.2021, 3:30:24,449 PM] # trying /root/rados.cpython-37m-x86_64-linux-gnu.so
[12.05.2021, 3:30:24,449 PM] # trying /root/rados.abi3.so
[12.05.2021, 3:30:24,449 PM] # trying /root/rados.so
[12.05.2021, 3:30:24,449 PM] # trying /root/rados.py
[12.05.2021, 3:30:24,449 PM] # trying /root/rados.pyc
[12.05.2021, 3:30:24,449 PM] # trying /usr/lib/python3.7/rados.cpython-37m-x86_64-linux-gnu.so
[12.05.2021, 3:30:24,449 PM] # trying /usr/lib/python3.7/rados.abi3.so
[12.05.2021, 3:30:24,449 PM] # trying /usr/lib/python3.7/rados.so
[12.05.2021, 3:30:24,449 PM] # trying /usr/lib/python3.7/rados.py
[12.05.2021, 3:30:24,449 PM] # trying /usr/lib/python3.7/rados.pyc
[12.05.2021, 3:30:24,449 PM] # trying /usr/lib/python3.7/lib-dynload/rados.cpython-37m-x86_64-linux-gnu.so
[12.05.2021, 3:30:24,449 PM] # trying /usr/lib/python3.7/lib-dynload/rados.abi3.so
[12.05.2021, 3:30:24,449 PM] # trying /usr/lib/python3.7/lib-dynload/rados.so
[12.05.2021, 3:30:24,449 PM] # trying /usr/lib/python3.7/lib-dynload/rados.py
[12.05.2021, 3:30:24,449 PM] # trying /usr/lib/python3.7/lib-dynload/rados.pyc
[12.05.2021, 3:30:24,449 PM] # trying /usr/local/lib/python3.7/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so
[12.05.2021, 3:30:24,450 PM] # trying /usr/local/lib/python3.7/dist-packages/rados.abi3.so
[12.05.2021, 3:30:24,450 PM] # trying /usr/local/lib/python3.7/dist-packages/rados.so
[12.05.2021, 3:30:24,450 PM] # trying /usr/local/lib/python3.7/dist-packages/rados.py
[12.05.2021, 3:30:24,450 PM] # trying /usr/local/lib/python3.7/dist-packages/rados.pyc
[12.05.2021, 3:30:24,450 PM] # trying /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so
[12.05.2021, 3:30:24,450 PM] # extension module 'rados' loaded from '/usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so'
[12.05.2021, 3:30:24,469 PM] root/copyreg.cpython37m-x86_64-linx-gnu.soroot/copyreg.abi3.oroot/copyreg.so
[12.05.2021, 3:30:24,470 PM] # trying /root/copyeg.py
[12.05.2021, 3:30:24,470 PM] # rying /root/copyreg.pyc
[12.05.2021, 3:30:24,470 PM] # trying /usr/lib/python3.7/copyreg.cpython-37m-x86_64-linux-gnu.so
[12.05.2021, 3:30:24,470 PM] # trying /usr/lib/pyton3.7/copyreg.abi3.sousr/lib/python3.7/cpyreg.sousr/lib/python3.7/copyreg.py/usr/lib/pyhon3.7/__pycache__/copyreg.cpython-37.pyc matches /usr/lib/python3.7/copyreg.pycode object fom '/usr/lib/python3.7/__pycache__/copyreg.cpython-37.pyc'
[12.05.2021, 3:30:24,470 PM] import 'copyreg' # <_frozen_importlib_external.SourceFileLoader object at 0x7fae9a916ba8>
[12.05.2021, 3:30:24,470 PM] import 're' # <_frozen_importlib_external.SourceFileLoader object at 0x7fae9a95b470>root/token.cpython-37m-x86_64-linux-gnu.soroot/token.abisoroot/token.sroot/token.pyroot/token.pyctryig /usr/lib/python3.7/token.cpythn-37m-x86_64-linux-gnu.sotryig /usr/lib/python3.7/token.abi3.so
[12.05.2021, 3:30:24,470 PM] # trying /usr/lib/python3.7/token.so
[12.05.2021, 3:30:24,470 PM] # trying /usr/lib/python3.7/tk.py/usr/lib/pyhon3.7/__pycahe__/token.cpython-37.pyc matches /usr/lib/python3.7/token.pycode object fom '/usr/lib/pythn3.7/__pycache__/token.cpython-37.pyc'
[12.05.2021, 3:30:24,470 PM] import 'token' # <_frozen_importlib_external.SourceFileLoader object at 0x7fae9a916e80>
[12.05.2021, 3:30:24,470 PM] import 'tokenize' # <_frozen_importlib_external.SourceFileLoader object at 0x7fae9a941208>
[12.05.2021, 3:30:24,470 PM] import 'linecache' # <_frozen_importlib_external.SourceFileLoader object at 0x7fae9a9393c8>
[12.05.2021, 3:30:24,470 PM] import 'traceback' # <_frozen_importlib_external.SourceFileLoader object at 0x7fae9a9b3278>root/_weakrefset.cpython-37m-x86_64-linux-gnu.soroot/_weakrefset.abi3.soroot/_weakrefset.sotrying /roo/_weakrefset.pytrying /root/_weakrefset.pyc
[12.05.2021, 3:30:24,470 PM] # trying /usr/ib/python3.7/_weakrefset.cpython-37m-x86_64-inux-gnu.so
[12.05.2021, 3:30:24,470 PM] # trying /usr/lib/python3.7/_weakrefset.abi3.sousr/lib/python3.7/_weakrefset.sousr/lib/python7/_weakrefset.py/usr/lib/pyhon3.7/__pycache__/_weakrefset.cpython-37.pyc matches /usr/lib/python3.7/_weakrefset.pycode object fom '/usr/lib/python3.7/__pycache__/_weakrefset.cpython-37.pyc'
[12.05.2021, 3:30:24,470 PM] import '_weakrefset' # <_frozen_importlib_external.SourceFileLoader object at 0x7fae9a9395f8>
[12.05.2021, 3:30:24,470 PM] import 'threading' # <_frozen_importlib_external.SourceFileLoader object at 0x7faea46db7f0>root/datetime.cpython-37m-x86_64-linux-gnu.soroot/datetime.abi3.soroot/datetime.soroot/datetime.pyroot/datetime.pycdatetime.cpython-37m-x86_64-linux-gnu.sodatetime.bi3.sdatetime.sodatetime.py/usr/lib/pythn3.7/__pycache__/dateime.cpython-37.pyc matches /usr/lib/python3.7/datetime.pycode object from '/usr/lib/python3.7/__pycache__/datetme.cpython-37.pyc'
[12.05.2021, 3:30:24,470 PM] import 'math' # <cass '_frozen_imporlib.BuiltinImporter'>
[12.05.2021, 3:30:24,470 PM] import '_datetime' # <class '_frozen_imortlib.BuiltinImporter'>
[12.05.2021, 3:30:24,470 PM] import 'datetime' # <_frozen_importlib_external.SourceFileLoader object at 0x7fae9a92bcf8>extesion module 'rados' executed from '/ur/lib/python3/dist-packages/rads.cpython-37m-x86_64-linux-gnu.so'
[12.05.2021, 3:30:24,470 PM] import 'rados' # <_fozen_importlib_external.ExtensionFileLoader objct t 0x7faea46c1c18>
...

and then from the broken node:

Python:

...
[12.05.2021, 3:32:47,814 PM] >>> import rados
[12.05.2021, 3:32:50,411 PM] # trying /root/rados.cpython-37m-x86_64-linux-gnu.so
[12.05.2021, 3:32:50,411 PM] # trying /root/rados.abi3.so
[12.05.2021, 3:32:50,411 PM] # trying /root/rados.so
[12.05.2021, 3:32:50,411 PM] # trying /root/rados.py
[12.05.2021, 3:32:50,411 PM] # trying /root/rados.pyc
[12.05.2021, 3:32:50,411 PM] # trying /usr/lib/python3.7/rados.cpython-37m-x86_64-linux-gnu.so
[12.05.2021, 3:32:50,411 PM] # trying /usr/lib/python3.7/rados.abi3.so
[12.05.2021, 3:32:50,411 PM] # trying /usr/lib/python3.7/rados.so
[12.05.2021, 3:32:50,411 PM] # trying /usr/lib/python3.7/rados.py
[12.05.2021, 3:32:50,411 PM] # trying /usr/lib/python3.7/rados.pyc
[12.05.2021, 3:32:50,411 PM] # trying /usr/lib/python3.7/lib-dynload/rados.cpython-37m-x86_64-linux-gnu.so
[12.05.2021, 3:32:50,411 PM] # trying /usr/lib/python3.7/lib-dynload/rados.abi3.so
[12.05.2021, 3:32:50,411 PM] # trying /usr/lib/python3.7/lib-dynload/rados.so
[12.05.2021, 3:32:50,411 PM] # trying /usr/lib/python3.7/lib-dynload/rados.py
[12.05.2021, 3:32:50,411 PM] # trying /usr/lib/python3.7/lib-dynload/rados.pyc
[12.05.2021, 3:32:50,411 PM] # trying /usr/local/lib/python3.7/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so
[12.05.2021, 3:32:50,411 PM] # trying /usr/local/lib/python3.7/dist-packages/rados.abi3.so
[12.05.2021, 3:32:50,412 PM] # trying /usr/local/lib/python3.7/dist-packages/rados.so
[12.05.2021, 3:32:50,412 PM] # trying /usr/local/lib/python3.7/dist-packages/rados.py
[12.05.2021, 3:32:50,412 PM] # trying /usr/local/lib/python3.7/dist-packages/rados.pyc
[12.05.2021, 3:32:50,412 PM] # trying /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so
[12.05.2021, 3:32:50,435 PM] Traceback (most recent call last):
[12.05.2021, 3:32:50,435 PM]   File "<stdin>", line 1, in <module>
[12.05.2021, 3:32:50,435 PM]   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
[12.05.2021, 3:32:50,435 PM]   File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
[12.05.2021, 3:32:50,435 PM]   File "<frozen importlib._bootstrap>", line 670, in _load_unlocked
[12.05.2021, 3:32:50,435 PM]   File "<frozen importlib._bootstrap>", line 583, in module_from_spec
[12.05.2021, 3:32:50,435 PM]   File "<frozen importlib._bootstrap_external>", line 1043, in create_module
[12.05.2021, 3:32:50,436 PM]   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
[12.05.2021, 3:32:50,436 PM] ImportError: /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so: undefined symbol: rados_mgr_command_target
            ...

Nevertheless the md5sum of the file is the same on both nodes:

Python:

md5sum /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so
4eaa9de85bdeadfb60e1d68a4ff4f60d  /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so

Do you have any idea for me?

Best regards,

Nico

Moayad · May 12, 2021

Let's more debugging first

Could you please do the following:
-- Post the output of list packages matching ceph.

Bash:

dpkg --list | grep ceph

-- Post the shared objects for the rados.os

Bash:

ldd /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so

-- Do update and upgrade and -f flag as well

Bash:

apt update && apt dist-upgrade && apt -f install

hoffmn01 · May 12, 2021

Hello @Moayad ,

here comes the output of dpkg --list | grep ceph:

Bash:

ii  ceph                                 15.2.11-pve1                  amd64        distributed storage and file system
ii  ceph-base                            15.2.11-pve1                  amd64        common ceph daemon libraries and management tools
ii  ceph-common                          15.2.11-pve1                  amd64        common utilities to mount and interact with a ceph storage cluster
ii  ceph-fuse                            15.2.11-pve1                  amd64        FUSE-based client for the Ceph distributed file system
ii  ceph-mds                             15.2.11-pve1                  amd64        metadata server for the ceph distributed file system
ii  ceph-mgr                             15.2.11-pve1                  amd64        manager for the ceph distributed storage system
ii  ceph-mgr-modules-core                15.2.11-pve1                  all          ceph manager modules which are always enabled
ii  ceph-mon                             15.2.11-pve1                  amd64        monitor server for the ceph storage system
ii  ceph-osd                             15.2.11-pve1                  amd64        OSD server for the ceph storage system
ii  libcephfs2                           15.2.11-pve1                  amd64        Ceph distributed file system client library
ii  python-cephfs                        12.2.11+dfsg1-2.1+b1          amd64        Python libraries for the Ceph libcephfs library
ii  python3-ceph-argparse                15.2.11-pve1                  all          Python 3 utility libraries for Ceph CLI
ii  python3-ceph-common                  15.2.11-pve1                  all          Python 3 utility libraries for Ceph
ii  python3-cephfs                       15.2.11-pve1                  amd64        Python 3 libraries for the Ceph libcephfs library

Interestingly the output of the ldd command on the working nodes (both the same) is:

Bash:

ldd /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffd70fe6000)
        librados.so.2 => /lib/librados.so.2 (0x00007fc2262d5000)
        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fc22629b000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc22627a000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc226275000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fc226270000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc2260ed000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc225f2a000)
        libceph-common.so.2 => /usr/lib/ceph/libceph-common.so.2 (0x00007fc21d287000)
        libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 (0x00007fc21d232000)
        libcrypto.so.1.1 => /lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007fc21cf49000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fc21cf3f000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fc21cf25000)
        libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007fc21cefd000)
        libibverbs.so.1 => /lib/x86_64-linux-gnu/libibverbs.so.1 (0x00007fc21cee2000)
        librdmacm.so.1 => /lib/x86_64-linux-gnu/librdmacm.so.1 (0x00007fc21cec5000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc21cca7000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fc21cb23000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc21cb09000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fc22655f000)
        libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007fc21cafe000)
        libnl-route-3.so.200 => /lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x00007fc21c885000)
        libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007fc21c664000)

The broken node has:

Bash:

ldd /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so: /lib/x86_64-linux-gnu/librados.so.2: no version information available (required by /usr/lib/python3/dist-packages/rados.cpython-37m-x86_64-linux-gnu.so)
        linux-vdso.so.1 (0x00007fffe351b000)
        librados.so.2 => /lib/x86_64-linux-gnu/librados.so.2 (0x00007f5bd19cc000)
        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f5bd1992000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5bd1971000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5bd196c000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f5bd1967000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5bd17e4000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5bd1621000)
        libceph-common.so.0 => /usr/lib/x86_64-linux-gnu/ceph/libceph-common.so.0 (0x00007f5bc8b59000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5bc8b4f000)
        libboost_thread.so.1.67.0 => /lib/x86_64-linux-gnu/libboost_thread.so.1.67.0 (0x00007f5bc8b23000)
        libboost_system.so.1.67.0 => /lib/x86_64-linux-gnu/libboost_system.so.1.67.0 (0x00007f5bc8b1c000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5bc8998000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5bc897c000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f5bd1c63000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f5bc8962000)
        libboost_regex.so.1.67.0 => /lib/x86_64-linux-gnu/libboost_regex.so.1.67.0 (0x00007f5bc884d000)
        libboost_iostreams.so.1.67.0 => /lib/x86_64-linux-gnu/libboost_iostreams.so.1.67.0 (0x00007f5bc882f000)
        libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 (0x00007f5bc87da000)
        libsmime3.so => /lib/x86_64-linux-gnu/libsmime3.so (0x00007f5bc87a9000)
        libnss3.so => /lib/x86_64-linux-gnu/libnss3.so (0x00007f5bc865b000)
        libnspr4.so => /lib/x86_64-linux-gnu/libnspr4.so (0x00007f5bc861a000)
        libibverbs.so.1 => /lib/x86_64-linux-gnu/libibverbs.so.1 (0x00007f5bc85ff000)
        libboost_atomic.so.1.67.0 => /lib/x86_64-linux-gnu/libboost_atomic.so.1.67.0 (0x00007f5bc85fa000)
        libicudata.so.63 => /lib/x86_64-linux-gnu/libicudata.so.63 (0x00007f5bc6c08000)
        libicui18n.so.63 => /lib/x86_64-linux-gnu/libicui18n.so.63 (0x00007f5bc692d000)
        libicuuc.so.63 => /lib/x86_64-linux-gnu/libicuuc.so.63 (0x00007f5bc675e000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5bc6540000)
        libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f5bc652d000)
        libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f5bc6524000)
        libnssutil3.so => /lib/x86_64-linux-gnu/libnssutil3.so (0x00007f5bc64f0000)
        libplc4.so => /lib/x86_64-linux-gnu/libplc4.so (0x00007f5bc64e9000)
        libplds4.so => /lib/x86_64-linux-gnu/libplds4.so (0x00007f5bc64e4000)
        libnl-route-3.so.200 => /lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x00007f5bc626b000)
        libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x00007f5bc604a000)

So the broken node has a lot more here.

The update did the following:

Bash:

apt update && apt dist-upgrade && apt -f install
Hit:1 http://security.debian.org buster/updates InRelease
Hit:2 http://ftp.de.debian.org/debian buster InRelease
Hit:3 http://ftp.de.debian.org/debian buster-updates InRelease
Hit:4 http://download.proxmox.com/debian/ceph-octopus buster InRelease
Hit:5 https://enterprise.proxmox.com/debian/pve buster InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

How to we get node 3 back to the standard for the shared objects?

Best regards,

Nico

[SOLVED] undefined symbol: rados_mgr_command_target

Member

Proxmox Staff Member

Member

Proxmox Staff Member

Member

Proxmox Staff Member

Proxmox Staff Member

Member

Proxmox Staff Member

Member

Proxmox Staff Member

Member

Proxmox Staff Member

Member

Proxmox Staff Member

Member

Member

Member

Proxmox Staff Member

Member