[SOLVED] cephadm: No module named 'remoto' after PVE update

[lesicnik1]

Hello,

I recently updated all three of our PVE hosts, that are also in a ceph cluster. During the update Ceph was updated to 16.2.9, however now I'm getting 3 warning for the ceph cluster.

Module 'cephadm' has failed dependency: loading remoto library:No module named 'remoto'

3 stray host(s) with 21 daemon(s) not managed by cephadm

21 stray daemon(s) not managed by cephadm

I've tested with the Python shell and I can import remoto as well as execute a command on another server.
I tried running pip install --upgrade remoto, but nothing was updated. pip list reports that remoto 1.2.1 is installed.

PVE Package versions:

proxmox-ve: 7.2-1 (running kernel: 5.15.64-1-pve)
pve-manager: 7.2-11 (running version: 7.2-11/b76d3178)
pve-kernel-5.15: 7.2-13
pve-kernel-helper: 7.2-13
pve-kernel-5.15.64-1-pve: 5.15.64-1
pve-kernel-5.15.35-1-pve: 5.15.35-3
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph: 16.2.9-pve1
ceph-fuse: 16.2.9-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-3
libpve-guest-common-perl: 4.1-4
libpve-http-server-perl: 4.1-4
libpve-storage-perl: 7.2-10
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
openvswitch-switch: 2.15.0+ds1-2+deb11u1
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-3
pve-docs: 7.2-2
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-6
pve-firmware: 3.5-6
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 7.0.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1

Full output of pip list:

Package          Version
---------------- --------------
bcrypt           3.1.7
beautifulsoup4   4.9.3
ceph             1.0.0
ceph-volume      1.0.0
cephfs           2.0.0
certifi          2020.6.20
chardet          4.0.0
CherryPy         8.9.1
crit             0.0.1
cryptography     3.3.2
execnet          1.9.0
httplib2         0.18.1
idna             2.10
Jinja2           2.11.3
jsonpatch        1.25
jsonpointer      2.0
logutils         0.3.3
Mako             1.1.3
MarkupSafe       1.1.1
Paste            3.5.0
PasteDeploy      2.1.1
pecan            1.3.3
pip              20.3.4
prettytable      0.7.2
protobuf         3.12.4
pycurl           7.43.0.6
PyJWT            1.7.1
pyOpenSSL        20.0.1
PySimpleSOAP     1.16.2
python-apt       2.2.1
python-dateutil  2.8.1
python-debian    0.1.39
python-debianbts 3.1.0
PyYAML           5.3.1
rados            2.0.0
rbd              2.0.0
remoto           1.2.1
reportbug        7.10.3+deb11u1
repoze.lru       0.7
requests         2.25.1
rgw              2.0.0
Routes           2.5.1
setuptools       52.0.0
simplegeneric    0.8.1
six              1.16.0
soupsieve        2.2.1
Tempita          0.5.2
urllib3          1.26.5
waitress         1.4.4
WebOb            1.8.6
WebTest          2.0.35
Werkzeug         1.0.1
wheel            0.34.2

 

[gurubert]

AFAIK Proxmox does not use cephadm to deploy its Ceph cluster.

Have you tried to disable this MGR module?

 

[lesicnik1]

AFAIK Proxmox does not use cephadm to deploy its Ceph cluster.

Have you tried to disable this MGR module?

Looking a bit, seems like cephadm is a management layer, which I guess PVE does itself.

Would disabling cephadm be OK with regards to the cluster? I wouldn't like our environment to go down

 

[aaron]

As mentioned, cephadm is the current deployment and management provided by Ceph itself. It is not compatible if you deploy Ceph via Proxmox VE.

I assume that somehow the ceph-mgr-cephadm package was installed.
You can check it by running apt search ceph-mgr-cephadm and see if you have a [installed] at the end of the line.

 

[lesicnik1]

As mentioned, cephadm is the current deployment and management provided by Ceph itself. It is not compatible if you deploy Ceph via Proxmox VE.

I assume that somehow the ceph-mgr-cephadm package was installed.
You can check it by running apt search ceph-mgr-cephadm and see if you have a [installed] at the end of the line.

ceph-mgr-cephadm is indeed installed. Strangely enough, not installed on my other 3 server cluster (they are more or less identical in setup)

Would uninstalling it cause any issues?

 

[aaron]

Shouldn't be an issue. If you get those errors from cephadm, I assume that the active MON is currently the node with the package installed. If you run ceph mgr module ls and search the output, you should see cephadm. If you stop that MON and wait until one of the other nodes, without the cephadm package, become active, the same list of modules should not have cephadm listed.

 

[lesicnik1]

Shouldn't be an issue. If you get those errors from cephadm, I assume that the active MON is currently the node with the package installed. If you run ceph mgr module ls and search the output, you should see cephadm. If you stop that MON and wait until one of the other nodes, without the cephadm package, become active, the same list of modules should not have cephadm listed.

Thanks for your assistance, I've disabled the cephadm mgr manager module and everything seems to still be ok.

 

[Bokki Ryu]

In my case, I operate ceph manual cluster without proxmox. I met above issue when i convert manual cluster to cephadm.

cephadm --image ceph-mon@${ceph-mon-node-hostname}

cephadm --image ceph-mgr@${ceph-mgr-node-hostname}

After I execute above two commands, "ceph mgr module enable cephadm" command run well.

[My Ceph Environment]
OS: Debian 12
Ceph: v16.2