pve6: libceph: auth method 'x' error -1

[cloudguy]

On a 4 node Proxmox + Ceph cluster -- Ceph was setup / is managed by Proxmox, seeing the following in the logs:

libceph: auth method 'x' error -1

Kernel:

Linux .. 5.0.21-5-pve #1 SMP PVE 5.0.21-10 (Wed, 13 Nov 2019 08:27:10 +0100) x86_64 GNU/Linux

Versions:

# pveversion
pve-manager/6.0-11/2140ef37 (running kernel: 5.0.21-5-pve)

# pveversion -v
proxmox-ve: 6.0-2 (running kernel: 5.0.21-5-pve)
pve-manager: 6.0-11 (running version: 6.0-11/2140ef37)
pve-kernel-helper: 6.0-12
pve-kernel-5.0: 6.0-11
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.21-2-pve: 5.0.21-7
pve-kernel-5.0.18-1-pve: 5.0.18-3
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph: 14.2.4-pve1
ceph-fuse: 14.2.4-pve1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-3
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-7
libpve-guest-common-perl: 3.0-2
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.0-9
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
openvswitch-switch: 2.10.0+2018.08.28+git.8ca7c82b7d+ds1-12
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-8
pve-cluster: 6.0-7
pve-container: 3.0-10
pve-docs: 6.0-8
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-7
pve-firmware: 3.0-4
pve-ha-manager: 3.0-3
pve-i18n: 2.0-3
pve-qemu-kvm: 4.0.1-5
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-13
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2

 

[Alwin]

In any case, you should update. There has been a corruption bug found in Bluestore and it was fixed with 14.2.5.
https://docs.ceph.com/docs/master/releases/nautilus/#v14-2-5-nautilus

The upgrade also brings a new kernel, where KRBD supports the same features as librbd.

What's in the ceph.conf?

 

[cloudguy]

The bug doesn't impact us, however doesn't excuse us from upgrading either. Can you confirm if this is a kernel issue vs. configuration. - Thank you.

ceph.conf:

# cat /etc/ceph/ceph.conf
[global]
     auth_client_required = cephx
     auth_cluster_required = cephx
     auth_service_required = cephx
     cluster_network = [redacted]
     fsid = [redacted]
     mon_allow_pool_delete = true
     mon_host = [redacted]
     osd_pool_default_min_size = 2
     osd_pool_default_size = 3
     public_network = [redacted]

[client]
     keyring = /etc/pve/priv/$cluster.$name.keyring

[mds]
     keyring = /var/lib/ceph/mds/ceph-$id/keyring

[mds[redacted]
     host =[redacted]
     mds_standby_for_name = pve

[mds.[redacted]]
     host = [redacted]
     mds_standby_for_name = pve

[mds.[redacted]
     host = [redacted]
     mds standby for name = pve

[mds.[redacted]
     host = [redacted]
     mds_standby_for_name = pve

#

 

[Alwin]

The config doesn't look out of the ordinary. Can you please check the ceph logs and dmesg to see if there is anything related in it?

Otherwise you may need to pump up the logging.
https://docs.ceph.com/docs/luminous...g-and-debug/#subsystem-log-and-debug-settings