[SOLVED] ceph mon warning after upgrade

[RobFantini]

Hello
ceph status had a warning about slowness at pve web portal. by the time i went to copy it to paste here that was gone.

however for that mon there is this:

>>> systemctl status ceph-mon@pve15.service
● ceph-mon@pve15.service - Ceph cluster monitor daemon
   Loaded: loaded (/lib/systemd/system/ceph-mon@.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/ceph-mon@.service.d
           └─ceph-after-pve-cluster.conf
   Active: active (running) since Wed 2020-04-22 20:29:32 EDT; 7min ago
 Main PID: 38874 (ceph-mon)
    Tasks: 27
   Memory: 111.1M
   CGroup: /system.slice/system-ceph\x2dmon.slice/ceph-mon@pve15.service
           └─38874 /usr/bin/ceph-mon -f --cluster ceph --id pve15 --setuser ceph --setgroup ceph

Apr 22 20:36:43 pve15 ceph-mon[38874]: 2020-04-22 20:36:43.267 7fabe85f4700 -1 set_mon_vals failed to set mon_crush_min_required_version = firefly: Configuration option 'mon_crush_min_required_version' may not be modified at runtime
Apr 22 20:36:44 pve15 ceph-mon[38874]: 2020-04-22 20:36:44.523 7fabe85f4700 -1 set_mon_vals failed to set mon_crush_min_required_version = firefly: Configuration option 'mon_crush_min_required_version' may not be modified at runtime
Apr 22 20:36:45 pve15 ceph-mon[38874]: 2020-04-22 20:36:45.527 7fabe85f4700 -1 set_mon_vals failed to set mon_crush_min_required_version = firefly: Configuration option 'mon_crush_min_required_version' may not be modified at runtime
Apr 22 20:36:46 pve15 ceph-mon[38874]: 2020-04-22 20:36:46.531 7fabe85f4700 -1 set_mon_vals failed to set mon_crush_min_required_version = firefly: Configuration option 'mon_crush_min_required_version' may not be modified at runtime
Apr 22 20:36:48 pve15 ceph-mon[38874]: 2020-04-22 20:36:48.411 7fabe85f4700 -1 set_mon_vals failed to set mon_crush_min_required_version = firefly: Configuration option 'mon_crush_min_required_version' may not be modified at runtime
Apr 22 20:36:49 pve15 ceph-mon[38874]: 2020-04-22 20:36:49.411 7fabe85f4700 -1 set_mon_vals failed to set mon_crush_min_required_version = firefly: Configuration option 'mon_crush_min_required_version' may not be modified at runtime
Apr 22 20:36:49 pve15 ceph-mon[38874]: 2020-04-22 20:36:49.415 7fabe85f4700 -1 set_mon_vals failed to set mon_crush_min_required_version = firefly: Configuration option 'mon_crush_min_required_version' may not be modified at runtime
Apr 22 20:36:50 pve15 ceph-mon[38874]: 2020-04-22 20:36:50.419 7fabe85f4700 -1 set_mon_vals failed to set mon_crush_min_required_version = firefly: Configuration option 'mon_crush_min_required_version' may not be modified at runtime
Apr 22 20:36:51 pve15 ceph-mon[38874]: 2020-04-22 20:36:51.419 7fabe85f4700 -1 set_mon_vals failed to set mon_crush_min_required_version = firefly: Configuration option 'mon_crush_min_required_version' may not be modified at runtime
Apr 22 20:36:51 pve15 ceph-mon[38874]: 2020-04-22 20:36:51.423 7fabe85f4700 -1 set_mon_vals failed to set mon_crush_min_required_version = firefly: Configuration option 'mon_crush_min_required_version' may not be modified at runtime

I searched the forum and saw some refrences to 'mon_crush_min_required_version' which pointed to this to fix :
https://pve.proxmox.com/wiki/Ceph_Luminous_to_Nautilus#Upgrade_Tunables

ceph crash ls-new does not list any issue.

I check another mon node and it does not have the issue.

at pve15 the warnings continue to generate.

Is the above issue harmless?

 

[RobFantini]

>>>> pveversion -v
proxmox-ve: 6.1-2 (running kernel: 5.4.27-1-pve)
pve-manager: 6.1-8 (running version: 6.1-8/806edfe1)
pve-kernel-5.4: 6.1-8
pve-kernel-helper: 6.1-8
pve-kernel-5.3: 6.1-6
pve-kernel-5.0: 6.0-11
pve-kernel-5.4.27-1-pve: 5.4.27-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-2-pve: 5.3.18-2
pve-kernel-5.0.21-5-pve: 5.0.21-10
ceph: 14.2.9-pve1
ceph-fuse: 14.2.9-pve1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-17
libpve-guest-common-perl: 3.0-5
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-23
pve-docs: 6.1-6
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.0-10
pve-firmware: 3.0-7
pve-ha-manager: 3.0-9
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-4
pve-xtermjs: 4.3.0-1
pve-zsync: 2.0-3
qemu-server: 6.1-7
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

 

[Alwin]

Apr 22 20:36:51 pve15 ceph-mon[38874]: 2020-04-22 20:36:51.423 7fabe85f4700 -1 set_mon_vals failed to set mon_crush_min_required_version = firefly: Configuration option 'mon_crush_min_required_version' may not be modified at runtime

The message tries to say, the setting may not be picked up by the running Ceph daemons. Best set it in the ceph.conf and restart the Ceph daemons. But you only need to do this if your CRUSH tunables are older than Hammer.

 

[RobFantini]

Alwin - thanks for the answer. I'll add that to ceph.conf

also for thread completeness here is the warning that was showing up

Apr 22 19:57:20 pve15 ceph-mon[3126]: 2020-04-22 19:57:20.445 7f4ee9901700 -1 mon.pve15@2(peon) e33 get_health_metrics reporting 3 slow ops, oldest is osd_pgtemp(e173867 {48.56=[2,7]} v173867)
Apr 22 19:57:25 pve15 ceph-mon[3126]: 2020-04-22 19:57:25.445 7f4ee9901700 -1 mon.pve15@2(peon) e33 get_health_metrics reporting 3 slow ops, oldest is osd_pgtemp(e173867 {48.56=[2,7]} v173867)
Apr 22 19:57:30 pve15 ceph-mon[3126]: 2020-04-22 19:57:30.445 7f4ee9901700 -1 mon.pve15@2(peon) e33 get_health_metrics reporting 3 slow ops, oldest is osd_pgtemp(e173867 {48.56=[2,7]} v173867)
..
Apr 22 20:29:10 pve15 ceph-mon[3126]: 2020-04-22 20:29:10.539 7f4ee9901700 -1 mon.pve15@2(peon) e33 get_health_metrics reporting 3 slow ops, oldest is osd_pgtemp(e173867 {48.56=[2,7]} v173867)
Apr 22 20:29:15 pve15 ceph-mon[3126]: 2020-04-22 20:29:15.539 7f4ee9901700 -1 mon.pve15@2(peon) e33 get_health_metrics reporting 3 slow ops, oldest is osd_pgtemp(e173867 {48.56=[2,7]} v173867)
Apr 22 20:29:20 pve15 ceph-mon[3126]: 2020-04-22 20:29:20.539 7f4ee9901700 -1 mon.pve15@2(peon) e33 get_health_metrics reporting 3 slow ops, oldest is osd_pgtemp(e173867 {48.56=[2,7]} v173867)
Apr 22 20:29:25 pve15 ceph-mon[3126]: 2020-04-22 20:29:25.539 7f4ee9901700 -1 mon.pve15@2(peon) e33 get_health_metrics reporting 3 slow ops, oldest is osd_pgtemp(e173867 {48.56=[2,7]} v173867)
Apr 22 20:29:30 pve15 ceph-mon[3126]: 2020-04-22 20:29:30.539 7f4ee9901700 -1 mon.pve15@2(peon) e33 get_health_metrics reporting 3 slow ops, oldest is osd_pgtemp(e173867 {48.56=[2,7]} v173867)

restarting the mon service at the complaining node stopped those.