[SOLVED] Ceph - Unable to remove config entry

David Herselman · Dec 10, 2019

We assimilated Ceph's ceph.conf file during our Nautilus upgrade and subsequently have a minimal configuration file. We are however now unable to remove configuration entries.

Code:

[admin@kvm1b ~]# ceph config dump | grep -e WHO -e rbd_default_features
WHO    MASK LEVEL    OPTION                             VALUE          RO
global      advanced rbd_default_features               7

[admin@kvm1b ~]# ceph config rm global rbd_default_features
[admin@kvm1b ~]# ceph config rm global rbd_default_features
[admin@kvm1b ~]# ceph config rm global rbd_default_features

[admin@kvm1b ~]# ceph config dump | grep -e WHO -e rbd_default_features
WHO    MASK LEVEL    OPTION                             VALUE          RO
global      advanced rbd_default_features               7

[admin@kvm1b ~]# ceph config set global rbd_default_features 31
[admin@kvm1b ~]# ceph config dump | grep -e WHO -e rbd_default_features
WHO    MASK LEVEL    OPTION                             VALUE          RO
global      advanced rbd_default_features               7
global      advanced rbd_default_features               31

Any ideas how I fix this?

Alwin · Dec 10, 2019

Can you please share your ceph.conf? And is there anything in the ceph logs? As I it should override the config setting, not add another one.

David Herselman · Dec 10, 2019

Ceph config file:

Code:

[global]
 cluster_network = 10.248.1.0/24
 filestore_xattr_use_omap = true
 fsid = 31f6ea46-12cb-47e8-a6f3-60fb6bbd1782
 mon_host = 10.248.1.60 10.248.1.61 10.248.1.62
 public_network = 10.248.1.0/24

[client]
 keyring = /etc/pve/priv/$cluster.$name.keyring

Alwin · Dec 11, 2019

Is the cluster healthy? And on which ceph versions are you?

EDIT: And is there anything in the ceph logs?

David Herselman · Dec 11, 2019

Ceph Nautilus 14.2.4.1,

Herewith our Ceph Nautilus upgrade nodes that we ran through on 7 separate clusters:

Code:

cd /etc/pve;
ceph config assimilate-conf -i ceph.conf -o ceph.conf.new;
mv ceph.conf.new ceph.conf;
pico /etc/ceph/ceph.conf
  # add back: cluster_network
  #           public_network
ceph config rm global cluster_network;
ceph config rm global public_network;
ceph config set global mon_osd_down_out_subtree_limit host;

Ceph configuration dump:

Code:

[admin@kvm1c ~]# ceph config dump
WHO    MASK LEVEL    OPTION                             VALUE          RO
global      advanced auth_client_required               cephx          *
global      advanced auth_cluster_required              cephx          *
global      advanced auth_service_required              cephx          *
global      advanced cluster_network                    10.248.1.0/24  *
global      advanced debug_filestore                    0/0
global      advanced debug_journal                      0/0
global      advanced debug_ms                           0/0
global      advanced debug_osd                          0/0
global      basic    device_failure_prediction_mode     cloud
global      advanced mon_allow_pool_delete              true
global      advanced mon_osd_down_out_subtree_limit     host
global      advanced osd_deep_scrub_interval            1209600.000000
global      advanced osd_pool_default_min_size          2
global      advanced osd_pool_default_size              3
global      advanced osd_scrub_begin_hour               19
global      advanced osd_scrub_end_hour                 6
global      advanced osd_scrub_sleep                    0.100000
global      advanced public_network                     10.248.1.0/24  *
global      advanced rbd_default_features               7
global      advanced rbd_default_features               31
  mgr       advanced mgr/balancer/active                true
  mgr       advanced mgr/balancer/mode                  upmap
  mgr       advanced mgr/devicehealth/enable_monitoring true

Code:

proxmox-ve: 6.1-2 (running kernel: 5.3.10-1-pve)
pve-manager: 6.1-3 (running version: 6.1-3/37248ce6)
pve-kernel-5.3: 6.0-12
pve-kernel-helper: 6.0-12
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph: 14.2.4.1-pve1
ceph-fuse: 14.2.4.1-pve1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-9
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-2
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
openvswitch-switch: 2.10.0+2018.08.28+git.8ca7c82b7d+ds1-12+deb10u1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-1
pve-cluster: 6.1-2
pve-container: 3.0-14
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191002-1
pve-firewall: 4.0-9
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 3.13.2-1
qemu-server: 6.1-2
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2

David Herselman · Dec 11, 2019

Apologies, yes the cluster is healthy:

Code:

[admin@kvm1c ~]# ceph -s
  cluster:
    id:     31f6ea46-12cb-47e8-a6f3-60fb6bbd1782
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum kvm1a,kvm1b,kvm1c (age 20h)
    mgr: kvm1c(active, since 20h), standbys: kvm1b, kvm1a
    mds: cephfs:1 {0=kvm1c=up:active} 2 up:standby
    osd: 30 osds: 30 up (since 20h), 30 in (since 20h)

  data:
    pools:   8 pools, 417 pgs
    objects: 1.46M objects, 5.3 TiB
    usage:   15 TiB used, 75 TiB / 90 TiB avail
    pgs:     417 active+clean

  io:
    client:   45 KiB/s rd, 4.5 MiB/s wr, 6 op/s rd, 354 op/s wr
    cache:    0 op/s promote

Alwin · Dec 11, 2019

When you try to delete the config setting is there anything in the ceph logs, that indicates why there might be two entries?

Another possibility could be that the setting is locked by a service and therefore not deleted. But I guess, that would be visible in the logs.

David Herselman · Dec 11, 2019

Hrm... Another user on the Ceph mailing list appears to have confirmed this as a bug since he upgraded to Mimic...

Alwin · Dec 12, 2019

David Herselman said:
Hrm... Another user on the Ceph mailing list appears to have confirmed this as a bug since he upgraded to Mimic...

Do you have the thread at hand? And ist there a bug report?
https://tracker.ceph.com/projects/ceph

David Herselman · Dec 12, 2019

Herewith the thread in the Ceph user mailing list. I'll log a bug report shortly...

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-December/037701.html

Alwin · Dec 12, 2019

David Herselman said:
Herewith the thread in the Ceph user mailing list. I'll log a bug report shortly...

Too bad. Maybe the ceph config-key command can unset the option. But this is just a shot in the dark.
https://docs.ceph.com/docs/nautilus/man/8/ceph/#config-key

David Herselman · Dec 13, 2019

I've been lost in trying to figure out how to use the 'ceph config-key' command and unfortunately don't recall figuring out how to remove an entry...

Logged a bug with Ceph: https://tracker.ceph.com/issues/43296

Alwin · Dec 13, 2019

David Herselman said:
Logged a bug with Ceph: https://tracker.ceph.com/issues/43296

Thanks. That might give it more attention.

Code:

ceph daemon mon.<id> config set debug_rbd 20/20

You can see the config set/rm with the higher debug level (/var/log/ceph/ceph-mon.<id>.log). This may hopefully shine some light on the situation.

EDIT: 0/5 is the default log value for rbd.

David Herselman · Dec 13, 2019

Setting this to 20/20 or 5/5 only generates the following debug information:

Enable debugging:

Code:

[admin@kvm1a ~]# ceph daemon mon.kvm1a config show | grep -i debug_rbd
  #     "debug_rbd": "0/5",
  #     "debug_rbd_mirror": "0/5",
  #     "debug_rbd_replay": "0/5",
[admin@kvm1a ~]# ceph daemon mon.kvm1a config set debug_rbd 20/20
[admin@kvm1a ~]# ceph daemon mon.kvm1a config show | grep -i debug_rbd
  #     "debug_rbd": "20/20",
  #     "debug_rbd_mirror": "0/5",
  #     "debug_rbd_replay": "0/5",

Commands:

Code:

[admin@kvm1a ~]# ceph config dump | grep -e WHO -e rbd_default_features
WHO    MASK LEVEL    OPTION                             VALUE          RO
global      advanced rbd_default_features               7
global      advanced rbd_default_features               31
[admin@kvm1a ~]# ceph config rm global rbd_default_features
[admin@kvm1a ~]# ceph config rm global rbd_default_features
[admin@kvm1a ~]# ceph config rm global rbd_default_features
[admin@kvm1a ~]# ceph config rm global rbd_default_features
[admin@kvm1a ~]# ceph config dump | grep -e WHO -e rbd_default_features
WHO    MASK LEVEL    OPTION                             VALUE          RO
global      advanced rbd_default_features               7
[admin@kvm1a ~]# ceph config set global rbd_default_features 31
[admin@kvm1a ~]# ceph config dump | grep -e WHO -e rbd_default_features
WHO    MASK LEVEL    OPTION                             VALUE          RO
global      advanced rbd_default_features               7
global      advanced rbd_default_features               31

Log entries (tail -f /var/log/ceph/ceph-mon.kvm1a.log):

Code:

2019-12-13 18:15:29.386 7fc9eff72700  0 mon.kvm1a@1(peon) e4 handle_command mon_command({"prefix": "config dump"} v 0) v1
2019-12-13 18:15:29.386 7fc9eff72700  0 log_channel(audit) log [DBG] : from='client.67360717 10.248.1.61:0/1627956866' entity='client.admin' cmd=[{"prefix": "config dump"}]: dispatch
2019-12-13 18:15:35.898 7fc9eff72700  0 mon.kvm1a@1(peon) e4 handle_command mon_command({"prefix": "config rm", "who": "global", "name": "rbd_default_features"} v 0) v1
2019-12-13 18:15:35.898 7fc9eff72700  0 log_channel(audit) log [INF] : from='client.67360783 10.248.1.61:0/472300355' entity='client.admin' cmd=[{"prefix": "config rm", "who": "global", "name": "rbd_default_features"}]: dispatch
2019-12-13 18:15:36.422 7fc9eff72700  0 mon.kvm1a@1(peon) e4 handle_command mon_command({"prefix": "config rm", "who": "global", "name": "rbd_default_features"} v 0) v1
2019-12-13 18:15:36.422 7fc9eff72700  0 log_channel(audit) log [INF] : from='client.? 10.248.1.61:0/3618748536' entity='client.admin' cmd=[{"prefix": "config rm", "who": "global", "name": "rbd_default_features"}]: dispatch
2019-12-13 18:15:37.258 7fc9eff72700  0 mon.kvm1a@1(peon) e4 handle_command mon_command({"prefix": "config rm", "who": "global", "name": "rbd_default_features"} v 0) v1
2019-12-13 18:15:37.258 7fc9eff72700  0 log_channel(audit) log [INF] : from='client.? 10.248.1.61:0/605608008' entity='client.admin' cmd=[{"prefix": "config rm", "who": "global", "name": "rbd_default_features"}]: dispatch
2019-12-13 18:15:39.710 7fc9eff72700  0 mon.kvm1a@1(peon) e4 handle_command mon_command({"prefix": "config dump"} v 0) v1
2019-12-13 18:15:39.710 7fc9eff72700  0 log_channel(audit) log [DBG] : from='client.? 10.248.1.61:0/2007128611' entity='client.admin' cmd=[{"prefix": "config dump"}]: dispatch
2019-12-13 18:15:44.682 7fc9eff72700  0 mon.kvm1a@1(peon) e4 handle_command mon_command({"prefix": "config set", "who": "global", "name": "rbd_default_features", "value": "31"} v 0) v1
2019-12-13 18:15:44.682 7fc9eff72700  0 log_channel(audit) log [INF] : from='client.67360882 10.248.1.61:0/3247534721' entity='client.admin' cmd=[{"prefix": "config set", "who": "global", "name": "rbd_default_features", "value": "31"}]: dispatch
2019-12-13 18:15:47.394 7fc9eff72700  0 mon.kvm1a@1(peon) e4 handle_command mon_command({"prefix": "config dump"} v 0) v1
2019-12-13 18:15:47.394 7fc9eff72700  0 log_channel(audit) log [DBG] : from='client.67360924 10.248.1.61:0/2411181763' entity='client.admin' cmd=[{"prefix": "config dump"}]: dispatch

Hrm... why only 3 ceph 'config rm' references when the commands were run 4 times?

David Herselman · Dec 13, 2019

Commands sometimes land on other monitors, herewith simultaneous debugs from all three monitors.

Commands:

Code:

[root@kvm1a ~]# ceph config dump | grep -e WHO -e rbd_default_features
WHO    MASK LEVEL    OPTION                             VALUE          RO
global      advanced rbd_default_features               7
global      advanced rbd_default_features               31
[root@kvm1a ~]# ceph config rm global rbd_default_features
[root@kvm1a ~]# ceph config rm global rbd_default_features
[root@kvm1a ~]# ceph config rm global rbd_default_features
[root@kvm1a ~]# ceph config rm global rbd_default_features
[root@kvm1a ~]# ceph config dump | grep -e WHO -e rbd_default_features
WHO    MASK LEVEL    OPTION                             VALUE          RO
global      advanced rbd_default_features               7
[root@kvm1a ~]# ceph config set global rbd_default_features 31
[root@kvm1a ~]# ceph config dump | grep -e WHO -e rbd_default_features
WHO    MASK LEVEL    OPTION                             VALUE          RO
global      advanced rbd_default_features               7
global      advanced rbd_default_features               31

Logs:
kvm1a:

Code:

2019-12-13 18:35:26.293 7feaac2f5700  0 mon.kvm1a@0(leader) e4 handle_command mon_command({"prefix": "config rm", "who": "global", "name": "rbd_default_features"} v 0) v1
2019-12-13 18:35:26.293 7feaac2f5700  0 log_channel(audit) log [INF] : from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "global", "name": "rbd_default_features"}]: dispatch
2019-12-13 18:35:26.305 7feaaaaf2700  0 log_channel(audit) log [INF] : from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "global", "name": "rbd_default_features"}]': finished
2019-12-13 18:35:30.401 7feaac2f5700  0 mon.kvm1a@0(leader) e4 handle_command mon_command({"prefix": "config rm", "who": "global", "name": "rbd_default_features"} v 0) v1
2019-12-13 18:35:30.401 7feaac2f5700  0 log_channel(audit) log [INF] : from='client.? 10.248.1.60:0/2894657506' entity='client.admin' cmd=[{"prefix": "config rm", "who": "global", "name": "rbd_default_features"}]: dispatch
2019-12-13 18:35:32.289 7feaac2f5700  0 mon.kvm1a@0(leader) e4 handle_command mon_command({"prefix": "config rm", "who": "global", "name": "rbd_default_features"} v 0) v1
2019-12-13 18:35:32.289 7feaac2f5700  0 log_channel(audit) log [INF] : from='client.? 10.248.1.60:0/3241273675' entity='client.admin' cmd=[{"prefix": "config rm", "who": "global", "name": "rbd_default_features"}]: dispatch
2019-12-13 18:35:33.833 7feaac2f5700  0 mon.kvm1a@0(leader) e4 handle_command mon_command({"prefix": "config rm", "who": "global", "name": "rbd_default_features"} v 0) v1
2019-12-13 18:35:33.833 7feaac2f5700  0 log_channel(audit) log [INF] : from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "global", "name": "rbd_default_features"}]: dispatch
2019-12-13 18:35:41.689 7feaac2f5700  0 mon.kvm1a@0(leader) e4 handle_command mon_command({"prefix": "config set", "who": "global", "name": "rbd_default_features", "value": "31"} v 0) v1
2019-12-13 18:35:41.689 7feaac2f5700  0 log_channel(audit) log [INF] : from='client.? 10.248.1.60:0/1930731897' entity='client.admin' cmd=[{"prefix": "config set", "who": "global", "name": "rbd_default_features", "value": "31"}]: dispatch
2019-12-13 18:35:41.705 7feaaaaf2700  0 log_channel(audit) log [INF] : from='client.? 10.248.1.60:0/1930731897' entity='client.admin' cmd='[{"prefix": "config set", "who": "global", "name": "rbd_default_features", "value": "31"}]': finished
2019-12-13 18:35:42.497 7feaac2f5700  0 mon.kvm1a@0(leader) e4 handle_command mon_command({"prefix": "config dump"} v 0) v1
2019-12-13 18:35:42.497 7feaac2f5700  0 log_channel(audit) log [DBG] : from='client.? 10.248.1.60:0/1087170905' entity='client.admin' cmd=[{"prefix": "config dump"}]: dispatch

kvm1b:

Code:

2019-12-13 18:35:26.294 7fc9eff72700  0 mon.kvm1b@1(peon) e4 handle_command mon_command({"prefix": "config rm", "who": "global", "name": "rbd_default_features"} v 0) v1
2019-12-13 18:35:26.294 7fc9eff72700  0 log_channel(audit) log [INF] : from='client.? 10.248.1.60:0/465427509' entity='client.admin' cmd=[{"prefix": "config rm", "who": "global", "name": "rbd_default_features"}]: dispatch
2019-12-13 18:35:40.826 7fc9eff72700  0 mon.kvm1b@1(peon) e4 handle_command mon_command({"prefix": "config dump"} v 0) v1
2019-12-13 18:35:40.826 7fc9eff72700  0 log_channel(audit) log [DBG] : from='client.? 10.248.1.60:0/3230089280' entity='client.admin' cmd=[{"prefix": "config dump"}]: dispatch

kvm1c:

Code:

2019-12-13 18:35:25.668 7f3dea759700  0 mon.kvm1c@2(peon) e4 handle_command mon_command({"prefix": "config dump"} v 0) v1
2019-12-13 18:35:25.668 7f3dea759700  0 log_channel(audit) log [DBG] : from='client.? 10.248.1.60:0/214678714' entity='client.admin' cmd=[{"prefix": "config dump"}]: dispatch
2019-12-13 18:35:33.836 7f3dea759700  0 mon.kvm1c@2(peon) e4 handle_command mon_command({"prefix": "config rm", "who": "global", "name": "rbd_default_features"} v 0) v1
2019-12-13 18:35:33.836 7f3dea759700  0 log_channel(audit) log [INF] : from='client.? 10.248.1.60:0/3381809958' entity='client.admin' cmd=[{"prefix": "config rm", "who": "global", "name": "rbd_default_features"}]: dispatch

Status:

Code:

[admin@kvm1c ~]# ceph -s
  cluster:
    id:     31f6ea46-12cb-47e8-a6f3-60fb6bbd1782
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum kvm1a,kvm1b,kvm1c (age 3d)
    mgr: kvm1c(active, since 3d), standbys: kvm1b, kvm1a
    mds: cephfs:1 {0=kvm1c=up:active} 2 up:standby
    osd: 30 osds: 30 up (since 14h), 30 in (since 3d)

  data:
    pools:   8 pools, 417 pgs
    objects: 1.49M objects, 5.4 TiB
    usage:   15 TiB used, 75 TiB / 90 TiB avail
    pgs:     289 active+clean
             91  active+clean+snaptrim_wait
             37  active+clean+snaptrim

  io:
    client:   2.6 MiB/s rd, 135 MiB/s wr, 43 op/s rd, 164 op/s wr
    cache:    130 MiB/s flush

Alwin · Dec 13, 2019

Code:

ceph config-key rm config/global/rbd_default_features

Try to remove the key like this. At least it worked on my test cluster.

I could also reproduce the issue, with having the same setting with two different values, one in the file & config db, the other only in the config db. In my case, it was two different cluster_network ranges.

Alwin · Dec 13, 2019

David Herselman said:
Hrm... why only 3 ceph 'config rm' references when the commands were run 4 times?

Possibly didn't go to that monitor.

David Herselman · Dec 13, 2019

Many thanks, that works perfectly!

Code:

[admin@kvm1a ~]# ceph config rm global rbd_default_features; ceph config-key rm config/global/rbd_default_features; ceph config dump | grep -e WHO -e rbd_default_features; ceph config set global rbd_default_features 31; ceph config dump | grep -e WHO -e rbd_default_features
key deleted
WHO    MASK LEVEL    OPTION                             VALUE          RO
WHO    MASK LEVEL    OPTION                             VALUE          RO
global      advanced rbd_default_features               31

Search

Search

[SOLVED] Ceph - Unable to remove config entry

David Herselman

Renowned Member

Alwin

Proxmox Retired Staff

David Herselman

Renowned Member

Alwin

Proxmox Retired Staff

David Herselman

Renowned Member

David Herselman

Renowned Member

Alwin

Proxmox Retired Staff

David Herselman

Renowned Member

Alwin

Proxmox Retired Staff

David Herselman

Renowned Member

Alwin

Proxmox Retired Staff

David Herselman

Renowned Member

Alwin

Proxmox Retired Staff

David Herselman

Renowned Member

David Herselman

Renowned Member

Alwin

Proxmox Retired Staff

Alwin

Proxmox Retired Staff

David Herselman

Renowned Member