[SOLVED] Ceph - Unable to remove config entry

Jun 8, 2016
244
46
33
43
Johannesburg, South Africa
We assimilated Ceph's ceph.conf file during our Nautilus upgrade and subsequently have a minimal configuration file. We are however now unable to remove configuration entries.

Code:
[admin@kvm1b ~]# ceph config dump | grep -e WHO -e rbd_default_features
WHO    MASK LEVEL    OPTION                             VALUE          RO
global      advanced rbd_default_features               7

[admin@kvm1b ~]# ceph config rm global rbd_default_features
[admin@kvm1b ~]# ceph config rm global rbd_default_features
[admin@kvm1b ~]# ceph config rm global rbd_default_features

[admin@kvm1b ~]# ceph config dump | grep -e WHO -e rbd_default_features
WHO    MASK LEVEL    OPTION                             VALUE          RO
global      advanced rbd_default_features               7

[admin@kvm1b ~]# ceph config set global rbd_default_features 31
[admin@kvm1b ~]# ceph config dump | grep -e WHO -e rbd_default_features
WHO    MASK LEVEL    OPTION                             VALUE          RO
global      advanced rbd_default_features               7
global      advanced rbd_default_features               31

Any ideas how I fix this?
 

Alwin

Proxmox Staff Member
Staff member
Aug 1, 2017
3,633
329
88
Can you please share your ceph.conf? And is there anything in the ceph logs? As I it should override the config setting, not add another one.
 
Jun 8, 2016
244
46
33
43
Johannesburg, South Africa
Ceph config file:

Code:
[global]
 cluster_network = 10.248.1.0/24
 filestore_xattr_use_omap = true
 fsid = 31f6ea46-12cb-47e8-a6f3-60fb6bbd1782
 mon_host = 10.248.1.60 10.248.1.61 10.248.1.62
 public_network = 10.248.1.0/24

[client]
 keyring = /etc/pve/priv/$cluster.$name.keyring
 

Alwin

Proxmox Staff Member
Staff member
Aug 1, 2017
3,633
329
88
Is the cluster healthy? And on which ceph versions are you?

EDIT: And is there anything in the ceph logs?
 
Jun 8, 2016
244
46
33
43
Johannesburg, South Africa
Ceph Nautilus 14.2.4.1,


Herewith our Ceph Nautilus upgrade nodes that we ran through on 7 separate clusters:
Code:
cd /etc/pve;
ceph config assimilate-conf -i ceph.conf -o ceph.conf.new;
mv ceph.conf.new ceph.conf;
pico /etc/ceph/ceph.conf
  # add back: cluster_network
  #           public_network
ceph config rm global cluster_network;
ceph config rm global public_network;
ceph config set global mon_osd_down_out_subtree_limit host;

Ceph configuration dump:
Code:
[admin@kvm1c ~]# ceph config dump
WHO    MASK LEVEL    OPTION                             VALUE          RO
global      advanced auth_client_required               cephx          *
global      advanced auth_cluster_required              cephx          *
global      advanced auth_service_required              cephx          *
global      advanced cluster_network                    10.248.1.0/24  *
global      advanced debug_filestore                    0/0
global      advanced debug_journal                      0/0
global      advanced debug_ms                           0/0
global      advanced debug_osd                          0/0
global      basic    device_failure_prediction_mode     cloud
global      advanced mon_allow_pool_delete              true
global      advanced mon_osd_down_out_subtree_limit     host
global      advanced osd_deep_scrub_interval            1209600.000000
global      advanced osd_pool_default_min_size          2
global      advanced osd_pool_default_size              3
global      advanced osd_scrub_begin_hour               19
global      advanced osd_scrub_end_hour                 6
global      advanced osd_scrub_sleep                    0.100000
global      advanced public_network                     10.248.1.0/24  *
global      advanced rbd_default_features               7
global      advanced rbd_default_features               31
  mgr       advanced mgr/balancer/active                true
  mgr       advanced mgr/balancer/mode                  upmap
  mgr       advanced mgr/devicehealth/enable_monitoring true
Code:
proxmox-ve: 6.1-2 (running kernel: 5.3.10-1-pve)
pve-manager: 6.1-3 (running version: 6.1-3/37248ce6)
pve-kernel-5.3: 6.0-12
pve-kernel-helper: 6.0-12
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph: 14.2.4.1-pve1
ceph-fuse: 14.2.4.1-pve1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-9
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-2
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
openvswitch-switch: 2.10.0+2018.08.28+git.8ca7c82b7d+ds1-12+deb10u1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-1
pve-cluster: 6.1-2
pve-container: 3.0-14
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191002-1
pve-firewall: 4.0-9
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 3.13.2-1
qemu-server: 6.1-2
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2
 
Jun 8, 2016
244
46
33
43
Johannesburg, South Africa
Apologies, yes the cluster is healthy:
Code:
[admin@kvm1c ~]# ceph -s
  cluster:
    id:     31f6ea46-12cb-47e8-a6f3-60fb6bbd1782
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum kvm1a,kvm1b,kvm1c (age 20h)
    mgr: kvm1c(active, since 20h), standbys: kvm1b, kvm1a
    mds: cephfs:1 {0=kvm1c=up:active} 2 up:standby
    osd: 30 osds: 30 up (since 20h), 30 in (since 20h)

  data:
    pools:   8 pools, 417 pgs
    objects: 1.46M objects, 5.3 TiB
    usage:   15 TiB used, 75 TiB / 90 TiB avail
    pgs:     417 active+clean

  io:
    client:   45 KiB/s rd, 4.5 MiB/s wr, 6 op/s rd, 354 op/s wr
    cache:    0 op/s promote
 

Alwin

Proxmox Staff Member
Staff member
Aug 1, 2017
3,633
329
88
When you try to delete the config setting is there anything in the ceph logs, that indicates why there might be two entries?

Another possibility could be that the setting is locked by a service and therefore not deleted. But I guess, that would be visible in the logs.
 

Alwin

Proxmox Staff Member
Staff member
Aug 1, 2017
3,633
329
88

Alwin

Proxmox Staff Member
Staff member
Aug 1, 2017
3,633
329
88
Thanks. That might give it more attention.

Code:
ceph daemon mon.<id> config set debug_rbd 20/20
You can see the config set/rm with the higher debug level (/var/log/ceph/ceph-mon.<id>.log). This may hopefully shine some light on the situation.

EDIT: 0/5 is the default log value for rbd.
 
Jun 8, 2016
244
46
33
43
Johannesburg, South Africa
Setting this to 20/20 or 5/5 only generates the following debug information:

Enable debugging:
Code:
[admin@kvm1a ~]# ceph daemon mon.kvm1a config show | grep -i debug_rbd
  #     "debug_rbd": "0/5",
  #     "debug_rbd_mirror": "0/5",
  #     "debug_rbd_replay": "0/5",
[admin@kvm1a ~]# ceph daemon mon.kvm1a config set debug_rbd 20/20
[admin@kvm1a ~]# ceph daemon mon.kvm1a config show | grep -i debug_rbd
  #     "debug_rbd": "20/20",
  #     "debug_rbd_mirror": "0/5",
  #     "debug_rbd_replay": "0/5",
Commands:
Code:
[admin@kvm1a ~]# ceph config dump | grep -e WHO -e rbd_default_features
WHO    MASK LEVEL    OPTION                             VALUE          RO
global      advanced rbd_default_features               7
global      advanced rbd_default_features               31
[admin@kvm1a ~]# ceph config rm global rbd_default_features
[admin@kvm1a ~]# ceph config rm global rbd_default_features
[admin@kvm1a ~]# ceph config rm global rbd_default_features
[admin@kvm1a ~]# ceph config rm global rbd_default_features
[admin@kvm1a ~]# ceph config dump | grep -e WHO -e rbd_default_features
WHO    MASK LEVEL    OPTION                             VALUE          RO
global      advanced rbd_default_features               7
[admin@kvm1a ~]# ceph config set global rbd_default_features 31
[admin@kvm1a ~]# ceph config dump | grep -e WHO -e rbd_default_features
WHO    MASK LEVEL    OPTION                             VALUE          RO
global      advanced rbd_default_features               7
global      advanced rbd_default_features               31
Log entries (tail -f /var/log/ceph/ceph-mon.kvm1a.log):
Code:
2019-12-13 18:15:29.386 7fc9eff72700  0 mon.kvm1a@1(peon) e4 handle_command mon_command({"prefix": "config dump"} v 0) v1
2019-12-13 18:15:29.386 7fc9eff72700  0 log_channel(audit) log [DBG] : from='client.67360717 10.248.1.61:0/1627956866' entity='client.admin' cmd=[{"prefix": "config dump"}]: dispatch
2019-12-13 18:15:35.898 7fc9eff72700  0 mon.kvm1a@1(peon) e4 handle_command mon_command({"prefix": "config rm", "who": "global", "name": "rbd_default_features"} v 0) v1
2019-12-13 18:15:35.898 7fc9eff72700  0 log_channel(audit) log [INF] : from='client.67360783 10.248.1.61:0/472300355' entity='client.admin' cmd=[{"prefix": "config rm", "who": "global", "name": "rbd_default_features"}]: dispatch
2019-12-13 18:15:36.422 7fc9eff72700  0 mon.kvm1a@1(peon) e4 handle_command mon_command({"prefix": "config rm", "who": "global", "name": "rbd_default_features"} v 0) v1
2019-12-13 18:15:36.422 7fc9eff72700  0 log_channel(audit) log [INF] : from='client.? 10.248.1.61:0/3618748536' entity='client.admin' cmd=[{"prefix": "config rm", "who": "global", "name": "rbd_default_features"}]: dispatch
2019-12-13 18:15:37.258 7fc9eff72700  0 mon.kvm1a@1(peon) e4 handle_command mon_command({"prefix": "config rm", "who": "global", "name": "rbd_default_features"} v 0) v1
2019-12-13 18:15:37.258 7fc9eff72700  0 log_channel(audit) log [INF] : from='client.? 10.248.1.61:0/605608008' entity='client.admin' cmd=[{"prefix": "config rm", "who": "global", "name": "rbd_default_features"}]: dispatch
2019-12-13 18:15:39.710 7fc9eff72700  0 mon.kvm1a@1(peon) e4 handle_command mon_command({"prefix": "config dump"} v 0) v1
2019-12-13 18:15:39.710 7fc9eff72700  0 log_channel(audit) log [DBG] : from='client.? 10.248.1.61:0/2007128611' entity='client.admin' cmd=[{"prefix": "config dump"}]: dispatch
2019-12-13 18:15:44.682 7fc9eff72700  0 mon.kvm1a@1(peon) e4 handle_command mon_command({"prefix": "config set", "who": "global", "name": "rbd_default_features", "value": "31"} v 0) v1
2019-12-13 18:15:44.682 7fc9eff72700  0 log_channel(audit) log [INF] : from='client.67360882 10.248.1.61:0/3247534721' entity='client.admin' cmd=[{"prefix": "config set", "who": "global", "name": "rbd_default_features", "value": "31"}]: dispatch
2019-12-13 18:15:47.394 7fc9eff72700  0 mon.kvm1a@1(peon) e4 handle_command mon_command({"prefix": "config dump"} v 0) v1
2019-12-13 18:15:47.394 7fc9eff72700  0 log_channel(audit) log [DBG] : from='client.67360924 10.248.1.61:0/2411181763' entity='client.admin' cmd=[{"prefix": "config dump"}]: dispatch
Hrm... why only 3 ceph 'config rm' references when the commands were run 4 times?
 
Jun 8, 2016
244
46
33
43
Johannesburg, South Africa
Commands sometimes land on other monitors, herewith simultaneous debugs from all three monitors.

Commands:
Code:
[root@kvm1a ~]# ceph config dump | grep -e WHO -e rbd_default_features
WHO    MASK LEVEL    OPTION                             VALUE          RO
global      advanced rbd_default_features               7
global      advanced rbd_default_features               31
[root@kvm1a ~]# ceph config rm global rbd_default_features
[root@kvm1a ~]# ceph config rm global rbd_default_features
[root@kvm1a ~]# ceph config rm global rbd_default_features
[root@kvm1a ~]# ceph config rm global rbd_default_features
[root@kvm1a ~]# ceph config dump | grep -e WHO -e rbd_default_features
WHO    MASK LEVEL    OPTION                             VALUE          RO
global      advanced rbd_default_features               7
[root@kvm1a ~]# ceph config set global rbd_default_features 31
[root@kvm1a ~]# ceph config dump | grep -e WHO -e rbd_default_features
WHO    MASK LEVEL    OPTION                             VALUE          RO
global      advanced rbd_default_features               7
global      advanced rbd_default_features               31
Logs:
kvm1a:
Code:
2019-12-13 18:35:26.293 7feaac2f5700  0 mon.kvm1a@0(leader) e4 handle_command mon_command({"prefix": "config rm", "who": "global", "name": "rbd_default_features"} v 0) v1
2019-12-13 18:35:26.293 7feaac2f5700  0 log_channel(audit) log [INF] : from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "global", "name": "rbd_default_features"}]: dispatch
2019-12-13 18:35:26.305 7feaaaaf2700  0 log_channel(audit) log [INF] : from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "global", "name": "rbd_default_features"}]': finished
2019-12-13 18:35:30.401 7feaac2f5700  0 mon.kvm1a@0(leader) e4 handle_command mon_command({"prefix": "config rm", "who": "global", "name": "rbd_default_features"} v 0) v1
2019-12-13 18:35:30.401 7feaac2f5700  0 log_channel(audit) log [INF] : from='client.? 10.248.1.60:0/2894657506' entity='client.admin' cmd=[{"prefix": "config rm", "who": "global", "name": "rbd_default_features"}]: dispatch
2019-12-13 18:35:32.289 7feaac2f5700  0 mon.kvm1a@0(leader) e4 handle_command mon_command({"prefix": "config rm", "who": "global", "name": "rbd_default_features"} v 0) v1
2019-12-13 18:35:32.289 7feaac2f5700  0 log_channel(audit) log [INF] : from='client.? 10.248.1.60:0/3241273675' entity='client.admin' cmd=[{"prefix": "config rm", "who": "global", "name": "rbd_default_features"}]: dispatch
2019-12-13 18:35:33.833 7feaac2f5700  0 mon.kvm1a@0(leader) e4 handle_command mon_command({"prefix": "config rm", "who": "global", "name": "rbd_default_features"} v 0) v1
2019-12-13 18:35:33.833 7feaac2f5700  0 log_channel(audit) log [INF] : from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "global", "name": "rbd_default_features"}]: dispatch
2019-12-13 18:35:41.689 7feaac2f5700  0 mon.kvm1a@0(leader) e4 handle_command mon_command({"prefix": "config set", "who": "global", "name": "rbd_default_features", "value": "31"} v 0) v1
2019-12-13 18:35:41.689 7feaac2f5700  0 log_channel(audit) log [INF] : from='client.? 10.248.1.60:0/1930731897' entity='client.admin' cmd=[{"prefix": "config set", "who": "global", "name": "rbd_default_features", "value": "31"}]: dispatch
2019-12-13 18:35:41.705 7feaaaaf2700  0 log_channel(audit) log [INF] : from='client.? 10.248.1.60:0/1930731897' entity='client.admin' cmd='[{"prefix": "config set", "who": "global", "name": "rbd_default_features", "value": "31"}]': finished
2019-12-13 18:35:42.497 7feaac2f5700  0 mon.kvm1a@0(leader) e4 handle_command mon_command({"prefix": "config dump"} v 0) v1
2019-12-13 18:35:42.497 7feaac2f5700  0 log_channel(audit) log [DBG] : from='client.? 10.248.1.60:0/1087170905' entity='client.admin' cmd=[{"prefix": "config dump"}]: dispatch
kvm1b:
Code:
2019-12-13 18:35:26.294 7fc9eff72700  0 mon.kvm1b@1(peon) e4 handle_command mon_command({"prefix": "config rm", "who": "global", "name": "rbd_default_features"} v 0) v1
2019-12-13 18:35:26.294 7fc9eff72700  0 log_channel(audit) log [INF] : from='client.? 10.248.1.60:0/465427509' entity='client.admin' cmd=[{"prefix": "config rm", "who": "global", "name": "rbd_default_features"}]: dispatch
2019-12-13 18:35:40.826 7fc9eff72700  0 mon.kvm1b@1(peon) e4 handle_command mon_command({"prefix": "config dump"} v 0) v1
2019-12-13 18:35:40.826 7fc9eff72700  0 log_channel(audit) log [DBG] : from='client.? 10.248.1.60:0/3230089280' entity='client.admin' cmd=[{"prefix": "config dump"}]: dispatch
kvm1c:
Code:
2019-12-13 18:35:25.668 7f3dea759700  0 mon.kvm1c@2(peon) e4 handle_command mon_command({"prefix": "config dump"} v 0) v1
2019-12-13 18:35:25.668 7f3dea759700  0 log_channel(audit) log [DBG] : from='client.? 10.248.1.60:0/214678714' entity='client.admin' cmd=[{"prefix": "config dump"}]: dispatch
2019-12-13 18:35:33.836 7f3dea759700  0 mon.kvm1c@2(peon) e4 handle_command mon_command({"prefix": "config rm", "who": "global", "name": "rbd_default_features"} v 0) v1
2019-12-13 18:35:33.836 7f3dea759700  0 log_channel(audit) log [INF] : from='client.? 10.248.1.60:0/3381809958' entity='client.admin' cmd=[{"prefix": "config rm", "who": "global", "name": "rbd_default_features"}]: dispatch
Status:
Code:
[admin@kvm1c ~]# ceph -s
  cluster:
    id:     31f6ea46-12cb-47e8-a6f3-60fb6bbd1782
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum kvm1a,kvm1b,kvm1c (age 3d)
    mgr: kvm1c(active, since 3d), standbys: kvm1b, kvm1a
    mds: cephfs:1 {0=kvm1c=up:active} 2 up:standby
    osd: 30 osds: 30 up (since 14h), 30 in (since 3d)

  data:
    pools:   8 pools, 417 pgs
    objects: 1.49M objects, 5.4 TiB
    usage:   15 TiB used, 75 TiB / 90 TiB avail
    pgs:     289 active+clean
             91  active+clean+snaptrim_wait
             37  active+clean+snaptrim

  io:
    client:   2.6 MiB/s rd, 135 MiB/s wr, 43 op/s rd, 164 op/s wr
    cache:    130 MiB/s flush
 

Alwin

Proxmox Staff Member
Staff member
Aug 1, 2017
3,633
329
88
Code:
ceph config-key rm config/global/rbd_default_features
Try to remove the key like this. At least it worked on my test cluster.

I could also reproduce the issue, with having the same setting with two different values, one in the file & config db, the other only in the config db. In my case, it was two different cluster_network ranges.
 
Jun 8, 2016
244
46
33
43
Johannesburg, South Africa
Many thanks, that works perfectly!

Code:
[admin@kvm1a ~]# ceph config rm global rbd_default_features; ceph config-key rm config/global/rbd_default_features; ceph config dump | grep -e WHO -e rbd_default_features; ceph config set global rbd_default_features 31; ceph config dump | grep -e WHO -e rbd_default_features
key deleted
WHO    MASK LEVEL    OPTION                             VALUE          RO
WHO    MASK LEVEL    OPTION                             VALUE          RO
global      advanced rbd_default_features               31
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!