When adding a new osd to ceph, the osd_mclock_max_capacity_iops_[hdd, ssd] values do not appear in the configuration database

[plastilin]

I have been configuring and testing ceph. When OSDs were first added, the performance values of the added osds were automatically added to the configuration database. However, at some point it stopped working. Should it work? Maybe some update came that turned off this functionality?



Running the test manually passes

root@nd01:~# ceph tell osd.0 cache drop
root@nd01:~# ceph tell osd.0 bench 12288000 4096 4194304 100
{
    "bytes_written": 12288000,
    "blocksize": 4096,
    "elapsed_sec": 2.2298931299999998,
    "bytes_per_sec": 5510577.9889998594,
    "iops": 1345.3559543456688
}

root@nd01:~# ceph tell osd.0 cache drop
root@nd01:~# ceph tell osd.0 bench
{
    "bytes_written": 1073741824,
    "blocksize": 4194304,
    "elapsed_sec": 4.1815870310000003,
    "bytes_per_sec": 256778542.70157841,
    "iops": 61.220775294680216
}

What is the default test when adding osd?

Please also explain how critical is the presence of this data in the configuration database? Is it possible to operate a ceph cluster without these counters?

 

[jsterr]

What software version are you using? Those parameters should not be that important, I used it with and without and with higher values, but it didnt make any difference in performance. Is performance the same on your site? If you want you should be able to set it manually.

What did you change, so the values disappeared?

 

[plastilin]

What software version are you using?

ceph version 17.2.6 (995dec2cdae920da21db2d455e55efbc339bde24) quincy (stable)

What did you change, so the values disappeared?

I tested cluster performance with wal/db hosted on nvme and cleared osd several times. The last time I added them, the counters did not appear. + I rolled back to the configuration version "0" with the ceph config revert command

 

[mcgarrah]

I'm seeing the same behavior with my longer term Proxmox 8.1-1 cluster upgraded along the way to Proxmox 8.2.4. The first three OSD's have the osd_mclock_max_capacity_iops_[hdd/ssd] values when I initially installed the OSDs. All my later OSDs after I upgraded to a later 8.1.x or 8.2.x, do not have values in my Configuration Database as entries...


Note: I have 12 OSDs total in this cluster on four nodes. Only these three are present.

I created a brand new three node Proxmox Dell Wyse 3040 cluster with the latest and greatest Ceph Reef (18.2) and Proxmox 8.2-1. That cluster has values set for both passes of three OSDs with values set for all six OSDs. I'm using it as a testbed to see what impact changes those values has and how to benchmark and set the values appropriately for my main cluster. So I could not reproduce the issue when starting on the latest Proxmox 8.2 and Ceph Reef. It is also a testbed for SDN behaviors but those are not part of this issue.


Note: Added three and another three OSDs with different performance.

So my assumption is someplace in the older 8.1 or 8.2, there was a period were the functionality for setting the max IOPS was missing. Or I have done something incredibly stupid. I'm going to keep working on this a bit to see if I can figure it out.