ceph rbd slow down read/write

[Alibek]

How much memory does the node have? And does the behavior change when you put the RBD image onto a different pool?

Node have 2TB RAM,
Container have 50G RAM

This behavior is present on differnet pools and not only on Ceph, also this behaviour present on image mapped from node local storage device ssd.

And please update to the latest packages, eg. currently there is Ceph 12.2.12.

As an opportunity, I will definitely do it.

 

[Alwin]

If drop caches - the performance is return up until cache is not filled:

To add, as the CT uses the krbd client it will fill up the page cache first. The drop afterwards will of course make space for the continued write.

 

[Alibek]

To add, as the CT uses the krbd client it will fill up the page cache first. The drop afterwards will of course make space for the continued write.

And which params need to tune? May be next (?):

    "bluestore_cache_kv_max": "536870912",
    "bluestore_cache_kv_ratio": "0.990000",
    "bluestore_cache_meta_ratio": "0.010000",
    "bluestore_cache_size": "0",
    "bluestore_cache_size_hdd": "1073741824",
    "bluestore_cache_size_ssd": "3221225472",
    "bluestore_cache_trim_interval": "0.050000",
    "bluestore_cache_trim_max_skip_pinned": "64",
    "bluestore_cache_type": "2q",

Or:

   "rbd_cache": "true",
    "rbd_cache_block_writes_upfront": "false",
    "rbd_cache_max_dirty": "25165824",
    "rbd_cache_max_dirty_age": "1.000000",
    "rbd_cache_max_dirty_object": "0",
    "rbd_cache_size": "33554432",
    "rbd_cache_target_dirty": "16777216",
    "rbd_cache_writethrough_until_flush": "true",

Or:

    "osd_pool_default_cache_max_evict_check_size": "10",
    "osd_pool_default_cache_min_evict_age": "0",
    "osd_pool_default_cache_min_flush_age": "0",
    "osd_pool_default_cache_target_dirty_high_ratio": "0.600000",
    "osd_pool_default_cache_target_dirty_ratio": "0.400000",
    "osd_pool_default_cache_target_full_ratio": "0.800000",

Or other?

 

[Alibek]

To add, as the CT uses the krbd client it will fill up the page cache first. The drop afterwards will of course make space for the continued write.

And why in second container with rbd (krbd, same pool) at the same time performance is not drop down?
It allows me to think - the problem is not in ceph or in krbd.

 

[Alwin]

And why in second container with rbd (krbd, same pool) at the same time performance is not drop down?
It allows me to think - the problem is not in ceph or in krbd.

I must have overlooked the second container, I thought there is only one. How do those containers differ?

And in general - what do you want to achieve? For now, I only understand that write speed drops as soon as the page cache is filled. But that is rather expected.

 

[Alibek]

I must have overlooked the second container, I thought there is only one. How do those containers differ?

Difference (exclude IP, MAC and identifiers) only one: cached not filled in second container

And in general - what do you want to achieve? For now, I only understand that write speed drops as soon as the page cache is filled. But that is rather expected.

This is not expected.
I want to achieve next: in container page cache, data is sync more often and invalidate synced older pages for allowing overwriting this pages and performance not drop when cached is full filled.

 

[Alibek]

But that is rather expected.

If you look in the post: https://forum.proxmox.com/threads/ceph-rbd-slow-down-write.55055/#post-254003 ,
you can see that only the read operation is performed from the storage, the write operation is performed in /dev/null.
And filling the page cache in the container sufficiently, that would degrade performance.