ceph rbd slow down read/write

How much memory does the node have? And does the behavior change when you put the RBD image onto a different pool?
Node have 2TB RAM,
Container have 50G RAM

This behavior is present on differnet pools and not only on Ceph, also this behaviour present on image mapped from node local storage device ssd.

And please update to the latest packages, eg. currently there is Ceph 12.2.12.
As an opportunity, I will definitely do it.
 
If drop caches - the performance is return up until cache is not filled:
To add, as the CT uses the krbd client it will fill up the page cache first. The drop afterwards will of course make space for the continued write.
 
To add, as the CT uses the krbd client it will fill up the page cache first. The drop afterwards will of course make space for the continued write.
And which params need to tune? May be next (?):
Code:
    "bluestore_cache_kv_max": "536870912",
    "bluestore_cache_kv_ratio": "0.990000",
    "bluestore_cache_meta_ratio": "0.010000",
    "bluestore_cache_size": "0",
    "bluestore_cache_size_hdd": "1073741824",
    "bluestore_cache_size_ssd": "3221225472",
    "bluestore_cache_trim_interval": "0.050000",
    "bluestore_cache_trim_max_skip_pinned": "64",
    "bluestore_cache_type": "2q",
Or:
Code:
   "rbd_cache": "true",
    "rbd_cache_block_writes_upfront": "false",
    "rbd_cache_max_dirty": "25165824",
    "rbd_cache_max_dirty_age": "1.000000",
    "rbd_cache_max_dirty_object": "0",
    "rbd_cache_size": "33554432",
    "rbd_cache_target_dirty": "16777216",
    "rbd_cache_writethrough_until_flush": "true",
Or:
Code:
    "osd_pool_default_cache_max_evict_check_size": "10",
    "osd_pool_default_cache_min_evict_age": "0",
    "osd_pool_default_cache_min_flush_age": "0",
    "osd_pool_default_cache_target_dirty_high_ratio": "0.600000",
    "osd_pool_default_cache_target_dirty_ratio": "0.400000",
    "osd_pool_default_cache_target_full_ratio": "0.800000",
Or other?
 
To add, as the CT uses the krbd client it will fill up the page cache first. The drop afterwards will of course make space for the continued write.
And why in second container with rbd (krbd, same pool) at the same time performance is not drop down?
It allows me to think - the problem is not in ceph or in krbd.
 
And why in second container with rbd (krbd, same pool) at the same time performance is not drop down?
It allows me to think - the problem is not in ceph or in krbd.
I must have overlooked the second container, I thought there is only one. How do those containers differ?

And in general - what do you want to achieve? For now, I only understand that write speed drops as soon as the page cache is filled. But that is rather expected.
 
I must have overlooked the second container, I thought there is only one. How do those containers differ?
Difference (exclude IP, MAC and identifiers) only one: cached not filled in second container

And in general - what do you want to achieve? For now, I only understand that write speed drops as soon as the page cache is filled. But that is rather expected.
This is not expected.
I want to achieve next: in container page cache, data is sync more often and invalidate synced older pages for allowing overwriting this pages and performance not drop when cached is full filled.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!