Number object in bucket full in a shard in ceph

[kiszero]

Dear All,
Currently, I have problem with "fill_status": "OVER 100.000000%" in bucket.
I play command "radosgw-admin bucket limit check" to check limit in bucket.
Result as below:

 {
                "bucket": "bucket-abc",
                "tenant": "",
                "num_objects": 10688897,
                "num_shards": 16,
                "objects_per_shard": 668056,
                "fill_status": "OVER 100.000000%"
            }

Result of command " radosgw-admin reshard list"

 {
        "time": "2019-01-14 09:09:41.032912Z",
        "tenant": "",
        "bucket_name": "bucket-abc",
        "bucket_id": "131d0901-4f62-4f2e-9818-3c27b5326a62.210933.5",
        "new_instance_id": "bucket-abc:131d0901-4f62-4f2e-9818-3c27b5326a62.416328.1",
        "old_num_shards": 8,
        "new_num_shards": 16
    }

and try check status reshard of the above bucket with command "radosgw-admin reshard status --bucket bucket-abc"

{
        "reshard_status": "in-progress",
        "new_bucket_instance_id": "131d0901-4f62-4f2e-9818-3c27b5326a62.416328.1",
        "num_shards": 16
    },
    {
        "reshard_status": "in-progress",
        "new_bucket_instance_id": "131d0901-4f62-4f2e-9818-3c27b5326a62.416328.1",
        "num_shards": 16
    },
    {
        "reshard_status": "in-progress",
        "new_bucket_instance_id": "131d0901-4f62-4f2e-9818-3c27b5326a62.416328.1",
        "num_shards": 16
    },
    {
        "reshard_status": "in-progress",
        "new_bucket_instance_id": "131d0901-4f62-4f2e-9818-3c27b5326a62.416328.1",
        "num_shards": 16
    },
    {
        "reshard_status": "in-progress",
        "new_bucket_instance_id": "131d0901-4f62-4f2e-9818-3c27b5326a62.416328.1",
        "num_shards": 16
    },
    {
        "reshard_status": "in-progress",
        "new_bucket_instance_id": "131d0901-4f62-4f2e-9818-3c27b5326a62.416328.1",
        "num_shards": 16
    },
    {
        "reshard_status": "in-progress",
        "new_bucket_instance_id": "131d0901-4f62-4f2e-9818-3c27b5326a62.416328.1",
        "num_shards": 16
    },
    {
        "reshard_status": "in-progress",
        "new_bucket_instance_id": "131d0901-4f62-4f2e-9818-3c27b5326a62.416328.1",
        "num_shards": 16
    },
    {
        "reshard_status": "in-progress",
        "new_bucket_instance_id": "131d0901-4f62-4f2e-9818-3c27b5326a62.416328.1",
        "num_shards": 16
    },
    {
        "reshard_status": "in-progress",
        "new_bucket_instance_id": "131d0901-4f62-4f2e-9818-3c27b5326a62.416328.1",
        "num_shards": 16
    },
    {
        "reshard_status": "in-progress",
        "new_bucket_instance_id": "131d0901-4f62-4f2e-9818-3c27b5326a62.416328.1",
        "num_shards": 16
    },
    {
        "reshard_status": "in-progress",
        "new_bucket_instance_id": "131d0901-4f62-4f2e-9818-3c27b5326a62.416328.1",
        "num_shards": 16
    },
    {
        "reshard_status": "in-progress",
        "new_bucket_instance_id": "131d0901-4f62-4f2e-9818-3c27b5326a62.416328.1",
        "num_shards": 16
    },
    {
        "reshard_status": "in-progress",
        "new_bucket_instance_id": "131d0901-4f62-4f2e-9818-3c27b5326a62.416328.1",
        "num_shards": 16
    },
    {
        "reshard_status": "in-progress",
        "new_bucket_instance_id": "131d0901-4f62-4f2e-9818-3c27b5326a62.416328.1",
        "num_shards": 16
    },
    {
        "reshard_status": "in-progress",
        "new_bucket_instance_id": "131d0901-4f62-4f2e-9818-3c27b5326a62.416328.1",
        "num_shards": 16
    }

I now that number of object in a shard is over 100%. but resharing process is not completed although I waited for many days. Can someone help me solve this problem ?? Currently I cannot upload any files to this bucket(Error generated when i try upload => "bucket is still resharding, please retry". thanks guys so much!

 

[Alwin]

First off, we don't support the RGW on our stack. Aside from that, do you have enough space for the pool? How is your 'ceph -s' looking?