Ceph - no compression

[Dragon19]

Ceph does not appear to be compressing my files.

root@node01:/mnt/primary# ceph df detail
RAW STORAGE:
    CLASS     SIZE       AVAIL      USED        RAW USED     %RAW USED
    hdd       25 TiB     22 TiB     3.0 TiB      3.0 TiB         11.82
    TOTAL     25 TiB     22 TiB     3.0 TiB      3.0 TiB         11.82

POOLS:
    POOL                           ID     STORED      OBJECTS     USED        %USED     MAX AVAIL     QUOTA OBJECTS     QUOTA BYTES     DIRTY       USED COMPR     UNDER COMPR
    primary_data                    1     1.4 TiB     387.40k     2.8 TiB     12.52       9.9 TiB     N/A               N/A             387.40k        2.9 GiB         5.9 GiB

root@node01:/mnt/primary# ceph osd pool get primary_data compression_algorithm
compression_algorithm: lz4

root@node01:/mnt/primary# ceph osd pool get primary_data compression_mode
compression_mode: aggressive

[global]
         auth_client_required = cephx
         auth_cluster_required = cephx
         auth_service_required = cephx
         bluestore_compression_algorithm = lz4
         bluestore_compression_mode = passive
         bluestore_csum_type = crc32c
         bluestore_compression_required_ratio = 0.875

 

[Alwin]

bluestore compression required ratio

Description: The ratio of the size of the data chunk after compression relative to the original size must be at least this small in order to store the compressed version.
Type: Floating point
Required: No
Default: .875

The compressed data needs to be 12.5% smaller, to be stored. Otherwise the uncompressed version will be used.
https://docs.ceph.com/docs/mimic/rados/configuration/bluestore-config-ref/#inline-compression

POOLS:
    POOL                           ID     STORED      OBJECTS     USED        %USED     MAX AVAIL     QUOTA OBJECTS     QUOTA BYTES     DIRTY       USED COMPR     UNDER COMPR
    primary_data                    1     1.4 TiB     387.40k     2.8 TiB     12.52       9.9 TiB     N/A               N/A             387.40k        2.9 GiB         5.9 GiB

The 'usde compr' shows 2.9 GiB is compressed, totaling to be 5.9 GiB uncompressed. Also data that has already been written before the setting, will not be compressed.

Redhat made also an interesting benchmark, concerning compression.
https://www.redhat.com/en/blog/red-hat-ceph-storage-33-bluestore-compression-performance

 

[Dragon19]

Also data that has already been written before the setting, will not be compressed.

The problem is I've removed most of the primary_data and rewritten it and the compression fell to 400MB total.

I'm not convinced it's working correctly. I'll keep fiddling with the settings and seeing where that goes.

 

[Dragon19]

Set compression_mode to force and I'm almost instantly noticing compressed items increase under ceph df detail.

I guess aggressive isn't enough with ceph. Wish their documentation was a bit more complete in regards to compression.

 

[Alwin]

• aggressive: Compress data unless the write operation has an incompressible hint set.
• force: Try to compress data no matter what.

https://docs.ceph.com/docs/nautilus/rados/configuration/bluestore-config-ref/#inline-compression

Maybe the incompressible hint was set by the client writing the data.