Compression in RBD

[Alexey Petrenko]

Dear Members,
Is it possible to enable compression in rbd?

ceph osd pool set mypool compression_algorithm snappy
ceph osd pool set mypool compression_mode aggressive
don't working.

 

[Alwin]

Is it possible to enable compression in rbd?

yes.

don't working.

What makes you think it isn't working? Compression is only used on new writes.

 

[Alexey Petrenko]

I created a pool "mypool"


Then I changed compression_algorithm and compression_mode and add to storage.

root@proxmox01:/var/lib/ceph/osd/ceph-10# ceph osd pool set mypool compression_algorithm lz4
set pool 12 compression_algorithm to lz4
root@proxmox01:/var/lib/ceph/osd/ceph-10# ceph osd pool set mypool compression_mode force
set pool 12 compression_mode to force
root@proxmox01:/var/lib/ceph/osd/ceph-10# ceph osd pool get mypool all
size: 3
min_size: 2
crash_replay_interval: 0
pg_num: 64
pgp_num: 64
crush_rule: replicated_rule
hashpspool: true
nodelete: false
nopgchange: false
nosizechange: false
write_fadvise_dontneed: false
noscrub: false
nodeep-scrub: false
use_gmt_hitset: 1
auid: 0
fast_read: 0
compression_mode: force
compression_algorithm: lz4



Then I create a virtual machine with RHEL7 in "mypool". After the virtual machine is created, the "used total" is ~ 2GB.


Then I created a 6Gb zero file



And the used space has increased by 6GB.

 

[Alwin]

What is 'ceph df detail' showing? The pool usage shown is before compression.
https://tracker.ceph.com/issues/20870

 

[Alexey Petrenko]

root@proxmox01:/# ceph df detail
GLOBAL:
SIZE AVAIL RAW USED %RAW USED OBJECTS
14307G 14209G 100040M 0.68 20394
POOLS:
NAME ID QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USED
VMs 1 N/A N/A 65193M 0.94 6737G 18309 18309 1118k 764k 127G
mypool 12 N/A N/A 8280M 0.12 4491G 2085 2085 17195 28064 16560M

 

[Alexey Petrenko]

Sorry

 

[Alwin]

It looks correct to me. You have 127GB + 16.5GB = 143.5GB raw used, by the pools but 100GB raw used in the global cluster wide view.

 

[Alexey Petrenko]

I added a second 32 GB disk to a virtual machine and wrote down an empty file of 31 GB in size.

pool raw used in the global cluster wide view increased by 17GB. This is very strange, because when compressed, it should not grow.

 

[Alwin]

32 GB * 3 = 96 GB, depending on the filesystem on the VM, the size that will be written varies. If you run fstrim, you will see the same amount disappearing, because the filesystem on the VM takes already care of 0 writes.

To have a proper test, run fio directly against the cluster.
http://telekomcloud.github.io/ceph/2014/02/26/ceph-performance-analysis_fio_rbd.html

Compression options in luminous
http://docs.ceph.com/docs/luminous/rados/operations/pools/#set-pool-values
http://docs.ceph.com/docs/luminous/rados/configuration/bluestore-config-ref/#inline-compression

 

[Alexey Petrenko]

I deleted old virtual machine and create new virtual machine with single disk. The new virtual machine uses 1.65GiB in the "mypool".



"ceph df detail" show


Then I create 6Gb file zero file in virtual machine


Now the virtual machine uses 7.63GiB in the pool.


In "ceph df detail" global raw used increased by 3GB.


Then I run the fstrim command on virtual machine...

But space was not released



After the file is deleted, the space is reduced.


The "fstrim" command works, but compression does not work.

 

[Alexey Petrenko]

"ceph df detail" after file delete and "fstrim"

 

[alexskysilk]

not that I would ever disparage our intrepid #proxmox devs, but this is probably a question better posed to the ceph devs...