compress-force=zstd:3 as standard for BTRFS?

[A.R.]

Hi,

I have chosen to use BTRFS since I have been using that since it was included with (Open)SUSE so also chose it for Proxmox.

I noticed very low compression on my disk images. Just a few percent. You can easily investigate this by using compsize (package btrfs-compsize).
The default mount option set by Proxmox for a btrfs partition is compress=zstd:3. This option will test if a file can be compressed at creation and when the 1st blocks are written. If it can be compressed, the whole file can be compressed. If not none(!) of the remainder of the raw image will be compressed.

As it now appears the disk images for my various guests (FreeBSD, various Linuxes and Windows) are not so easily compressible in the beginning, but later can be compressed very well.

I have made this changes to my fstab and ran btrfs filesystem defragment -r -czstd /var/lib/pve/local-btrfs/images/ and other (image) storages with very good compression results, saving about 40% disk space on average. Plus, since it has been proven that compressed btrfs disks usually yield higher io throughput vm's get some better IO.

So I was wondering if it would perhaps be better to have compress-force=zstd:3 as the standard mount option for BTRFS volumes to reap the benefits of compression with BTRFS?

 

[RolandK]

that seems to make sense, i don't see the point why btrfs should consider a 100gb virtual disk file to be stored uncompressed because it's first blocks are badly compressible. if there is compression active, people will excpect that it's stored compressed.

would you like to file a bugzilla ticket for this ?

also see, as this may be of interest for you. there is some more details / untold truth on btrfs compression heuristics

https://github.com/kilobyte/compsize/issues/24

 

[A.R.]

Sure will do. I had it on hold since I have an enormous slowness in backing up my mv's. I maybe think why. Opening another threat for it.

 

[alexskysilk]

So I was wondering if it would perhaps be better to have compress-force=zstd:3 as the standard mount option for BTRFS volumes to reap the benefits of compression with BTRFS?

This can have some pretty unpleasant write performance implications. better to let the compression algorithm decide.

 

[RolandK]

>This can have some pretty unpleasant write performance implications.

in which way? lzo is blazingly fast and zstd is also fast on modern cpu.

>better to let the compression algorithm decide

but the compression heuristics in btrfs sucks, especially when used in conjunction with large virtual disk files which are used as "block based storage". why should compression on this only be decided by the first data block of such file ? (which is what standard btrfs compression algorithm does)

https://marc.info/?l=linux-btrfs&m=170853747617447&w=2
https://bugzilla.proxmox.com/show_bug.cgi?id=5250#c6

 

[alexskysilk]

because uncompressible data slows down writes dramatically when forced through a compression algo; it also ends up LARGER then original. even if the hinting algorithm is insufficient, its better on balance to leave a suspected file uncompressed then the other way around- its also why its the default behavior.

 

[A.R.]

because uncompressible data slows down writes dramatically when forced through a compression algo; it also ends up LARGER then original. even if the hinting algorithm is insufficient, its better on balance to leave a suspected file uncompressed then the other way around- its also why its the default behavior.

BTRFS will indeed try to compress all data. And thus there is some penalty in writing data. However, if a block (128KB) cannot be compressed, it will be stored uncompressed. So it does not get larger. There is also an extra penalty because of the block size of 128KB. A small write will result in a 128KB write.

However, at least on my system, I noticed an unexpected slow down in reading. To my surprise since it should be faster in theory.
running pv zstd3_compressed_disk.raw > /dev/null on a system with no vm's running yielded a 268MiB/s average speed on a 300 GiB file.
An uncompressed copy did that with 1,50GiB/s. With no difference between raid1 and raid0. So I better turn the latter in a jbod config.

So for me there is another issue at play with zstd:3 at least. More research to do. Is this cpu, memory, later kernels, different tweaks? I don't know yet. Hardware wise I do not think dual Intel Xeon E5-2630 v4 (20) @ 3.100GHz could be the issue.