The wish to get a XZ as a compression option in VM and Container BackUp

[Darkyere]

Hi i am a happy user after upgrading from "Debian 11 Proxmox 7" to "Debian 12 Proxmox 8".
I am trying to write in the Proxmox forum in the hope there will be more finding this feature request possible.

I may have requested this on bugzilla in an inconvenient time when we were close to going from Proxmox 7 to 8.

I have upgraded the feature request from 7 to 8 in bugzilla and will link to it here
The wish to get a XZ as a compression option in VM and Container BackUp

This is a copy of the request from bugzilla so it will be easy to see what i have written.

----

I really enjoy and loves the fact that i took the big step and made a Proxmox server compared to my old Virtualbox W10 PC.

The only thing i would like to request is XZ as a compression option in backup.
The reason being is that sometimes it is not as important with the speed it compresses.
But more the size when done compressing.

The reason i would like this is because i used to use 7Zip to compress the Virtualbox ".vdi".
This meant that 150GB would end up filling my HDD with 30GB's of data.
Though as a note it took me 12 hours to compress because of my large dictionary size.

The best i can get on Proxmox i ZSTD.
Which in theory works great because it only takes 30 minutes for a 150GB VM.
But the problem being is that the file size on my HDD is 80GB after compression.

To me the size after backup is basically more important than the time.
And i would like to believe others would feel the same.

Is it possible to implement XZ as a compression option in Proxmox BackUp?

Now a days it is multi threaded and would still be quite use full.

Preferable something like

xz -9 -T0

Since "-9" this will use the standard best compression and should be good in comparison with ZSTD in the ending file size

and "-T0" will use all the available cores for compression.

Having an option like this doesn't make use of to much memory in the end here is a list from the XZ man page

Preset Compression Decompression

-0 6 MiB 1 MiB

-1 6 MiB 1 MiB

-2 10 MiB 1 MiB

-3 20 MiB 2 MiB

-4 30 MiB 3 MiB

-5 60 MiB 6 MiB

-6 100 MiB 10 MiB

-7 200 MiB 20 MiB

-8 400 MiB 40 MiB

-9 800 MiB 80 MiB

XZ –9 -T0

seems like a standard great tradeoff for File Size + RAM Usage + Time.

I really hope you would implement this compression option in the VM BackUp.


It would be of great help for me, and probably use full for other people concerned about backUp's filling up in size on their disks.

Best Regards,

Darkyere

 

[Darkyere]

I found this in the Proxmox forum.
Seems like a lot of people had already researched what other compression options there is.
And how they compare to each other.

It seemed like XZ or lrzip -L 9 were the best two options compared to the ending file size.
XZ compressed faster and was kinda on a second place because of in ending file size.
lrzip -L 9 compressed like 50 percent slower but had 9% smaller size.

Now given that it is an old thread. A new version of lrzip called lrzip-next has come out.
I cant say anything about lrzip/lrzip-next or how they work/compare to xz.
Or what commands the new lrzip-next would require compared to the old lrzip tested on the forum.

But it is a rather interesting article.
Seems like there are some out there, who is also concerned about ending filesize like me.

Posting the thread for reference, in case anyone would like to read it.
Suggestion: Add lrzip to Backup compression options

Best regards,
Darkyere

EDIT: Typo's

 

[LnxBil]

Seems like there are some out there, who is also concerned about ending filesize like me.

In my experience, the best way so save space is a proper space handling inside of the VM. This is unfortunately not given in any OS I know and has to be done manually (e.g. via cron jobs). Trimming alone is not enough and can be improved further.

Besides, If you're really concerned about backup file size, just go with PBS. With its deduplcation capability and changed-block-differential backup it'll be faster and less space hungry than anything else ... with a lot of headroom.

 

[Darkyere]

I also has to mention that since making this original post I have gotten some cheap SSD's and enabled discard in each individual VM settings, autotrim in ZFS pool and trim inside the VM's.

But that doesn't change the actual size in use by the VM and the files/software in it.
I do not have the hardware nor the money for something that can properly run PBS with deduplication or overflow with space on HDD's to save the backups to.

I do not believe that it would hurt to give the option for one of the more size conserving compressions like xz/lrzip with good compression settings and if necessary call it (good / slowest).
So people with a lot of VM's or large VM's could conserve space if they wished to sacrifice the extra time for it.

It is my firm believe that it never hurt to have more options when having to choose something in the open source world and I really feel like Proxmox is missing more options for compression, explicitly the ones that gives the smallest size with a tradeoff in slower compression time.

 

[LnxBil]

I do not have the hardware nor the money for something that can properly run PBS with deduplication or overflow with space on HDD's to save the backups to.

Why would you need a lot of hardware? Don't you have external backup storage already? That needs hardware too and that is totally capable of running PBS ... and overflowing with space?? Have you seen how much space you need with PBS? If you have on your backups storage two fully compressed backups, PBS will need less and that is not counting what space savings are possible with a third backup or even the redundant stuff for multiple VMs. That is how good PBS is.

So people with a lot of VM's or large VM's could conserve space if they wished to sacrifice the extra time for it.

As someone who has a lot and large VMs, no you don't have time. You need throughput and of course small backups. Having better compression is not the answer as I already shown in my ProxTalks 2017 talk. PBS make it totally obsolete and it is really awesome, so please take a look.