[SOLVED] Backup VM increases...

[a.hoedt]

Hi everyone,


I have a virtual machine and I do a vzdump every day of a VM with LZO compression.


But there are a problem because every day, the size of the backup increases considerably.
Ex : monday -> 1,42 Gb, tuesday -> 2,20 Gb, wednesday -> 2,99 Gb, thursday -> 3,71 Gb, saturday -> 4,49 Gb...


No files was added meanwhile.


Did someone had the same problem and solve it?


Thanks,


Alex.

 

[dietmar]

No files was added meanwhile.

Are you sure about that? Backup size can even grow if you remove files, because the underlying file system writes logs and never frees up any disk space.

 

[a.hoedt]

I'm not sure about that because there are still files that are added but it not explain the grow up of the backup...

I add you a screenshot of my disk space for more precision :


I performed the following commands but the size of the vm increases the next day :

dd if=/dev/zero of=zero.small.file bs=1024 count=102400
dd if=/dev/zero of=zero.file bs=1024
rm zero.small.file
sync ; sleep 60 ; sync
rm zero.file

 

[a.hoedt]

Nobody would have an answer to solve this problem ??

Thanks,


Alex

 

[a.hoedt]

Hi everyone,

My VM backup size is now : 8.72Go...

I need help because the transfert of the file is being so long!

Thanks,


Alex

 

[escoreal]

Hello Alex,

could you copy the disk manually and compress it? To check if this increases too?

I would say that there is running something which generates "disk traffic" but doesn't use much space in total.

Possibilities:
Traffic at /tmp: move it to RAM
Other location: add additional small disk

esco

 

[wahmed]

Swap can also increase virtual disk size.If you havent done so reboot the VM and do a backup immediately to see the size of the backup.
What disk type is it and where is the image stored? Locally? Shared Storage? Is the virtual disk thinly provisioned?

 

[a.hoedt]

Hi,

First of all thank you for the help you have given me.

I restart the VM and then I started backing up immediately, but it is still large (8GB).


The hard drive is (ide0) - local, format = raw and size = 50G


The ISO image is stored locally on the proxmox server.


I looked at the size of the disk over 3 days and it has not changed (see screenshot)





Thanks,


Alex

 

[escoreal]

Hello Alex,

Don't check the free space inside the VM! You didn't checked the size of the disk! You checked the actually used space inside the VM! And not if there is any disk traffic causing this?

Check the disk from outside! Copy it and compress it manually. And check the size of this.

And what "ISO image" do you talking about?

esco

 

[e100]

What you see is expected behaviour for any VM that writes data.
Most likely your log files are part is not all of the data writing that is causing this.

When your disk is brand new it is filled with zeros.
Over time as you write files and delete them, those zeros get turned into random ones and zeros.
A disk full of zeros compresses really well.
A disk full of random data, not so much.

It does not matter how much disk space your are using in the VM, what matters is how many of the physical data blocks on the VMs disk have been changed from zeros into random data.
As time progresses eventually all of the data blocks will have been written to, even if you never use all of the space.
That is what you are observing, data is written (logs) then that data is compressed and written to a new file (gzipped logs created during rotation) and then the log file is deleted (and old gzipped log files too)
Each time that happens more and more virgin blocks are transformed from zeros into random data and your backup compresses less and less.

What you need to do is write zeros to all of the free space inside your vm. Search this forum for directions, they are posted in at least a few threads.
This is something you will need to run periodically if you want to keep the backups small.

 

[mo_]

This ominous "operation" is called trimming (same thing you do with SSDs) by the way
Might as well give the guy a the proper search term to use

 

[a.hoedt]

Hi,

@escoreal:
Is it possible to know the cmd to check disk space? Should I be connected to the physic server or VM?


The server is not hosted at my house, how to make a compression of it locally?


For the ISO image, I wrote about the OS of the VM (debian 7).


@e100:
I searched the forum and the various processes to be done to reduce the size of the VM but it increases management after handling ...


To put zeros correctly I did this:

dd if = / dev / zero of = zero.small.file bs = 1024 count = 102400
dd if = / dev / zero of = zero.file bs = 1024
rm zero.small.file
sync; sleep 60; sync
rm zero.file

Every day I make a backup of / home to my VM with ZIP compression after doing the vzdump VM, then I transfer that backup to a NAS server and then I delete the ZIP file of the VM.


Will he a trash to the data? How empty it?



Thank you for all the help given to this topic!

Alex

 

[escoreal]

Every day I make a backup of / home to my VM with ZIP compression after doing the vzdump VM, then I transfer that backup to a NAS server and then I delete the ZIP file of the VM.

Ok, than this increases your backup.

Instead of deleting you could overwrite it with zeros:
http://linux.die.net/man/1/shred

something like that could do the job:

shred -n 0 -z yourbackupfile.zip

esco

 

[a.hoedt]

Thank you for the help escoreal, I'll try that in the day and coming back to you !

Alex

 

[e100]

You could add a 2nd disk to your VM that is excluded from the backup.
Store the temporary zip of your home dir on that disk.

Won't need to waste time zeroing the zip file, just delete it.

 

[a.hoedt]

Thank you,


The solution to "shred -n yourbackupfile.zip 0 z" has solved my problem, now the size of my backups is ~ 2GB.


I'll try to add a second disk on each VM for backups.


Alex