Backup size is bigger than real used space

[vawaver]

Good evening,

I would like to ask for help.
Ubuntu server 20.04.2 + Nextcloud (via snap) is installed on one of my VMs.

There is used 322 GB of space (I have 2.8 TB disk space for this VM).

I created a backup job on my Datacenter -> whole VM is saved to my external USB drive.

And here is my problem.
The size of the backup is actually cca 462 GB is on my USB drive

So the difference is over 130 GB between the used space on VM and the backup file (vma.zsf).

I would like to know why? Is it possible to fix it?
Thank you.

My version - Proxmox 6.4.8 VE

 

[Dunuin]

Are you using zfs with raidz1/2/3? If yes your volblocksize migut he too low so that you get overhead because of bad padding.

 

[vawaver]

Yes, it is ZFS and RAID1.
I used default values during the installation process.
What should I do?
Thank you for your answer.

 

[Dunuin]

Raid1 (= mirror) or raidz1 (=raid5)?

If it is Raidz1 how many drives are you using with that pool and what ashift value did you choose?

 

[vawaver]

I have created RAID1 -> mirroring the first HDD to the second HDD (both hard drives are the same)
I use 2x 4TB hard drives.

 

[Dunuin]

Then the volblocksize shouldn'T be the problem.

 

[vawaver]

Hmm... a few weeks ago everything was working fine but after some updates on Proxmox, it changed.
Reason unknown.
Any other idea?

 

[leesteken]

Does running fstrim -av reduce the size of the backup? It might only be available when using VirtIO SCSI and virtual SCSI disks.

 

[vawaver]

Does running fstrim -av reduce the size of the backup? It might only be available when using VirtIO SCSI and virtual SCSI disks.

Thank you for your answer.
Unfortunately, I am not an advanced user of Proxmox, a beginner only.
That is why I do not exactly understand your reply.
I attached a screenshot of my VM configuration.

Any idea is welcome.

 

[leesteken]

The backup is based on disk blocks. Deleted files can still take up disk blocks and therefore the backup could be larger than the currently used space. For Linux, use the fstrim command to signal to the underlying storage that those blocks are actually free. Please run fstrim -av (as root or use sudo) in a console inside the VM, and check if the next backup is smaller. (You are using VirtIO SCSI and virtual SCSI disks, so the fstrim command should be supported.)

 

[vawaver]

Thank you for your reply.
Now I understand you perfectly!
Your explanation is easy to understand.
I gonna try your solution and let you know about the result.

 

[Dunuin]

Thank you for your reply.
Now I understand you perfectly!
Your explanation is easy to understand.
I gonna try your solution and let you know about the result.

If that works you should add "discard" to the mounting options in your guests /etc/fstab or better add a daily/weekly "fstrim - a" to your guests /etc/crontab so you don't need to it manually.

 

[vawaver]

Backup ended after 4 hours.... filesize 462 GB
No change.
It did not help.
Another idea?

 

[leesteken]

What did the fstrim command output? What is the output of sudo df -h?

 

[vawaver]

tony@nxtc:~$ sudo df -h
[sudo] password for tony:
Filesystem                         Size  Used Avail Use% Mounted on
udev                               1.9G     0  1.9G   0% /dev
tmpfs                              348M  1.1M  347M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv  2.7T  321G  2.3T  13% /
tmpfs                              1.6G     0  1.6G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
tmpfs                              1.6G     0  1.6G   0% /sys/fs/cgroup
/dev/sda2                          976M  203M  707M  23% /boot
/dev/loop1                          56M   56M     0 100% /snap/core18/2066
/dev/loop0                          56M   56M     0 100% /snap/core18/1997
/dev/loop2                          71M   71M     0 100% /snap/lxd/19647
/dev/loop3                          68M   68M     0 100% /snap/lxd/20326
/dev/loop4                         218M  218M     0 100% /snap/nextcloud/27920
/dev/loop5                         218M  218M     0 100% /snap/nextcloud/28088
/dev/loop6                          33M   33M     0 100% /snap/snapd/11841
/dev/loop7                          33M   33M     0 100% /snap/snapd/12057
tmpfs                              294M     0  294M   0% /run/user/1000

tony@nxtc:~$ sudo fstrim -av
/boot: 769.4 MiB (806809600 bytes) trimmed on /dev/sda2
/: 2.4 TiB (2612910456832 bytes) trimmed on /dev/mapper/ubuntu--vg-ubuntu--lv

 

[leesteken]

You are using LVM inside the VM. It might interfere with discard/trim, or it might contain old data that cannot be trimmed. Would it be possible to reinstall without LVM or add another virtual disk without using LVM and copy all partitions?
EDIT: What are you using as underlying storage? ZFS (which supports discard/trim) or LVM or something else?

 

[vawaver]

I am afraid that I will lose my data because there is my Nextcloud server.
I am thinking about what kind of old data is "hidden" and how can I find it and remove it manually,

There are some old snaps of Nextcloud I suppose as well.

 

[leesteken]

Maybe it is not what I think it is. Maybe it is some LVM thing. Are you using LVM thin provisioning in the VM?
Can you try a backup with LZO to see what size it is? Then we know it is not a ZSTD thing.
Is Discard enabled on the virtual disk (see section 10.2.4 of the manual)? Maybe try enabling SSD emulation?

 

[vawaver]

Thank you for your reply. A lot of new ideas from you.

Are you using LVM thin provisioning in the VM? - I do not know how to check it.
Can you try a backup with LZO to see what size it is? - I will try it. It is a great suggestion.
Is Discard enabled on the virtual disk? - Of course, this was disabled.
Maybe try enabling SSD emulation? -I gonna try all the things above first.

I will let you know.

 

[vawaver]

So I found a solution for me now.
Compression ZSTD (fast and good)
Mode: Stop

Settings for HDD in the virtual machine
Discard - Enable
SSD Emulation - Enable
Backup - Enable

Oťher settings are disabled.

I am back to 300 GB now from 460 GB.

Thank you once again.