Backup Size

[Editor]

Hello,

My VM total size is 400 GB and I only use 8-9 GB but the backup size is 63 GB.

A few days ago, while only a maximum of 5 GB, right now 63GB and I don't understand this.. I have read all previous writes but could not reach the solution

This is Proxmox:

root@prox:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             10M     0   10M   0% /dev
tmpfs           6.3G  9.0M  6.3G   1% /run
/dev/dm-0        95G  5.5G   85G   7% /
tmpfs            16G   46M   16G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/sda1       917G  347G  523G  40% /H2_SiyahBig
/dev/sdb1       917G  129G  742G  15% /H1_Gri
/dev/fuse        30M   20K   30M   1% /etc/pve

This is VM (Centos 7):

[root@server5 ~]# du -sch /*
0       /backup
0       /bin
151M    /boot
0       /dev
35M     /etc
1.6G    /home
0       /lib
0       /lib64
0       /media
0       /mnt
543M    /opt
du: cannot access ‘/proc/22579/task/22579/fd/4’: No such file or directory
du: cannot access ‘/proc/22579/task/22579/fdinfo/4’: No such file or directory
du: cannot access ‘/proc/22579/fd/4’: No such file or directory
du: cannot access ‘/proc/22579/fdinfo/4’: No such file or directory
0       /proc
59M     /root
17M     /run
0       /sbin
0       /scripts
0       /srv
0       /sys
766M    /tmp
3.5G    /usr
911M    /var
7.5G    total

Also I checked "zero" file and size: 0 bayt in FTPzilla

root@prox:~# dd if=/dev/zero of=zero.small.file bs=1024 count=102400
102400+0 records in
102400+0 records out
104857600 bytes (105 MB) copied, 0.318198 s, 330 MB/s
root@prox:~# rm zero.small.file
root@prox:~#

Can you help me?
Thank you

 

[mailinglists]

just a shoot in the dark,
... try fstrim -v / in the guest vm, then reboot, if you are using qcow2 or thin lvm.

 

[fireon]

If the VM grows up and you delete after a time files, the image will not be shrinked. This works but only with "scsi disks" with activated "discard". This does not work with virtiodiskimages. On centos the trimfunction is default disabled. You must enable and start it:

systemctl enable fstrim.timer
systemctl start fstrim.timer
systemctl status fstrim.timer

From the Proxmox Documantation:
If your storage supports thin provisioning (see the storage chapter in the Proxmox VE guide), and your VM has a SCSI controller you can activate the Discard option on the hard disks connected to that controller. With Discard enabled, when the filesystem of a VM marks blocks as unused after removing files, the emulated SCSI controller will relay this information to the storage, which will then shrink the disk image accordingly.

 

[Editor]

If the VM grows up and you delete after a time files, the image will not be shrinked. This works but only with "scsi disks" with activated "discard". This does not work with virtiodiskimages. On centos the trimfunction is default disabled. You must enable and start it:

systemctl enable fstrim.timer
systemctl start fstrim.timer
systemctl status fstrim.timer

From the Proxmox Documantation:
If your storage supports thin provisioning (see the storage chapter in the Proxmox VE guide), and your VM has a SCSI controller you can activate the Discard option on the hard disks connected to that controller. With Discard enabled, when the filesystem of a VM marks blocks as unused after removing files, the emulated SCSI controller will relay this information to the storage, which will then shrink the disk image accordingly.

Thank you for your time
I did this commands for Centos 7 but still backup size: 67GB~
Do you know another alternative way?

 

[Editor]

just a shoot in the dark,
... try fstrim -v / in the guest vm, then reboot, if you are using qcow2 or thin lvm.

[root@server5 ~]# fstrim -v /
/: 44.2 GiB (47422259200 bytes) trimmed

what is this and what should I do?
-
Also PS. I dont use SSD.. I use SAS hard-disk

 

[wbumiller]

Also PS. I dont use SSD.. I use SAS hard-disk

Particularly with virtual environments trimming/discarding isn't limited to SSDs. With virtual machines the host system doesn't know (or care) much about the contents of a VM's disk, which means if you delete files within the guest system, this is not visible to the host. The guest's file system driver will write metadata to the disk marking the regions of the disk which used to contain the file as unused. The only thing the host sees in this case, however, is the fact that there's something being written to the disk. But if the guest performs a trim/discard operation the host is made aware of the unused region and can deallocate it, ideally passing this on to the underlying layers all the way down to the disk itself if it is an SSD. And if it's not an SSD at least it knows what doesn't need backing up anymore.

 

[Editor]

Particularly with virtual environments trimming/discarding isn't limited to SSDs. With virtual machines the host system doesn't know (or care) much about the contents of a VM's disk, which means if you delete files within the guest system, this is not visible to the host. The guest's file system driver will write metadata to the disk marking the regions of the disk which used to contain the file as unused. The only thing the host sees in this case, however, is the fact that there's something being written to the disk. But if the guest performs a trim/discard operation the host is made aware of the unused region and can deallocate it, ideally passing this on to the underlying layers all the way down to the disk itself if it is an SSD. And if it's not an SSD at least it knows what doesn't need backing up anymore.

So what do I do? I found many documents on the internet but I can not be sure. What do you suggest me to do?

Proxmox:

root@prox:~# fstrim -v /
fstrim: /: the discard operation is not supported

VM Guest CentOS 7:

[root@server5 ~]# fstrim -v /
/: 44.2 GiB (47486414848 bytes) trimmed

 

[wbumiller]

For this to take effect the VM needs to be using a disk type which supports discards (scsi) and the discard option must be activated. Can you post the output of `qm config $VMID`?

 

[Editor]

Do I need to buy a new SSD disk?

root@prox:~# vmid=101
root@prox:~# ps -ef | grep "kvm -id $vmid" | sed -r 's/ -/\n-/g' | grep scsi
-iscsi initiator-name=iqn.1993-08.org.debian:01:1017e46abca1
-device virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5
-drive file=/dev/pve/vm-101-disk-1,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on
-device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100

root@prox:~# qm config 101
agent: 1
bootdisk: scsi0
cores: 2
ide2: none,media=cdrom
memory: 15000
name: CWP
net0: virtio=AA:7A:E9:FA:58:EF,bridge=vmbr0
numa: 0
onboot: 1
ostype: l26
scsi0: local-lvm:vm-101-disk-1,size=450G
scsihw: virtio-scsi-pci
smbios1: uuid=2d818eda-f1c5-4610-b0c1-dfadfc4ebbf0
sockets: 2

 

[wbumiller]

Do I need to buy a new SSD disk?

No, that's not the problem.

You do not have the discard option set on your VM's disk. You can do this in the web GUI in the VM's Hardware tab. (You'll need to stop and start the VM afterwards).

 

[Editor]

I have recreated the backup but the size of backup did not change again. (still 65GB). I guess there isn't much I can do

INFO: starting new backup job: vzdump 101 --remove 0 --storage H1-Gri --mode snapshot --compress lzo --node prox
INFO: Starting Backup of VM 101 (qemu)
INFO: status = running
INFO: update VM 101: -lock backup
INFO: VM Name: cwp
INFO: include disk 'scsi0' 'local-lvm:vm-101-disk-1' 450G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/H1_Gri/dump/vzdump-qemu-101-2017_04_27-13_03_49.vma.lzo'
INFO: started backup task 'bf52fc26-c237-4faa-8ff6-b7ec6420e755'
INFO: status: 0% (1053032448/483183820800), sparse 0% (887885824), duration 3, 351/55 MB/s
INFO: status: 1% (5141168128/483183820800), sparse 1% (4976005120), duration 10, 584/0 MB/s
INFO: status: 2% (9717219328/483183820800), sparse 1% (9367085056), duration 24, 326/13 MB/s
...
INFO: status: 99% (479110955008/483183820800), sparse 81% (391772381184), duration 3073, 538/0 MB/s
INFO: status: 100% (483183820800/483183820800), sparse 81% (395845246976), duration 3079, 678/0 MB/s
INFO: transferred 483183 MB in 3079 seconds (156 MB/s)
INFO: archive file size: 65.94GB
INFO: Finished Backup of VM 101 (00:51:20)
INFO: Backup job finished successfully
TASK OK

 

[fireon]

Please post your VMconfig

 

[Editor]

Please post your VMconfig

root@prox:~# qm config 101
agent: 1
bootdisk: scsi0
cores: 2
ide2: none,media=cdrom
memory: 15000
name: CWP
net0: virtio=AA:7A:E9:FA:58:EF,bridge=vmbr0
numa: 0
onboot: 1
ostype: l26
scsi0: local-lvm:vm-101-disk-1,size=450G
scsihw: virtio-scsi-pci
smbios1: uuid=2d818eda-f1c5-4610-b0c1-dfadfc4ebbf0
sockets: 2

 

[fireon]

Thanks but, can't find on your config that the discardflag is set. It should look like this:

..
scsi0: HDD-vmdata-KVM:vm-103-disk-1,discard=on,size=32G 
scsi1: HDD-vmdata-KVM:vm-103-disk-2,discard=on,size=4G
..

 

[Editor]

Thanks but, can't find on your config that the discardflag is set. It should look like this:

..
scsi0: HDD-vmdata-KVM:vm-103-disk-1,discard=on,size=32G
scsi1: HDD-vmdata-KVM:vm-103-disk-2,discard=on,size=4G
..

Yes I did and restarted the machine,
backup size is still the same (65GB)

scsi0: local-lvm:vm-101-disk-1,discard=on,size=450G

 

[fireon]

Ok, when when you do this trimcommand now? Did this help?
"fstrim -v /" or "fstrim --all || true "

 

[Editor]

"fstrim --all || true"
What is this? - Still backup size: 69GB

[root@server5 ~]# fstrim -v /
/: 42.7 GiB (45835853824 bytes) trimmed
[root@server5 ~]# fstrim --all || true
[root@server5 ~]#

 

[gosha]

[root@server5 ~]# fstrim -v /
/: 42.7 GiB (45835853824 bytes) trimmed

This is free space on your disk in fact.

 

[Editor]

This is free space on your disk in fact.

I did and now backup size: 5GB thank you but do you recommend "discarding"?
Do I have problems if I want to use backup files in the future?

 

[fireon]

Discarding is an Default!