[SOLVED] Strange Backup Behavior, Total Bytes Written 100 X actual data size, backup slow due to this...

[effgee]

I have an LXC container that takes an extremely long time to backup, and mostly because the backup process thinks it is backing up data that isnt there!
This container was converted to LXC from a VM image

NCDU tells me the entirety of the disk contents is about 22 gigabytes.
du -hs says 23G

The backup process is reading/writing 300+ gigabytes, but the final archive is 3gigs. Anyone know what is going on?

INFO: Starting Backup of VM 6970 (lxc)
INFO: Backup started at 2020-06-19 06:55:25
INFO: status = running
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: NMS
INFO: stopping vm
INFO: creating archive '/mypool_8tb_backup/BACKUPS/proxmox-backups/dump/vzdump-lxc-6970-2020_06_19-06_55_25.tar.zst'
INFO: Total bytes written: 329407897600 (307GiB, 18MiB/s)
INFO: archive file size: 3.13GB
INFO: restarting vm
INFO: guest is online again after 17734 seconds
INFO: Finished Backup of VM 6970 (04:55:34)
INFO: Backup finished at 2020-06-19 11:50:59
INFO: Backup job finished successfully
TASK OK

 

[fabian]

sounds like you have highly compressible data. is this container stored on (compressed) ZFS? if so, you might not notice how much logical data is contained within.

 

[effgee]

Hmm, yes you are correct.
I just checked the zfs stats for that volume, and yes, logically it is using 300 gigs.
The vm is a monitoring vm (LibreNMS and CheckMK) only monitoring about 10 servers and has a bunch of RRD files.

I do not know what tools to use to figure out what files are so big and compressible in the vm. du and ncdu just show me 22 gigs of used space with no hints to which files are so big.


logicalreferenced 326G
logicalused 384G

refcompressratio 20.50x
referenced 22.8G
refquota 40G

Ok, I found the right switches.

du -hs --apparent-size
or in ncdu after counting, press a to switch to apparent size.

So it turns out RRD files are extraordinarily compressible with zfs, (lz4) in this case. But it really gives the backup a hard time.

 

[LnxBil]

Do you TRIM your data?