Backup of LXC on zfs with 3 subvolumes - 12GB of backup size vs 2GB extracted

[pongraczi]

Hi,

I had an LXC which had 3 volumes on a local zfs storage:

• rootfs: local-zfs:subvol-9301-disk-0,size=64G
• mp0: local-zfs:subvol-9301-disk-1,mp=/var/lib/postgresql/9.6/main/,size=32G
• mp1: local-zfs:subvol-9301-disk-2,mp=/var/lib/postgresql/9.6/main/pg_xlog,size=32G

When I made a backup of this LXC, I got 12.3GB on the backup server.

After the size of the backup seemed fine (the size and all was ok), the original dataset was deleted.

After a while, it was necessary to check something on this retired archive, I restored this lxc without problem.

Now the surprise was:
- the mp0 and mp1 were created, but were empty
- the rootfs showed the following:

ZFS report:

rpool/data/subvol-9301-disk-0  used                  1.97G                           -
rpool/data/subvol-9301-disk-0  available             62.0G                           -
rpool/data/subvol-9301-disk-0  referenced            1.97G                           -
rpool/data/subvol-9301-disk-0  compressratio         6.53x                           -
rpool/data/subvol-9301-disk-0  usedbydataset         1.97G                           -
rpool/data/subvol-9301-disk-0  usedbychildren        0B                              -
rpool/data/subvol-9301-disk-0  usedbyrefreservation  0B                              -
rpool/data/subvol-9301-disk-0  refcompressratio      6.53x                           -
rpool/data/subvol-9301-disk-0  written               1.97G                           -
rpool/data/subvol-9301-disk-0  logicalused           12.4G                           -
rpool/data/subvol-9301-disk-0  logicalreferenced     12.4G                           -

Data inside the dataset:

root@hn3:/rpool/data/subvol-9301-disk-0# du -h --max-depth=1
1.1G    ./var
512    ./dev
3.4M    ./sbin
512    ./media
1.0K    ./lib64
512    ./sys
512    ./home
12K    ./tmp
512    ./mnt
512    ./srv
177K    ./opt
20K    ./share
22M    ./lib
244M    ./root
512    ./proc
4.7M    ./bin
729M    ./usr
15K    ./run
2.8M    ./etc
512    ./boot
2.0G    .

So, somehow the backup seems problematic.
Obviously, I made a mistake to not check the restore of this lxc on the target, I can't fix the past. Others should learn this lesson.

I checked the log of the backup:

2022-09-04 23:43:06 INFO: Starting Backup of VM 301 (lxc)
2022-09-04 23:43:06 INFO: status = running
2022-09-04 23:43:06 INFO: CT Name: postgres01
2022-09-04 23:43:06 INFO: including mount point rootfs ('/') in backup
2022-09-04 23:43:06 INFO: excluding volume mount point mp0 ('/var/lib/postgresql/9.6/main/') from backup (disabled)
2022-09-04 23:43:06 INFO: excluding volume mount point mp1 ('/var/lib/postgresql/9.6/main/pg_xlog') from backup (disabled)
2022-09-04 23:43:06 INFO: backup mode: snapshot
2022-09-04 23:43:06 INFO: ionice priority: 7
2022-09-04 23:43:06 INFO: create storage snapshot 'vzdump'
2022-09-04 23:43:06 INFO: creating Proxmox Backup Server archive 'ct/301/2022-09-04T21:43:06Z'
2022-09-04 23:43:06 INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp17771/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --backup-type ct --backup-id 301 --backup-time 1662327786 --repository xxxxxx@yyyyy@zzzzzzzz.hu:samba
2022-09-04 23:43:06 INFO: Starting backup: ct/301/2022-09-04T21:43:06Z
2022-09-04 23:43:06 INFO: Client name: hn1
2022-09-04 23:43:06 INFO: Starting backup protocol: Sun Sep  4 23:43:06 2022
2022-09-04 23:43:07 INFO: Upload config file '/var/tmp/vzdumptmp17771/etc/vzdump/pct.conf' to 'xxxxxx@yyyyy@zzzzzzzz.hu:samba' as pct.conf.blob
2022-09-04 23:43:08 INFO: Upload directory '/mnt/vzsnap0' to 'xxxxxx@yyyyy@zzzzzzzz.hu:8007:samba' as root.pxar.didx
2022-09-04 23:44:42 INFO: root.pxar: had to upload 41.75 MiB of 12.30 GiB in 94.54s, average speed 452.25 KiB/s).
2022-09-04 23:44:42 INFO: root.pxar: backup was done incrementally, reused 12.26 GiB (99.7%)
2022-09-04 23:44:43 INFO: Uploaded backup catalog (1.14 MiB)
2022-09-04 23:44:43 INFO: Duration: 96.66s
2022-09-04 23:44:43 INFO: End Time: Sun Sep  4 23:44:43 2022
2022-09-04 23:44:43 INFO: remove vzdump snapshot
2022-09-04 23:44:43 INFO: Finished Backup of VM 301 (00:01:37)

It seems, the data under the mp0 and mp1 are gone forever, however the backup size is around 12.3GB, which means, it should contain all the data.

My question, what is really going on with this backup?

• So, the log said the mp0 and mp1 excluded from rootfs: that's ok, these should kept on a different volume. What happened? mp0 and mp1 never copied into the backup?
• The backup size is 12.3GB, while the countable data is around 2GB (du and df shows only 2GB).
• The zfs reported the total data around 12.41GB, which seems closer to the reality, but the difference is invisible: where are the 10GB of data?

Is there any way to access to the data, convert the backup somehow, because using proxmox backup server to browse the filesystem shows only the 2GB, mount points are empty.

Does anybody have similar experiences?

Thanks

 

[fabian]

the mp0 and mp1 were not included in the backup (as indicated by the log). the size difference is likely because of compression. du and "used" in zfs will show you how much space the data takes on disk, the (uncompressed) logical data stored (which is what is fed to proxmox-backup-client when doing a backup) might be more.

 

[pongraczi]

the mp0 and mp1 were not included in the backup (as indicated by the log). the size difference is likely because of compression. du and "used" in zfs will show you how much space the data takes on disk, the (uncompressed) logical data stored (which is what is fed to proxmox-backup-client when doing a backup) might be more.

Thank you, I forgot the switch for the du: --apparent-size

Anyway, is that a normal thing, the backup just excluded the lxc mount points, without any warning?
What would be the proper way to backup an lxc with multiple volume?
Thanks!

 

[fabian]

yes, the default in the backend is for mountpoints to not be backed up (for historical reasons, and those defaults are hard to change). the GUI defaults to setting the backup option on newly created mountpoints. I wouldn't exactly call it without warning though - the backup task log does tell you what is included and excluded, and you should definitely check that when setting up backups to see it does what you expect it to do (ideally combined with regular restore tests).

 

[pongraczi]

Thank you.
When I wrote without warning, I meant before the task start, not in the log. It is a similar situation to the Hitchhikers guide to Galaxy ))

Thank you again for your help and explanation.

Reading only for fun:
"But Mr Dent, the plans have been available in the local planning office for the last nine month."
"Oh yes, well as soon as I heard I went straight round to see them, yesterday afternoon. You hadn't exactly gone out of your way to call attention to them had you? I mean like actually telling anybody or anything."
"But the plans were on display..."
"On display? I eventually had to go down to the cellar to find them."
"That's the display department."
"With a torch."
"Ah, well the lights had probably gone."
"So had the stairs."
"But look, you found the notice didn't you?"
"Yes," said Arthur, "yes I did. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying Beware of the Leopard."

 

[fabian]

but excluding a mountpoint is perfectly valid as a choice, it's not like anything is wrong with the config..

 

[pongraczi]

Excluding a mount point is ok, but on the GUI I do not see any choice to exclude/include.

Unfortunately you are right, the Backup should be ticked in the settings of the mount point.



It was a PEBCAK