Reduce read IO for 10TB lxc backup

ThomasBlock

Member
Sep 6, 2022
12
1
8
Hi,

i run Proxmox lxc containers on zfs, some of ~ 10 TB size.
i do daily backups on PBS ( lvm there, no zfs )

The Problem: The data changes very little ( mostly appending ).
But every day the full 10 TB are read, which takes 10 hours. ( see log below )

So it's good that I use snapshots on proxmox-backup-client and i only transfer 240 GB .
My Question: Can i somehow also use snapshpots for reading? If the zfs snapshot from the last day would be kept, then the client would know that he would only have to read the changes of the last 24 hours, right?

I would compare that to zfs replication - is that behaviour somehow reachable? ( do i need zfs on PBS? )

or should i disbale encryption and compression, so that the source and traget are the same, and then only the delta can be read?


INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: creating Proxmox Backup Server archive 'ct/111/2023-07-16T09:00:05Z'
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=encrypt --keyfd=11 pct.conf:/var/tmp/vzdumptmp2629631_111/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 111 --backup-time 1689498005 --repository ***
INFO: Starting backup: ct/111/2023-07-16T09:00:05Z
INFO: Client name: Rotkohl
INFO: Starting backup protocol: Sun Jul 16 11:00:10 2023
INFO: Using encryption key from file descriptor..
INFO: Encryption key fingerprint: ***
INFO: Downloading previous manifest (Sat Jul 15 11:00:01 2023)
INFO: Upload config file '/var/tmp/vzdumptmp2629631_111/etc/vzdump/pct.conf' to *** as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to *** as root.pxar.didx
INFO: root.pxar: had to backup 247.724 GiB of 9.516 TiB (compressed 239.132 GiB) in 35365.27s
INFO: root.pxar: average backup speed: 7.173 MiB/s
INFO: root.pxar: backup was done incrementally, reused 9.274 TiB (97.5%)
INFO: Uploaded backup catalog (4.156 MiB)
INFO: Duration: 35372.60s
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!