Hey Matrix,to help better, please provide your storage config cat /etc/pve/storage.cfg and your container config
rbd: Default
content images,rootdir
krbd 0
pool Default
Yes, all VM's and LXC's are on the Ceph StorageOther VMs are in the same storage?
Hey Fabian,because for containers, PBS needs to read and chunk all the data to find out what to upload, for (running) VMs there is a shortcut because we can tell Qemu to keep track of which chunks changed.
How can one check/change this?of course, increasing your read/chunk performance might also be an option depending on your current hardware
besides the big difference (dirty bitmaps with running VMs allow skipping of read operations of unchanged chunks), there's also way less complexity involved in the processing of backup data for VMsSo I'm in the same boat. Before implementing PBS for production, I deployed it in test cluster. Test LXC (4tb) takes 2 hours to backup, while nothing changed. The strange thing is that durign the 2 hour that takes the backup to finish, there isn't much of a disk activity on PBS or machine with LXC ... CPU is even cold.
Test VMs that are 128BG, 300GB and 512GB take seconds to backup. There is one thing that doesn't make sense, all proxmox staff keeps saying that only VM's that are running can have the diff done quickly, but one machine with VM's only get's up periodically and does the backup right after startup, and this one happens within few seconds. All this, while LXC from machine that runs 24/7 takes forever.
what do you mean by verify? a PBS verification happens on the PBS side, and consists of reading all chunks of the snapshot and hashing their contents, so it's both read and CPU intensive, but it's basically the same for fidx and didx backups, except that the index format is a bit different. the contained data is not logically analysed or verified at all, so the difference in contents doesn't matter.To add insult to the injury, all VM verification takes a second or two for small incremental backups, while LXC takes 7 hours to verify incremental backup (which is few meg might I add). and that does hog disks like crazy for LXC.
Yes, you are absolutelly right. THOU, you (ie, operating system of proxmox) already knows what is the filesystem for underlying storage, because I select it from drop down in the gui - ZFS. There can be a separate logic per storage type, it's not that much complicated to encapsulate different diff functionality within switch case.you can probably guess the latter is affected by how your file systems performs w.r.t. directory and metadata operations.
Yes, this point - I meant on PBS. Still, this one is a bit strange - even thous backups are incremental for CT (amount reported to be sent from PVE to PBS), the verify (periodic job OR on receive - tested both) takes 6 - 7 hours, and PBS seems to scratch through all 4TB of data on the disk. And yes previous backups of this CT are verified ... one would assume that only the incremental backup of 10MB that was sent to PBS would get verified ... as you know, an increment ?. And alto now there are now 10 backups of 4tb CT in PBS, it only reports that storage is 4tb (and a bit more) occupied, so I don't presume every single backup is whole 4tb of data. Even the ZFS list shows that it's not that big:what do you mean by verify? a PBS verification happens on the PBS side, and consists of reading all chunks of the snapshot and hashing their contents, so it's both read and CPU intensive, but it's basically the same for fidx and didx backups, except that the index format is a bit different. the contained data is not logically analysed or verified at all, so the difference in contents doesn't matter.
zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 12.7G 1.74T 96K /rpool
rpool/ROOT 12.7G 1.74T 96K /rpool/ROOT
rpool/ROOT/pve-1 12.7G 1.74T 12.7G /
rpool/data 96K 1.74T 96K /rpool/data
storage_cold 3.39T 49.3T 3.39T /mnt/datastore/storage_cold
no, the backup client is file-system agnostic, it uses regular file/directory operations, there is no diffing at that level.Yes, you are absolutelly right. THOU, you (ie, operating system of proxmox) already knows what is the filesystem for underlying storage, because I select it from drop down in the gui - ZFS. There can be a separate logic per storage type, it's not that much complicated to encapsulate different diff functionality within switch case.
if you verify a single snapshot, all the chunks of that snapshot will be verified. a snapshot is always complete, there are no incremental or full snapshots - all snapshots are "equal". only the uploading part is incremental in the sense that it skips re-uploading chunks that are already there on the server. the chunk store used by the datastore takes care of the deduplication. so yes, in your case, (re)verifying a single snapshot will read and verify 4TB of data. (re)verifying ten snapshots of the same CT with little churn between the snapshots will cause only a little bit more load than verifying a single one of them, since already verified chunks within a single verification task will not be verified a second time for obvious reasons.Yes, this point - I meant on PBS. Still, this one is a bit strange - even thous backups are incremental for CT (amount reported to be sent from PVE to PBS), the verify (periodic job OR on receive - tested both) takes 6 - 7 hours, and PBS seems to scratch through all 4TB of data on the disk. And yes previous backups of this CT are verified ... one would assume that only the incremental backup of 10MB that was sent to PBS would get verified ... as you know, an increment ?. And alto now there are now 10 backups of 4tb CT in PBS, it only reports that storage is 4tb (and a bit more) occupied, so I don't presume every single backup is whole 4tb of data. Even the ZFS list shows that it's not that big:
Code:zfs list NAME USED AVAIL REFER MOUNTPOINT rpool 12.7G 1.74T 96K /rpool rpool/ROOT 12.7G 1.74T 96K /rpool/ROOT rpool/ROOT/pve-1 12.7G 1.74T 12.7G / rpool/data 96K 1.74T 96K /rpool/data storage_cold 3.39T 49.3T 3.39T /mnt/datastore/storage_cold
Side point (third one):
What I was thinking (more hoping) was that PBS would be (more) aware of it's storage, and if incremental backups would be performed on storage like ZFS, it would be based on built in snapshots,