container backup is slow

[grin]

On my Large container the pbs takes 25 hours now. The same contaner with backuppc (incremental, rsync) takes about 90 minutes.
Backing up a VM image is fast.

(Non-incremental backuppc take 12 - 40 hours.)

It seems pbs could use a feature where files with old atime/mtime would be skipped.

 

[wolfgang]

Hi,

What is a large container? Can you please provide the size in GB/TB?
I can't see this problem with a container over 100GB.

 

[grin]

You're right, "large" is subjective.
The one in question is 2.6 TB with about 18 million files.

There may be some relevant factors here:

1. full filesystem traversal time (have to be done in all cases)
2. stat()'ing the file (also had to be done to see incrementality based on update time)
3. reading the whole file
4. checksumming the whole file
5. transferring any part or all of the file

Rsync incremental does step 1,2 for old files, 3,4 for the rest and 5 for the pissible difference. However rsync cannot detect dupes between different files, so - theoretically - transfers a lot of unnecessary data as well at step 5.

I guess the main problem with pbs is that it does 1,2,3,4 for all the files, and possibly all of the checks (but I believe it uses dupe detection for the whole backup, not just the same file).
Unfortunately step 3 is very heavy, especially since most of the data is not in any cache: combined with steps 1 and 2 it may be the reason for the extended duration (my guess).

 

[wolfgang]

I let run a test setup that will create this amount of files and then test the Backup.
Have a nice weekend.

 

[carsten2]

I opened a feature request https://bugzilla.proxmox.com/show_bug.cgi?id=3138 to speed up container backup, but it gets closed multiple times without discussion and even with wrong claims. The feature request is perfectly valid, because PBS would be the ONLY backup solution in the world, which needs several hours to backup UNCHANGED file systems. I claim that a dramatic improvement IS possible and it is even not too complicated to implement.

 

[tom]

closed multiple times without discussion and even with wrong claims.

No, You just ignored and mis-understood the answer. You were asked to continue the discussion on the right place, the pbs mailing list.

We would be really happy if we could easily speedup the system dramatically - you just need us to tell how ...

So far we miss this.

 

[carsten2]

1) I subscribed to the mailing list, but it is very difficult to follow anything there, because there there are no ordered discussion threads, all topics are mixed up. I don't see how to do a disussion there. Is there any tool for this or how do you use it?

2) Why don't you explain the problem here in the forum (which is much easier to use than the mailing list)?

3) I really dont' understand WHY my feature request is closed all the time, without any discussion. The REQUEST is perfectly valid and necessary. The only thing which could be discussed is how it can be solved. It is ridiculous to have a backup solution which takes hours to backup an UNCHANGED file system and say, that NOTHING will be ever done to improve it. So please keep the feature request open (even if my solution suggestion might be not the right one)!

4) I did explain how to improve the speed, so why are you saying it didn't?

 

[tom]

1) I subscribed to the mailing list, but it is very difficult to follow anything there, because there there are no ordered discussion threads, all topics are mixed up. I don't see how to do a disussion there. Is there any tool for this or how do you use it?

If you know how to improve a complex backup framework, you should be able to work with mailman. Just start a new topic/email.

2) Why don't you explain the problem here in the forum (which is much easier to use than the mailing list)?

Because the developers works primarily on the pbs developer list.

3) I really dont' understand WHY my feature request is closed all the time, without any discussion.

Because the bug tracker is not for discussions, almost all devs will miss your posts there.

4) I did explain how to improve the speed, so why are you saying it didn't?

At least for all of us, no.

And PLEASE continue on the pbs mailing list. NOT here and NOT in the bugzilla, should be clear now.