[SOLVED] Backup client seems to "hang" when backing up bigger files

[Tim_kwakman]

Hello,

I've been trying to figure out what's wrong, but I'm not able to find anything.

When I run a backup job on a specific server, it hangs with bigger files, when I add a 3MB file, it works just fine. But when I try to upload a 100MB .SQL file, it just hangs.

I checked;
I/O top: No usage
if top: No usage
htop: No CPU usages

Command I'm using:
proxmox-backup-client backup web01-server-databases.pxar:/home/backups/store/databases/ --repository server1-offsite@pbs@backup.example.com:Store --backup-id web01-offsite-databases --keyfile /home/backup/fetchers/encryption/encryption.key --verbose

This is what it does;
Starting backup: host/<snip>/2021-03-23T19:36:28Z
Client name: offsite
Starting backup protocol: Tue Mar 23 20:36:28 2021
Using encryption key from '/home/<snip>/fetchers/encryption/encryption.key'..
Encryption key fingerprint: <snip>
Downloading previous manifest (Tue Mar 23 20:36:15 2021)
Upload directory '/home/<snip>/backups/databases/' to '<snip>@pbs@<snip>:8007:<snip>' as <snip>.pxar.didx
"fallback.sql" (this one is 116KB, uploads just fine)
"haproxy.sql" (this one is 205KB, uploads just fine)
"tickets.sql" (this one is 244MB, stuck)

I see the tasks within the GUI, no issues at all there, and when I ctrl + c, it will set it to failed.
This also happend when I rsynced a big file to the local backup server, and execute it through there.

It also happens when I ZIP / tar the .SQL files. The logs do not show any information, so I'm not sure where to look next, so that's why I'm posting here.

Thanks for any replies!

-Tim

 

[fabian]

which PBS client version are you using?

 

[Tim_kwakman]

Hi,

I'm running the following versions;
client version: 1.0.11
Server: Backup Server 1.0-11

-Tim

 

[fabian]

is there progress in the server side tasks while the client appears to "hang"?

 

[ar7yss]

Hi,
I was having the exactly same problem.
It was not happening on all servers.
Version not working: 1.0.9
version working: 1.0.6

I uninstalled 1.0.9-1 that was not working and installed 1.0.6-1.
Everything is working now.
I also tested 1.0.8-1 with success.
I thing there is something wrong in 1.0.9-1

 

[Tim_kwakman]

Hi,

This is what it does; <snip video link>

It has been sitting there since your last message (49 minutes) and still nothing.

Just as @ar7yss says, it only happens on some servers, I forgot to mention that. I used 1.0.9 before, I updated to 1.0.11 a few hours ago. I haven't tried older versions.

-Tim

 

[ar7yss]

I just forgot to mention that it seems to happen when the global size is larger than 30 or 40 Mo.
if I create a 20M file, it can backup it.
if I copy this file, It can backup the first one and hang on the second one.
when I launch tcpdump, there is something when it's backuping, but when it's hanged, it only send a keepalive every 20 or 30 seconds, but there is no data.

 

[fabian]

it's definitely possible to backup more than 30 or 40MB can you try attaching with strace to the client process and see whether it makes any progress when it is stalled? ideally if you can reproduce it with files that are not sensitive, you could create a test account and back those up and then attach the full strace output (e.g., something like 'strace -ff -o strace_pbs_client_ proxmox-backup-client ....', and then attach a zip/tar of the strace_pbs_client_* files generated)

 

[fabian]

just out of interest, could you also run proxmox-backup-client benchmark --repository YOUR_REPOSITORY and post the results?

 

[fabian]

also, are the client systems single or dual core machines by any chance?

 

[Tim_kwakman]

Downgrading it to 1.0.8-1 works great;

web01-offsite-databases.pxar: had to upload 245.19 MiB of 245.19 MiB in 5.50s, average speed 44.56 MiB/s).
Uploaded backup catalog (121 B)
Duration: 5.72s
End Time: Wed Mar 24 12:48:27 2021

This client is a single core VPS.

I have the issue on two servers, and both have one core, that may be the cause here. I have another server that is not really used right now that also has one core, I'll check if it happens there as well.

-Tim

 

[fabian]

yes, if all your hanging systems are single-core than it's likely #3359, fixed in git already. the fix should be in the next released version (> 1.0.11-1, likely 1.0.12-1)

 

[Tim_kwakman]

Ah great, then I'll wait for an update and use the older version for the time being. As there is already a bug report for it; Do you still require additional information, or the "strace" output (I'll have to look into how that works in that case). Or is all the information already known?

-Tim

 

[fabian]

unless you also see hangs on systems with more than two cores, no additional information is needed

 

[ar7yss]

it's definitely possible to backup more than 30 or 40MB can you try attaching with strace to the client process and see whether it makes any progress when it is stalled? ideally if you can reproduce it with files that are not sensitive, you could create a test account and back those up and then attach the full strace output (e.g., something like 'strace -ff -o strace_pbs_client_ proxmox-backup-client ....', and then attach a zip/tar of the strace_pbs_client_* files generated)

I know it's possible to backup more than 30 or 40 MB with pbs, our 3.5TB datastore is near full.
like I said, my test is simple.
1 file, 20 MB -> OK
2 files, 20MB each -> NOK
1 file 40MB, -> NOK
So in 1.0.9 with one core, I can assure you, it cannot backup 40MB.
I tested with two cores in 1.0.9, it's working so it was indeed bug 3359.
Thanks.