TASK ERROR: connection error: connection closed before reading preface

[roadrunner_rad]

Hi,

we are running a Test Proxmox Backup Server as VM with Version 1.0.8 an are doing file Backups with proxmox -backup-client 1.0.8 from a bash script.

When the script is run via terminal everything works fine. When we schedule the script via crontab we get an RC=255 and the task shows the error

2021-03-02T04:25:25+01:00: starting new backup on datastore *********': "host/********/2021-03-02T03:25:01Z" 2021-03-02T04:25:25+01:00: backup failed: connection error: connection closed before reading preface 2021-03-02T04:25:25+01:00: removing failed backup 2021-03-02T04:25:25+01:00: TASK ERROR: connection error: connection closed before reading preface

in the web ui of pbs.

Seems not related to proxmox-backup-client as it works with terminal but I could not find any solution or hit in the forum.

Any Idea to solve this ?

 

[Stoiko Ivanov]

please share the crontab where you've entered the job - and also the log from a backup-run on the terminal (make sure to obfuscate any passwords)

 

[roadrunner_rad]

VM is Debian 10.8. There is only a link in /etc/cron.daily called backup pointing to the script

root@********:~# ls -al /etc/cron.daily/backup lrwxrwxrwx 1 root root 40 Mär 2 08:53 /etc/cron.daily/backup -> /root/nextcloud-scripts/nc_pbs_backup.sh

The scipt looks like this (anonymized ********):


#!/bin/bash export PBS_REPOSITORY=******** export PBS_PASSWORD=******** export PBS_INCLUDE="root.pxar:/root etc.pxar:/etc nextcloud.pxar:/var/www/nextcloud mysqldata.pxar:/var/lib/mysql mysqllog.pxar:/var/log/mysql nextclouddata.pxar:/var/nextcloud-data" exe=proxmox-backup-client pbs=$(which $exe) if [ ! -z "$pbs" ] then backup_result=$($pbs backup $PBS_INCLUDE) backup_rc=$? logger "backup to $PBS_REPOSITORY RC=$backup_rc" if [ $backup_rc = 0 ] then $pbs snapshots|grep $(hostname)| awk '{print $2}' else logger "$pbs $backup_result" fi else echo "$exe not executable" fi


The system log shows the following:

Mar 2 05:25:21 ******** root: backup to *********:********* RC=255 Mar 2 05:25:21 ******** root: /usr/bin/proxmox-backup-client Starting backup: host/********/2021-03-02T04:25:01Z#012Client name: ********01#012Starting backup protocol: Thu Mar 2 05:25:01 2021

 

[roadrunner_rad]

Console output (proxmox-backup-client detail output is kept in variable see script):

root@********:~# /etc/cron.daily/backup host/********/2021-02-07T10:40:20Z host/********/2021-02-14T04:25:01Z .....

 

[Stoiko Ivanov]

what's the output of proxmox-backup-client when run via cron and what does it print on stderr (you only log the output if the exit code is 0 and you don't log the stderr)?

try running:

${pbs} backup $PBS_INCLUDE > /tmp/pbs.output 2> /tmp/pbs.err

instead of backup_result=$($pbs backup $PBS_INCLUDE) and then check the 2 resulting files.

on a sidenote - the check for the existence of $pbs checks if the variable is zero - if you log that proxmox-backup-client is not executable - I'd use '-x' as test (see `help test` in bash)

as a general recommendation - I'm quite the fan of `shellcheck` https://github.com/koalaman/shellcheck (installable in debian and most linux-distros) for quickly seeing if I can improve something in a shellscript

I hope this helps!

 

[roadrunner_rad]

Hi Stoiko

thank you for the suggestion and the tipp with -x as well as shellcheck.

The stdout of proxmox-backup-client is logged to syslog in case of an error see post above. But I will give the redirection to files a try and see what happens tomorrow...

Regards Joachim

 

[roadrunner_rad]

OK this morning I found the following from the cron scheduled script:

stderr:

Error: http upgrade request timed out

stdout:

Starting backup: host/********/2021-03-05T04:25:01Z Client name: ********

 

[Stoiko Ivanov]

Error: http upgrade request timed out

this sounds strange like a network/proxy/firewall issue:
* it seems the backup-client can connect to the backup-server, but the upgrade of the connection to http/2 (which happens for the actual backup) gets dropped somewhere in the middle?

It seems to me that the issue is not rooted in the run via cron vs. via command-line.
* maybe the network is loaded a bit more in the morning (or whenever cron daily runs on your machine (check /etc/crontab))
* might also be worth to take a look at the logs on the PBS - server side

I hope this helps!

 

[roadrunner_rad]

Hi Stoiko,

thank you for the suggestions. As this is on the local network only no proxy ore firewall in between. Serverside log is in the first message of this thread. May be that there are some network activities, so I changed the crontab to run later and suprise, it works.

But I think if you are right there should be a configurable timeout in the client or a retry mechnism because we want reliable backups...

I think this is not solved with this workaroung. I will have an eye on this.

Regards
Joachim

 

[roadrunner_rad]

Today there were new breakdowns in Backup with same behaviour as above.
I think there is a Problem with our setup and timeouts.

The Backup Server VM is running with backup storage on an nfs share and that nfs share resides for testing purposes on a cheap synology diskstation.

This diskstation has a sleep mechanism and needs about 30 seconds to wake up for serving nfs requests. So I think that may be the reason for this behaviour. Is there a possible solution to configure/increase/specify the timeout in PBS ?

Regards
Joachim

 

[maik83]

We are seeing the same issue in our setup.

We are using Proxmox Backup Server 1.1-2. The Datastore a directory which we mounted via CIFS on a storage server.

To save electritity and cooling the storage server is configured to put the hdds in sleep mode.