Backup jobs breaks with connection refused

[carsten2]

I have a SSH tunnel from the Remote PBS port 8008 to the local PBS port 8007. The Remote PBS should sync the backups from the local PBS via this tunnel. However the sync jobs breaks after 30-120 min with the following error message:​

error trying to connect: error connecting to http://localhost:8008 tcp connect error: Connection refused

This error appears quickly for every VM to be synched and the job is aborted. Restarting the job helps.

For some reasons the SSH tunnel brekas sometimes but is there no retry on the PBS side before terminating the whole backup job?

 

[Stefan_R]

No, that's the job of TCP (and it generally does that really well). But since you're trying to connect via an SSH tunnel, the error you're seeing emerges from the kernel directly: TCP will retransmit until it receives a response, not until it receives a *successful* response. And since your SSH tunnel is down (as you mention), your host kernel will say "well, port 8008, there's nothing there, so that's an error" and immediately return a valid (but unsuccessful) TCP packet.

 

[carsten2]

Yes, but even if i had no ssh tunnel, it could happen that a connections breaks and a retry is necessary from the client. The sync job doesn't seem to retry but abort at first failure.

 

[Stefan_R]

Right, and as I said, if a connection breaks then TCP will retry for a bit until the connection either comes back or times out. A regular unreliable network thus will not fully interrupt your sync. That is standard practice, TCP handles the reliability - we do not retry on the application layer if the network times out.

In your case however, it doesn't time out, so no retry is done - it actively gets back a response from your kernel (since your SSH tunnel is down) stating "don't retry, there's nothing here", and TCP obliges.