[solved] connection error: Transport endpoint is not connected (os error 107)

Elleni

Member
Jul 6, 2020
133
6
23
50
I got many of the following error messages on the weekend:
Code:
2020-12-19T23:08:31+01:00: starting new backup reader datastore 'pbs-backup': "/pbs-backup"
2020-12-19T23:08:31+01:00: protocol upgrade done
2020-12-19T23:08:31+01:00: GET /download
2020-12-19T23:08:31+01:00: download "/pbs-backup/vm/121/2020-11-11T06:17:33Z/index.json.blob"
2020-12-19T23:08:31+01:00: TASK ERROR: connection error: Transport endpoint is not connected (os error 107)

Code:
2020-12-19T23:08:28+01:00: starting new backup reader datastore 'pbs-backup': "/pbs-backup"
2020-12-19T23:08:28+01:00: protocol upgrade done
2020-12-19T23:08:28+01:00: GET /download
2020-12-19T23:08:28+01:00: download "/pbs-backup/vm/106/2020-12-18T17:26:30Z/index.json.blob"
2020-12-19T23:08:28+01:00: GET /download
2020-12-19T23:08:28+01:00: download "/pbs-backup/vm/106/2020-12-18T17:26:30Z/qemu-server.conf.blob"
2020-12-19T23:08:28+01:00: GET /download
2020-12-19T23:08:28+01:00: download "/pbs-backup/vm/106/2020-12-18T17:26:30Z/drive-scsi0.img.fidx"
2020-12-19T23:08:29+01:00: register chunks in 'drive-scsi0.img.fidx' as downloadable.
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/67c0/67c0b22f8b83187c7a2b6ddfb9e5806f8d04f48afea09bcfc1f2c65f38fd955f"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/bbf7/bbf78b5308dbc2ecead77ee443bb2a2450a13860d052ad440a4d84a24667819d"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/86b9/86b97f537079beabdbf1c5e47d387ca46360dde9dbaf28ff23fa65bba6a063e9"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/5d59/5d59058782ed164478adc82580ff2b3f89f40f3d224283a7e3b3dcdc656bb4a9"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/5a18/5a18aa9fb3c459f69f603fbf803c8ac6e3187e30161c9c17d70abb9958d33e3d"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/308b/308b4f403d83e065c3763515293723f5061f5a15a16de2f42d0e12a06c89db3d"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/80b6/80b647d4fd3115e0cc17788a90d3a676376f9da22e8787ac06d6e4c553441877"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/cc5f/cc5f5de499a0142897a52a9c935c93c875ded01f436e577452df39453050ede5"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/16f6/16f602494ff0a626f685f63e072734490d090e76550a3f77c764214e3cc1c3d3"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/e960/e9605688e420104aa459458b32abfa7cd1c2c335b4ae56152cbc7ee2058fbca6"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/e917/e917b77f72dc304db0ce7d1702b85f7cd062dbe92863e73809d76f869b787563"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/d2bb/d2bb007147d93bab357be31093586082232335ff6bab3879559960f3f658346b"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/3fe6/3fe6a9edbbde5a9e80f452bdd378712a3a0414c067551c7ef2a539a29bb8d63d"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/dde5/dde5d8c60ee71f58adf71f2928c3790b6c8453b67f1588616a18c5abeba21a95"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/e2ad/e2ad277b9fa5f5acba95f5dc7e6aabd4b5ed4cbcc7e6274c4e22bd3082d86582"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/c5be/c5be9d2c077d3c39569f4bbe4860e780b3b1f5a567a7c69cd3e70517f4f458e1"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/5560/55604ed98832ed42c1912c4ca8caadca8b44c490bdc70ad3d15a97b479e65d09"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/0c67/0c67f00838d4a4524d82d37a0bfa2e50975efd7764af17b671df350990c98b56"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/ff3c/ff3cd823f7cfc8f68a6b699afebc2b17d38ab3fad35ca76ad9310f01b1635d4d"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/4cae/4cae2b2b26604337212f09d6e3c3a92ad61779e4514de62bea43e6ef1417685f"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/595f/595f19bfaafd411bf7dbd6f4f4a4e5d5e64f496fdeccea6d48a9182bfe21f425"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/0130/0130817acc522f0b42dd43f57cbff651141c63e6723ece4444b3111969a61e67"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/4972/49724ea1bd939d8a31de3ce077e7fc7fd255cb918caf1fc9d00a6dac100be99e"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/6806/6806cde50f88a6fe35d2407826c37eb1129fddfaf40b19eebb60560b4ed26c01"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/ce07/ce07309a27817cb1144711f67d8d3fa1548c18f4283d77c965b31b6cff204bd4"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/9efb/9efb6ebf7bdf1846d8a7492d35350c9339e3cdae8873e830708a33466918cdc3"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/f100/f10031a4c903eb23ff2f343bfc5f79be459c1871d760d402857a43667d211b18"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/e6e3/e6e35d86cb40b752743fc2e8c15b434f8b87920b64e8afd3c0c892676d8577e6"
2020-12-19T23:08:29+01:00: GET /chunk
2020-12-19T23:08:29+01:00: download chunk "/pbs-backup/.chunks/0fd9/0fd98b496f02c6ab9d8ac18352d81f13a30611aadaa602097bc147bbb99a18fc"
2020-12-19T23:08:29+01:00: GET /download
2020-12-19T23:08:29+01:00: download "/pbs-backup/vm/106/2020-12-18T17:26:30Z/client.log.blob"
2020-12-19T23:08:31+01:00: TASK ERROR: connection error: Transport endpoint is not connected (os error 107)

What does that mean? During weekend we have not configured any backups to datastore pbs-backup. Only a sync job for to a local remote datastore (removable usb disk) which did successfully get executed.

Is it possible that these are backups that were pruned from pbs-backup datastore thus cannot be verified on the removable datastore?

Talking of prune, I checked and realized that the configured prunge and gc settings are gone, and I don't know why.
 
Last edited:
sync basically pulls the metadata and chunks via the download API path, so likely those log entries were just the sync? that they end with the transport message can happen when the client closes the connection before the server expected too.
 
Hm, well the client and the server is on the same machine, as I configured a remote -> localhost as workaround to be able to sync the datastore content to an offline medium. So am I supposed to just ignore those kind of messages, or can I do something to make them disapear?
 
yes, you can ignore those messages if the client-side (sync/backup/restore) does not complain/show any errors.
 
Just to clarify - you mean as the following message which appears just after the last error message indicates the the sync finished successfully, then I can ignore it, right?:

Code:
2020-12-19T22:15:02+01:00: Starting datastore sync job 'local:pbs-backup:sandisk:s-f0eb4571-25bf'
2020-12-19T22:15:02+01:00: task triggered by schedule 'sat 22:15'
2020-12-19T22:15:02+01:00: Sync datastore 'sandisk' from 'local/pbs-backup'
2020-12-19T22:15:10+01:00: re-sync snapshot "vm/100/2020-12-11T17:10:01Z"
2020-12-19T22:15:10+01:00: no data changes
2020-12-19T22:15:10+01:00: re-sync snapshot "vm/100/2020-12-11T17:10:01Z" done
2020-12-19T22:15:10+01:00: percentage done: 7.58%
2020-12-19T22:15:10+01:00: sync snapshot "vm/100/2020-12-14T08:10:01Z"
2020-12-19T22:15:10+01:00: sync archive qemu-server.conf.blob
2020-12-19T22:15:11+01:00: sync archive drive-scsi0.img.fidx
2020-12-19T22:15:27+01:00: downloaded 660037055 bytes (38.04 MiB/s)
2020-12-19T22:15:27+01:00: got backup log file "client.log.blob"
2020-12-19T22:15:27+01:00: sync snapshot "vm/100/2020-12-14T08:10:01Z" done
2020-12-19T22:15:27+01:00: percentage done: 7.73%
2020-12-19T22:15:28+01:00: sync snapshot "vm/100/2020-12-14T09:10:01Z"
2020-12-19T22:15:28+01:00: sync archive qemu-server.conf.blob
2020-12-19T22:15:28+01:00: sync archive drive-scsi0.img.fidx
2020-12-19T22:15:33+01:00: downloaded 83810981 bytes (21.65 MiB/s)
2020-12-19T22:15:33+01:00: got backup log file "client.log.blob"
...
...
...
2020-12-19T23:08:29+01:00: downloaded 11392695 bytes (16.32 MiB/s)
2020-12-19T23:08:31+01:00: got backup log file "client.log.blob"
2020-12-19T23:08:31+01:00: sync snapshot "vm/106/2020-12-18T17:26:30Z" done
2020-12-19T23:08:31+01:00: percentage done: 85.57%
2020-12-19T23:08:31+01:00: re-sync snapshot "vm/121/2020-11-11T06:17:33Z"
2020-12-19T23:08:31+01:00: re-sync snapshot "vm/121/2020-11-11T06:17:33Z" done
2020-12-19T23:08:31+01:00: percentage done: 85.71%
2020-12-19T23:08:31+01:00: sync job 'local:pbs-backup:sandisk:s-f0eb4571-25bf' end
2020-12-19T23:08:31+01:00: TASK OK
 
yes. basically the client (=sync in this case) is done, closes the connection, and the server only sees "connection is closed" and logs the warning.
 
Is it still relevant that error in PBS Tasks after "Sync Job" on the same machine can be ignored ?
Code:
TASK ERROR: connection error: Transport endpoint is not connected (os error 107)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!