Issue with backup - TASK ERROR: connection error: not connected [ Rolling back to PVE 8 and PBS 3]

[Gilberto Ferreira]

Hi there
We have 4 nodes, and one of them, is pve and pbs at same time.
Everything is up to date, running pve 9 e pbs 4.
When I tried to backup a vm, which has a bunch of dockers inside, I got a lot o errors, like that:

2025-10-22T18:31:08-03:00: starting new backup reader datastore 'backup': "/mnt/datastore/backup"
2025-10-22T18:31:08-03:00: protocol upgrade done
2025-10-22T18:31:08-03:00: GET /download
2025-10-22T18:31:08-03:00: download "/mnt/datastore/backup/ns/alta-tech-old/vm/1090/2025-10-21T15:16:17Z/index.json.blob"
2025-10-22T18:31:08-03:00: GET /download
2025-10-22T18:31:08-03:00: download "/mnt/datastore/backup/ns/alta-tech-old/vm/1090/2025-10-21T15:16:17Z/fw.conf.blob"
2025-10-22T18:31:08-03:00: TASK ERROR: connection error: not connected

After a while, the backup task hung in 20% and the vm start to suffer of disk IO error, and than after fsck return to normal.
But this is pretty bad and I never encounter such issue before.
The datastore is NFS but the disk in NFS server are NVME enterprise level.
Disk are all SSD and NVME enterprise level.
VMS running normal and pretty fast.

Anybody else has the same issue?

Thanks.

PS. As a measure of caution, we are rolling back to Proxmox 8 and PBS 3, for now.

 

[Chris]

2025-10-22T18:31:08-03:00: TASK ERROR: connection error: not connected

Since this is the task log from the PBS, this error indicates that the connection to the client is lost. Are you sure there were no transient networking issues?

the backup task hung in 20% and the vm start to suffer of disk IO error

You might want to enable backup fleecing to decouple the VM IO from the upload to the PBS.

 

[Gilberto Ferreira]

Since this is the task log from the PBS, this error indicates that the connection to the client is lost. Are you sure there were no transient networking issues?

We already ask to the people whom manage the infrastructure there, and no issue were found.
Strangely enough, this issue about connection error, occurs in only one VM.
But this VM right now is running in another cluster with PVE 8 and PBS 3, and everything is ok.

You might want to enable backup fleecing to decouple the VM IO from the upload to the PBS.

I'll try that.

Thank you for your reply

 

[Chris]

We already ask to the people whom manage the infrastructure there, and no issue were found.
Strangely enough, this issue about connection error, occurs in only one VM.

Did you also check the task log and systemd journal on the PVE host? Might give some further clue on why the connection was lost.

 

[Gilberto Ferreira]

Another 3 host in other infra, with the same error:

 

[Chris]

Another 3 host in other infra, with the same error:
View attachment 91967

View attachment 91969

But these are not related at all to your initial issue. These are benign error messages being created when the client reader closes it's connection, which the server does not know about and therefore logs it as error. Unfortunately a fix is not so trivial as discussed alongside the patch series which tries to fix these https://lore.proxmox.com/pbs-devel/20250408125839.196668-1-c.ebner@proxmox.com/T/