Backups fail with qmp command 'backup' failed - backup register image failed: command error: stream error received: stream no longer needed

[cpt_duck_dodgers]

Hello,

Proxmox VE - 8.1.4
Backup server - 3.1-4

Rather new to the Proxmox stack so please let me know if I'm missing any information or if there is something in my config that's not right/supported!

Basically, I have the backupserver configured as a VM on Proxmox VE, my storage for the backup server is a SMB share from my NAS (Truenas) mounted to the backup server VM to use as a datastore.

The backups run fine until my NAS goes temporarily unavailable (I'm having hardware issues at the moment that causes the NIC to intermittently fail and be no longer available on the network which I'm waiting on a replacement part for...I realize this is basically the root of the issue but I hope my problem still stands).

If the backup try's to run when the NAS is offline it fails with the below error, which I would expect the backup to fail of course. My concern/issue is that after the NAS has come back online the jobs continue to fail with the same error (it's scheduled to run nightly).

INFO: starting new backup job: vzdump 103 101 100 102 --notes-template '{{guestname}}' --node pve01 --storage pbs --prune-backups 'keep-last=3' --notification-mode notification-system --mailto chris.k88@hotmail.com --quiet 1 --mode snapshot --mailnotification always
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2024-04-16 21:00:02
INFO: status = running
INFO: VM Name: pve02-pihole02
INFO: include disk 'scsi0' 'local-lvm:vm-100-disk-0' 30G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/100/2024-04-17T01:00:02Z'
INFO: enabling encryption
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 100 qmp command 'backup' failed - backup register image failed: command error: stream error received: stream no longer needed
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 100 failed - VM 100 qmp command 'backup' failed - backup register image failed: command error: stream error received: stream no longer needed
INFO: Failed at 2024-04-16 21:00:03
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2024-04-16 21:00:03
INFO: status = running
INFO: VM Name: pve02-dc02
INFO: include disk 'virtio0' 'local-lvm:vm-101-disk-0' 60G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/101/2024-04-17T01:00:03Z'
INFO: enabling encryption
INFO: skipping guest-agent 'fs-freeze', agent configured but not running?
ERROR: VM 101 qmp command 'backup' failed - backup register image failed: command error: stream error received: stream no longer needed
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 101 failed - VM 101 qmp command 'backup' failed - backup register image failed: command error: stream error received: stream no longer needed
INFO: Failed at 2024-04-16 21:00:07
INFO: Starting Backup of VM 102 (qemu)
INFO: Backup started at 2024-04-16 21:00:07
INFO: status = running
INFO: VM Name: pve02-pfsense02
INFO: include disk 'scsi0' 'local-lvm:vm-102-disk-0' 50G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/102/2024-04-17T01:00:07Z'
INFO: enabling encryption
ERROR: VM 102 qmp command 'backup' failed - backup register image failed: command error: stream error received: stream no longer needed
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 102 failed - VM 102 qmp command 'backup' failed - backup register image failed: command error: stream error received: stream no longer needed
INFO: Failed at 2024-04-16 21:00:07
INFO: Starting Backup of VM 103 (qemu)
INFO: Backup started at 2024-04-16 21:00:07
INFO: status = running
INFO: VM Name: pve02-duoprox02
INFO: include disk 'scsi0' 'local-lvm:vm-103-disk-0' 30G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/103/2024-04-17T01:00:07Z'
INFO: enabling encryption
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 103 qmp command 'backup' failed - backup register image failed: command error: stream error received: stream no longer needed
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 103 failed - VM 103 qmp command 'backup' failed - backup register image failed: command error: stream error received: stream no longer needed
INFO: Failed at 2024-04-16 21:00:08
INFO: Backup job finished with errors
INFO: notified via target `mail-duckworld-ca`
TASK ERROR: job errors

The only other similar post I've found was this one, restarting the backup VM or service does resolve my problem but beyond that I can't find anything else in the logs (I'm probably not looking in the right place)

As I mentioned, obviously the root problem is to fix my NAS but I'm wondering why once the NAS comes back online the jobs continue to fail until the server/service is restarted.

Again, please let me know what other info might be helpful.

EDIT: To add, while in the state where backups fail the share is still mounted and through cli I can navigate through it and create files manually as tests to confirm I can still write to the mounted share.

 

[Chris]

Hi,
this is expected as the datastores are cached and keep a file handle to the chunk store. This file handle is however no longer valid when you loose and reestablish the connection. You would have to set the maintenance mode of the datastore to offline before the connection vanishes. See https://pbs.proxmox.com/docs/maintenance.html#maintenance-mode

 

[cpt_duck_dodgers]

Thanks for the response Chris.

So if I understand, if it goes unavailable for any reason long enough for the cache to expire it won't automagically re enable it's connection? Which as you said is expected behaviour.

While obviously your storage source shouldn't ideally be going unavailable regularly, I find it strange there is on way for it to recover on it's own after an outage and to my understanding, will always require manual intervention after an outage?