Backup Errors

[AWP Technologies]

I have been using proxmox and proxmox backup server for a year or so now. In the last 3 days i have had more backup errors then ever.
I came to check backups and 3 of my vms had these errors.

2024-02-08T21:15:35-05:00: starting new backup on datastore 'PBS-1' from ::ffff:10.5.1.5: "vm/108/2024-02-09T02:15:21Z"
2024-02-08T21:15:35-05:00: download 'index.json.blob' from previous backup.
2024-02-08T21:15:35-05:00: register chunks in 'drive-efidisk0.img.fidx' from previous backup.
2024-02-08T21:15:35-05:00: download 'drive-efidisk0.img.fidx' from previous backup.
2024-02-08T21:15:35-05:00: created new fixed index 1 ("vm/108/2024-02-09T02:15:21Z/drive-efidisk0.img.fidx")
2024-02-08T21:15:35-05:00: register chunks in 'drive-scsi0.img.fidx' from previous backup.
2024-02-08T21:15:35-05:00: download 'drive-scsi0.img.fidx' from previous backup.
2024-02-08T21:15:35-05:00: created new fixed index 2 ("vm/108/2024-02-09T02:15:21Z/drive-scsi0.img.fidx")
2024-02-08T21:15:35-05:00: add blob "/mnt/PBS-Backup/vm/108/2024-02-09T02:15:21Z/qemu-server.conf.blob" (577 bytes, comp: 577)
2024-02-08T21:16:59-05:00: backup ended and finish failed: backup ended but finished flag is not set.
2024-02-08T21:16:59-05:00: removing unfinished backup
2024-02-08T21:16:59-05:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: stream closed because of a broken pipe
2024-02-08T21:16:59-05:00: TASK ERROR: removing backup snapshot "/mnt/PBS-Backup/vm/108/2024-02-09T02:15:21Z" failed - Directory not empty (os error 39)

all 15 of my other vms did just fine both before and after the three.

Please help as i don't want to go back to the days where i have to watch backups as they happen to make sure they finish.

Also when logging in to proxmox backup server the 3 backups that failed still have a spinning loading symbol.

 

[AWP Technologies]

I am also on the 6.5 kernel and have used apt-mark hold to stop proxmox from trying to downgrade to 6.2....

Im not crazy right the 6.5 kernel is much newer correct?

 

[fabian]

how does the backup task log on the PVE side look like? what about the system logs?

 

[AWP Technologies]

here is the dump file

VM 104 105 and 108 are the ones with errors.

 

[fabian]

that sounds a bit like the source storage might have trouble? what is "VMq"? could you maybe check the system logs on the PVE side, and, if applicable, smart values for the underlying disks?