One VM - Backup failed - catalog upload error - channel closed

O

oops404

New Member
May 27, 2023
26
1
3
Hello all,

one VM can`t get backuped? In the past it worked but now it fails. I have done reboots and updated everything on to the latest but still get this issue if i try to do a snapshot of this VM:

"....
INFO: processed 447.342 GiB in 2h 14m 0s, uploaded 439.171 GiB
INFO: catalog upload error - channel closed
INFO: unclosed encoder dropped
INFO: closed encoder dropped with state
INFO: unfinished encoder state dropped
INFO: finished encoder state with errors
INFO: Error: inserting chunk on store 'usb' failed for 7b6acfbff2a499528838c6f42e91d2fbaaf99c46ec7f6c9ed83cfcb2b6e25e42 - mkstemp "/mnt/datastore/usb/.chunks/7b6a/7b6acfbff2a499528838c6f42e91d2fbaaf99c46ec7f6c9ed83cfcb2b6e25e42.tmp_XXXXXX" failed: ENODATA: No data available
INFO: cleanup temporary 'vzdump' snapshot
ERROR: Backup of VM 100 failed - command '/usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp18845_100/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 100 --backup-time 1741682419 --entries-max 1048576 --repository root@pam@192.168.0.21:usb' failed: exit code 255
INFO: Failed at 2025-03-11 11:54:41
INFO: Backup job finished with errors
TASK ERROR: job errors"

The other VMs get successfully updated and in general everything looks healthy and free space.

Any idea?

Best
 
oops404 said:
Error: inserting chunk on store 'usb' failed for 7b6acfbff2a499528838c6f42e91d2fbaaf99c46ec7f6c9ed83cfcb2b6e25e42 - mkstemp "/mnt/datastore/usb/.chunks/7b6a/7b6acfbff2a499528838c6f42e91d2fbaaf99c46ec7f6c9ed83cfcb2b6e25e42.tmp_XXXXXX" failed: ENODATA: No data available
Click to expand...
Hi,
this seems like the root cause of the error is the failed chunk insert (the other errors are just a side effect of the client being in an unfinished state after backup).

Please check the systemd journal on the PBS for errors, especially regarding disk IO. And check the output of
Code: 
stat /mnt/datastore/usb/.chunks/7b6a/
on the PBS
 
Chris said:
Hi,
this seems like the root cause of the error is the failed chunk insert (the other errors are just a side effect of the client being in an unfinished state after backup).

Please check the systemd journal on the PBS for errors, especially regarding disk IO. And check the output of
Code: 
stat /mnt/datastore/usb/.chunks/7b6a/
on the PBS
Click to expand...

Mmmmh.... Is my hard disk dying slowly??

----

root@pbs:~# stat /mnt/datastore/usb/.chunks/7b6a/
File: /mnt/datastore/usb/.chunks/7b6a/
Size: 16 Blocks: 97 IO Block: 16384 directory
Device: 0,37 Inode: 31603 Links: 2
Access: (0750/drwxr-x---) Uid: ( 34/ backup) Gid: ( 34/ backup)
Access: 2024-08-15 21:19:04.550054266 +0200
Modify: 2025-02-21 01:47:52.072569305 +0100
Change: 2025-02-21 01:47:52.072569305 +0100
Birth: 2024-08-15 21:19:04.550054266 +0200

---

Mar 11 11:54:10 pbs proxmox-backup-proxy[720]: successfully added chunk ae5c537bbf02f8cb8cb32db5650f9a1e876966cfa795dd6f14d6faa08dcb2a6c to dynamic index 2 (offset 480022322273, size 1750346)
Mar 11 11:54:10 pbs proxmox-backup-proxy[720]: successfully added chunk 94c6c6e2f37011adad4427b0b8f576191046b3e61a4a67b8028d135dc440cd31 to dynamic index 2 (offset 480024072619, size 4281672)
Mar 11 11:54:10 pbs proxmox-backup-proxy[720]: successfully added chunk dcfdbab9525c735f309a9a74b1364748037b932ff937989101105d5bd61c95e3 to dynamic index 2 (offset 480028354291, size 4452291)
Mar 11 11:54:10 pbs proxmox-backup-proxy[720]: upload_chunk done: 1262176 bytes, cc3d8addca36fba1d2513104b9f6a629477bf614c9f8c0dba1eb3cb168888ba2
Mar 11 11:54:10 pbs proxmox-backup-proxy[720]: upload_chunk done: 2846260 bytes, d42c9c5254346ca9886c5bfe4f619db2d589f8c8639064dc07009fca240dddc2
Mar 11 11:54:10 pbs proxmox-backup-proxy[720]: upload_chunk done: 1361744 bytes, 5f416017bd311accf3a867d59b2b7882afeb5b0d2e0a4cb76f053a8db4d063cc
Mar 11 11:54:14 pbs kernel: sd 2:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=3s
Mar 11 11:54:14 pbs kernel: sd 2:0:0:0: [sdb] tag#0 Sense Key : Medium Error [current]
Mar 11 11:54:14 pbs kernel: sd 2:0:0:0: [sdb] tag#0 Add. Sense: Unrecovered read error
Mar 11 11:54:14 pbs kernel: sd 2:0:0:0: [sdb] tag#0 CDB: Read(16) 88 00 00 00 00 01 c4 ca 2f 50 00 00 00 10 00 00
Mar 11 11:54:14 pbs kernel: critical medium error, dev sdb, sector 7596552016 op 0x0:(READ) flags 0x100 phys_seg 2 prio class 0
Mar 11 11:54:14 pbs kernel: zio pool=usb vdev=/dev/disk/by-id/wwn-0x50014ee26a4137a8-part1 error=61 type=1 offset=3889433583616 size=4096 flags=1572992
Mar 11 11:54:14 pbs kernel: zio pool=usb vdev=/dev/disk/by-id/wwn-0x50014ee26a4137a8-part1 error=61 type=1 offset=3889433587712 size=4096 flags=1572992
Mar 11 11:54:14 pbs zed[1645]: eid=15 class=io pool='usb' vdev=wwn-0x50014ee26a4137a8-part1 size=2560 offset=3889433583616 priority=0 err=61 flags=0x180080 delay=3850ms bookmark=54:31603:0:1
Mar 11 11:54:14 pbs zed[1644]: eid=14 class=io pool='usb' vdev=wwn-0x50014ee26a4137a8-part1 size=2560 offset=3889433587712 priority=0 err=61 flags=0x180080 delay=4057ms bookmark=54:31603:0:1
Mar 11 11:54:17 pbs kernel: sd 2:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=3s
Mar 11 11:54:17 pbs kernel: sd 2:0:0:0: [sdb] tag#0 Sense Key : Medium Error [current]
Mar 11 11:54:17 pbs kernel: sd 2:0:0:0: [sdb] tag#0 Add. Sense: Unrecovered read error
Mar 11 11:54:17 pbs kernel: sd 2:0:0:0: [sdb] tag#0 CDB: Read(16) 88 00 00 00 00 01 c6 03 bf 00 00 00 00 08 00 00
Mar 11 11:54:17 pbs kernel: critical medium error, dev sdb, sector 7617101568 op 0x0:(READ) flags 0x100 phys_seg 1 prio class 0
Mar 11 11:54:17 pbs kernel: zio pool=usb vdev=/dev/disk/by-id/wwn-0x50014ee26a4137a8-part1 error=61 type=1 offset=3899954954240 size=4096 flags=1572992
Mar 11 11:54:18 pbs zed[1648]: eid=16 class=io pool='usb' vdev=wwn-0x50014ee26a4137a8-part1 size=2560 offset=3899954954240 priority=0 err=61 flags=0x180080 delay=3098ms bookmark=54:31603:0:1
Mar 11 11:54:21 pbs kernel: sd 2:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=3s
Mar 11 11:54:21 pbs kernel: sd 2:0:0:0: [sdb] tag#0 Sense Key : Medium Error [current]
Mar 11 11:54:21 pbs kernel: sd 2:0:0:0: [sdb] tag#0 Add. Sense: Unrecovered read error
Mar 11 11:54:21 pbs kernel: sd 2:0:0:0: [sdb] tag#0 CDB: Read(16) 88 00 00 00 00 01 c4 ca 2f 58 00 00 00 08 00 00
Mar 11 11:54:21 pbs kernel: critical medium error, dev sdb, sector 7596552024 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Mar 11 11:54:21 pbs kernel: zio pool=usb vdev=/dev/disk/by-id/wwn-0x50014ee26a4137a8-part1 error=61 type=1 offset=3889433587712 size=4096 flags=1605761
Mar 11 11:54:24 pbs kernel: sd 2:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=3s
Mar 11 11:54:24 pbs kernel: sd 2:0:0:0: [sdb] tag#0 Sense Key : Medium Error [current]
Mar 11 11:54:24 pbs kernel: sd 2:0:0:0: [sdb] tag#0 Add. Sense: Unrecovered read error
Mar 11 11:54:24 pbs kernel: sd 2:0:0:0: [sdb] tag#0 CDB: Read(16) 88 00 00 00 00 01 c6 03 bf 00 00 00 00 08 00 00
Mar 11 11:54:24 pbs kernel: critical medium error, dev sdb, sector 7617101568 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Mar 11 11:54:24 pbs kernel: zio pool=usb vdev=/dev/disk/by-id/wwn-0x50014ee26a4137a8-part1 error=61 type=1 offset=3899954954240 size=4096 flags=1605761
Mar 11 11:54:24 pbs zed[1653]: eid=17 class=data pool='usb' priority=0 err=61 flags=0x800080 bookmark=54:31603:0:1
Mar 11 11:54:27 pbs kernel: sd 2:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=3s
Mar 11 11:54:27 pbs kernel: sd 2:0:0:0: [sdb] tag#0 Sense Key : Medium Error [current]
Mar 11 11:54:27 pbs kernel: sd 2:0:0:0: [sdb] tag#0 Add. Sense: Unrecovered read error
Mar 11 11:54:27 pbs kernel: sd 2:0:0:0: [sdb] tag#0 CDB: Read(16) 88 00 00 00 00 01 c4 ca 2f 48 00 00 00 18 00 00
Mar 11 11:54:27 pbs kernel: critical medium error, dev sdb, sector 7596552008 op 0x0:(READ) flags 0x100 phys_seg 3 prio class 0
Mar 11 11:54:27 pbs kernel: zio pool=usb vdev=/dev/disk/by-id/wwn-0x50014ee26a4137a8-part1 error=61 type=1 offset=3889433579520 size=4096 flags=1572992
Mar 11 11:54:27 pbs kernel: zio pool=usb vdev=/dev/disk/by-id/wwn-0x50014ee26a4137a8-part1 error=61 type=1 offset=3889433583616 size=4096 flags=1572992
Mar 11 11:54:27 pbs kernel: zio pool=usb vdev=/dev/disk/by-id/wwn-0x50014ee26a4137a8-part1 error=61 type=1 offset=3889433587712 size=4096 flags=1572992
Mar 11 11:54:27 pbs zed[1657]: eid=18 class=io pool='usb' vdev=wwn-0x50014ee26a4137a8-part1 size=3072 offset=3889433579520 priority=0 err=61 flags=0x180080 delay=3013ms bookmark=54:31603:0:1
Mar 11 11:54:30 pbs kernel: sd 2:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=3s
Mar 11 11:54:30 pbs kernel: sd 2:0:0:0: [sdb] tag#0 Sense Key : Medium Error [current]
Mar 11 11:54:30 pbs kernel: sd 2:0:0:0: [sdb] tag#0 Add. Sense: Unrecovered read error
Mar 11 11:54:30 pbs kernel: sd 2:0:0:0: [sdb] tag#0 CDB: Read(16) 88 00 00 00 00 01 c6 03 be f0 00 00 00 18 00 00
Mar 11 11:54:30 pbs kernel: critical medium error, dev sdb, sector 7617101552 op 0x0:(READ) flags 0x100 phys_seg 3 prio class 0
Mar 11 11:54:30 pbs kernel: zio pool=usb vdev=/dev/disk/by-id/wwn-0x50014ee26a4137a8-part1 error=61 type=1 offset=3899954946048 size=4096 flags=1572992
Mar 11 11:54:30 pbs kernel: zio pool=usb vdev=/dev/disk/by-id/wwn-0x50014ee26a4137a8-part1 error=61 type=1 offset=3899954950144 size=4096 flags=1572992
Mar 11 11:54:30 pbs kernel: zio pool=usb vdev=/dev/disk/by-id/wwn-0x50014ee26a4137a8-part1 error=61 type=1 offset=3899954954240 size=4096 flags=1572992
Mar 11 11:54:30 pbs zed[1662]: eid=19 class=io pool='usb' vdev=wwn-0x50014ee26a4137a8-part1 size=2560 offset=3899954950144 priority=0 err=61 flags=0x180080 delay=3198ms bookmark=54:31603:0:1
Mar 11 11:54:30 pbs zed[1664]: eid=20 class=io pool='usb' vdev=wwn-0x50014ee26a4137a8-part1 size=3072 offset=3899954946048 priority=0 err=61 flags=0x180080 delay=3071ms bookmark=54:31603:0:1
Mar 11 11:54:33 pbs kernel: sd 2:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=2s
Mar 11 11:54:33 pbs kernel: sd 2:0:0:0: [sdb] tag#0 Sense Key : Medium Error [current]
Mar 11 11:54:33 pbs kernel: sd 2:0:0:0: [sdb] tag#0 Add. Sense: Unrecovered read error
Mar 11 11:54:33 pbs kernel: sd 2:0:0:0: [sdb] tag#0 CDB: Read(16) 88 00 00 00 00 01 c4 ca 2f 58 00 00 00 08 00 00
Mar 11 11:54:33 pbs kernel: critical medium error, dev sdb, sector 7596552024 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Mar 11 11:54:33 pbs kernel: zio pool=usb vdev=/dev/disk/by-id/wwn-0x50014ee26a4137a8-part1 error=61 type=1 offset=3889433587712 size=4096 flags=1605761
Mar 11 11:54:37 pbs kernel: sd 2:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=3s

1741710844412.png
 
I think its gonna be really hard for people in this forum to tell the difference between the fact that you've got a USB disk mounted and actual hard drive errors. (BTW ... I did this for fun in my lab with an ancient spinner attached w/usb cradle. I've seen these errors before.)

It's USB, so you aren't getting SMART data, right? So much for checking SMART.

I don't know, maybe take it to another machine and run some scans on the disk.
 
Last edited:
You must log in or register to reply here.
Top Bottom