Backups to External SSD Keep Failing Randomly

[i-try-pxmx]

Sometimes they can complete successfully, sometimes they fail. Searching these forums and other sources for the error message in question, it seems like the common advice is that it's a problem with the drive, the controller, or the connector... but I can't see how that would be possible in my case given what I've tried.

My setup:

• I've got PVE and PBS running on the same host.
• I backup to a *brand new* 4TB Crucial mx500 SSD in a Sabrent external (unpowered) USB3 enclosure

Things I would add:

• My drive passes SMART
• I've tried a different USB port on my machine; no dice
• I've tried swapping out my Sabrent enclosure for a new identical one to rule out a problem there

I'm at a loss for where to look next! Logs are below; any help would be much appreciated!

These are the logs from my PVE backup job:

Spoiler: PVE logs

INFO: starting new backup job: vzdump 103 101 102 100 --mailnotification failure --mode snapshot --prune-backups 'keep-last=15' --notes-template '{{guestname}}' --node bran --mailto myemail --storage backup --all 0

INFO: Starting Backup of VM 100 (lxc)

INFO: Backup started at 2024-01-05 11:28:42

INFO: status = running

INFO: CT Name: ingress

INFO: including mount point rootfs ('/') in backup

INFO: backup mode: snapshot

INFO: ionice priority: 7

INFO: create storage snapshot 'vzdump'

INFO: creating Proxmox Backup Server archive 'ct/100/2024-01-05T16:28:42Z'

INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp12539_100/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 100 --backup-time 1704472122 --repository backup@pbs@192.168.50.2:bran-backup

INFO: Starting backup: ct/100/2024-01-05T16:28:42Z

INFO: Client name: bran

INFO: Starting backup protocol: Fri Jan  5 11:28:42 2024

INFO: No previous manifest available.

INFO: Upload config file '/var/tmp/vzdumptmp12539_100/etc/vzdump/pct.conf' to 'backup@pbs@192.168.50.2:8007:bran-backup' as pct.conf.blob

INFO: Upload directory '/mnt/vzsnap0' to 'backup@pbs@192.168.50.2:8007:bran-backup' as root.pxar.didx

INFO: root.pxar: had to backup 3.142 GiB of 3.154 GiB (compressed 1.169 GiB) in 28.45s

INFO: root.pxar: average backup speed: 113.074 MiB/s

INFO: root.pxar: backup was done incrementally, reused 12.17 MiB (0.4%)

INFO: Uploaded backup catalog (1.737 MiB)

INFO: Duration: 28.71s

INFO: End Time: Fri Jan  5 11:29:11 2024

INFO: adding notes to backup

INFO: prune older backups with retention: keep-last=15

INFO: running 'proxmox-backup-client prune' for 'ct/100'

INFO: pruned 0 backup(s)

INFO: cleanup temporary 'vzdump' snapshot

INFO: Finished Backup of VM 100 (00:00:29)

INFO: Backup finished at 2024-01-05 11:29:11

INFO: Starting Backup of VM 101 (lxc)

INFO: Backup started at 2024-01-05 11:29:11

INFO: status = running

INFO: CT Name: docker-host

INFO: including mount point rootfs ('/') in backup

INFO: backup mode: snapshot

INFO: ionice priority: 7

INFO: create storage snapshot 'vzdump'

INFO: creating Proxmox Backup Server archive 'ct/101/2024-01-05T16:29:11Z'

INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp12539_101/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 101 --backup-time 1704472151 --repository backup@pbs@192.168.50.2:bran-backup

INFO: Starting backup: ct/101/2024-01-05T16:29:11Z

INFO: Client name: bran

INFO: Starting backup protocol: Fri Jan  5 11:29:12 2024

INFO: No previous manifest available.

INFO: Upload config file '/var/tmp/vzdumptmp12539_101/etc/vzdump/pct.conf' to 'backup@pbs@192.168.50.2:8007:bran-backup' as pct.conf.blob

INFO: Upload directory '/mnt/vzsnap0' to 'backup@pbs@192.168.50.2:8007:bran-backup' as root.pxar.didx

INFO: root.pxar: had to backup 2.391 GiB of 2.391 GiB (compressed 927.889 MiB) in 19.43s

INFO: root.pxar: average backup speed: 126.033 MiB/s

INFO: Uploaded backup catalog (807.977 KiB)

INFO: Duration: 19.62s

INFO: End Time: Fri Jan  5 11:29:31 2024

INFO: adding notes to backup

INFO: prune older backups with retention: keep-last=15

INFO: running 'proxmox-backup-client prune' for 'ct/101'

INFO: pruned 0 backup(s)

INFO: cleanup temporary 'vzdump' snapshot

INFO: Finished Backup of VM 101 (00:00:21)

INFO: Backup finished at 2024-01-05 11:29:32

INFO: Starting Backup of VM 102 (lxc)

INFO: Backup started at 2024-01-05 11:29:32

INFO: status = running

INFO: CT Name: lightning-node

INFO: including mount point rootfs ('/') in backup

INFO: backup mode: snapshot

INFO: ionice priority: 7

INFO: create storage snapshot 'vzdump'

INFO: creating Proxmox Backup Server archive 'ct/102/2024-01-05T16:29:32Z'

INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp12539_102/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 102 --backup-time 1704472172 --repository backup@pbs@192.168.50.2:bran-backup

INFO: Starting backup: ct/102/2024-01-05T16:29:32Z

INFO: Client name: bran

INFO: Starting backup protocol: Fri Jan  5 11:29:32 2024

INFO: No previous manifest available.

INFO: Upload config file '/var/tmp/vzdumptmp12539_102/etc/vzdump/pct.conf' to 'backup@pbs@192.168.50.2:8007:bran-backup' as pct.conf.blob

INFO: Upload directory '/mnt/vzsnap0' to 'backup@pbs@192.168.50.2:8007:bran-backup' as root.pxar.didx

INFO: catalog upload error - channel closed

INFO: Error: inserting chunk on store 'bran-backup' failed for d962d72de10e4671e2f8706da33b4caff57219164aedd753b5d0dc8d2f942ce3 - mkstemp "/mnt/datastore/backup/.chunks/d962/d962d72de10e4671e2f8706da33b4caff57219164aedd753b5d0dc8d2f942ce3.tmp_XXXXXX" failed: EIO: I/O error

INFO: cleanup temporary 'vzdump' snapshot

ERROR: Backup of VM 102 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp12539_102/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 102 --backup-time 1704472172 --repository backup@pbs@192.168.50.2:bran-backup' failed: exit code 255

INFO: Failed at 2024-01-05 13:10:33

INFO: Starting Backup of VM 103 (qemu)

INFO: Backup started at 2024-01-05 13:10:33

INFO: status = running

INFO: VM Name: home-assistant

INFO: include disk 'scsi0' 'nvme-zfs:vm-103-disk-1' 32G

INFO: include disk 'efidisk0' 'nvme-zfs:vm-103-disk-0' 4M

INFO: backup mode: snapshot

INFO: ionice priority: 7

INFO: creating Proxmox Backup Server archive 'vm/103/2024-01-05T18:10:33Z'

INFO: issuing guest-agent 'fs-freeze' command

INFO: issuing guest-agent 'fs-thaw' command

ERROR: VM 103 qmp command 'backup' failed - backup connect failed: command error: namespace not found

INFO: aborting backup job

INFO: resuming VM again

ERROR: Backup of VM 103 failed - VM 103 qmp command 'backup' failed - backup connect failed: command error: namespace not found

INFO: Failed at 2024-01-05 13:10:34

INFO: Backup job finished with errors

INFO: notified via target `<myemail>`

TASK ERROR: job errors

And a corresponding dump from one of the backup jobs in PBS (trimmed down to the last bit where the errors start because the log is massive):

Spoiler: PBS logs

2024-01-05T13:04:12-05:00: POST /dynamic_chunk

2024-01-05T13:04:12-05:00: POST /dynamic_chunk

2024-01-05T13:04:12-05:00: POST /dynamic_chunk

2024-01-05T13:04:12-05:00: POST /dynamic_chunk

2024-01-05T13:04:12-05:00: POST /dynamic_chunk

2024-01-05T13:04:12-05:00: POST /dynamic_chunk

2024-01-05T13:04:12-05:00: POST /dynamic_chunk

2024-01-05T13:04:12-05:00: POST /dynamic_chunk

2024-01-05T13:04:12-05:00: POST /dynamic_chunk

2024-01-05T13:04:12-05:00: POST /dynamic_chunk

2024-01-05T13:10:32-05:00: POST /dynamic_chunk: 400 Bad Request: inserting chunk on store 'bran-backup' failed for 8125d5d73a87b1ff5e96bcf5aa6889c3392b11c2c4ef603338171eb74e125a92 - mkstemp "/mnt/datastore/backup/.chunks/8125/8125d5d73a87b1ff5e96bcf5aa6889c3392b11c2c4ef603338171eb74e125a92.tmp_XXXXXX" failed: EIO: I/O error

2024-01-05T13:10:32-05:00: POST /dynamic_chunk: 400 Bad Request: inserting chunk on store 'bran-backup' failed for a14090f2a439dfe294b6d15981a796059d9d717468d1cc0aa28c6415477d6005 - mkstemp "/mnt/datastore/backup/.chunks/a140/a14090f2a439dfe294b6d15981a796059d9d717468d1cc0aa28c6415477d6005.tmp_XXXXXX" failed: EIO: I/O error

2024-01-05T13:10:32-05:00: POST /dynamic_chunk: 400 Bad Request: inserting chunk on store 'bran-backup' failed for dfad5de55baa50edfb5de5dbb964106db6526d3cf0e6fc49d5465b1f5a5eedf7 - mkstemp "/mnt/datastore/backup/.chunks/dfad/dfad5de55baa50edfb5de5dbb964106db6526d3cf0e6fc49d5465b1f5a5eedf7.tmp_XXXXXX" failed: EIO: I/O error

2024-01-05T13:10:32-05:00: POST /dynamic_chunk: 400 Bad Request: inserting chunk on store 'bran-backup' failed for ce259d0ea495cb509d1b98713887b86c79a39615d9bc6355994d34fd21e7b039 - mkstemp "/mnt/datastore/backup/.chunks/ce25/ce259d0ea495cb509d1b98713887b86c79a39615d9bc6355994d34fd21e7b039.tmp_XXXXXX" failed: EIO: I/O error

2024-01-05T13:10:32-05:00: POST /dynamic_chunk: 400 Bad Request: inserting chunk on store 'bran-backup' failed for 48f73a63abd12a1e052e7057ef72d8f58ddb98f674d4bdd7ea1e8991b34f2cd2 - mkstemp "/mnt/datastore/backup/.chunks/48f7/48f73a63abd12a1e052e7057ef72d8f58ddb98f674d4bdd7ea1e8991b34f2cd2.tmp_XXXXXX" failed: EIO: I/O error

2024-01-05T13:10:32-05:00: POST /dynamic_chunk: 400 Bad Request: inserting chunk on store 'bran-backup' failed for d962d72de10e4671e2f8706da33b4caff57219164aedd753b5d0dc8d2f942ce3 - mkstemp "/mnt/datastore/backup/.chunks/d962/d962d72de10e4671e2f8706da33b4caff57219164aedd753b5d0dc8d2f942ce3.tmp_XXXXXX" failed: EIO: I/O error

2024-01-05T13:10:32-05:00: POST /dynamic_chunk: 400 Bad Request: inserting chunk on store 'bran-backup' failed for fe1319c2b56db840e3dfb42ccabc33c8b438e78815a243ae55a5d710872f92de - mkstemp "/mnt/datastore/backup/.chunks/fe13/fe1319c2b56db840e3dfb42ccabc33c8b438e78815a243ae55a5d710872f92de.tmp_XXXXXX" failed: EIO: I/O error

2024-01-05T13:10:32-05:00: POST /dynamic_chunk: 400 Bad Request: inserting chunk on store 'bran-backup' failed for 66a18f477aab2a5578be917ecc6221f15364b38de04ebbd006156d331ae91168 - mkstemp "/mnt/datastore/backup/.chunks/66a1/66a18f477aab2a5578be917ecc6221f15364b38de04ebbd006156d331ae91168.tmp_XXXXXX" failed: EIO: I/O error

2024-01-05T13:10:32-05:00: POST /dynamic_chunk: 400 Bad Request: inserting chunk on store 'bran-backup' failed for 51f45a7b97b08e200b861abcced0e1c993c61ec7401f0d3072630b87968ab2fa - mkstemp "/mnt/datastore/backup/.chunks/51f4/51f45a7b97b08e200b861abcced0e1c993c61ec7401f0d3072630b87968ab2fa.tmp_XXXXXX" failed: EIO: I/O error

2024-01-05T13:10:32-05:00: POST /dynamic_chunk: 400 Bad Request: inserting chunk on store 'bran-backup' failed for 07163c27eb9136c7b483509a45c886c050b4dee398fb0006f768a7abb022ac94 - mkstemp "/mnt/datastore/backup/.chunks/0716/07163c27eb9136c7b483509a45c886c050b4dee398fb0006f768a7abb022ac94.tmp_XXXXXX" failed: EIO: I/O error

2024-01-05T13:10:32-05:00: POST /dynamic_chunk: 400 Bad Request: inserting chunk on store 'bran-backup' failed for 6000ec68a85c3e9b96c3e3901035025d6938bda7ff4adb05ec65ae5d4f53a94e - mkstemp "/mnt/datastore/backup/.chunks/6000/6000ec68a85c3e9b96c3e3901035025d6938bda7ff4adb05ec65ae5d4f53a94e.tmp_XXXXXX" failed: EIO: I/O error

2024-01-05T13:10:32-05:00: POST /dynamic_chunk: 400 Bad Request: inserting chunk on store 'bran-backup' failed for 4d6b17e880e5d03cd8d4122740d01dc568500dc3d1aac6c9046d15fa6f5aa236 - mkstemp "/mnt/datastore/backup/.chunks/4d6b/4d6b17e880e5d03cd8d4122740d01dc568500dc3d1aac6c9046d15fa6f5aa236.tmp_XXXXXX" failed: EIO: I/O error

2024-01-05T13:10:32-05:00: POST /dynamic_chunk: 400 Bad Request: inserting chunk on store 'bran-backup' failed for 679920f50f581b1676829b404fd33a991bf26260d42b707e9a6a8ff9999c909e - mkstemp "/mnt/datastore/backup/.chunks/6799/679920f50f581b1676829b404fd33a991bf26260d42b707e9a6a8ff9999c909e.tmp_XXXXXX" failed: EIO: I/O error

2024-01-05T13:10:32-05:00: POST /dynamic_chunk: 400 Bad Request: inserting chunk on store 'bran-backup' failed for b4941541288684e3ed831f93a96eb98d618ecc1aaf1fb34a941bd6bd3e7a89b3 - mkstemp "/mnt/datastore/backup/.chunks/b494/b4941541288684e3ed831f93a96eb98d618ecc1aaf1fb34a941bd6bd3e7a89b3.tmp_XXXXXX" failed: EIO: I/O error

2024-01-05T13:10:32-05:00: POST /dynamic_chunk: 400 Bad Request: inserting chunk on store 'bran-backup' failed for e20bd4f3fb8265d596030aee417f11723f15ac6dc34b36a7f43fd129c5954e0a - mkstemp "/mnt/datastore/backup/.chunks/e20b/e20bd4f3fb8265d596030aee417f11723f15ac6dc34b36a7f43fd129c5954e0a.tmp_XXXXXX" failed: EIO: I/O error

2024-01-05T13:10:32-05:00: POST /dynamic_chunk: 400 Bad Request: inserting chunk on store 'bran-backup' failed for fe4f929f37f248a3dab3a5f0b962ecd89f3a3a7a0b49559cb9b35339fe8570ae - mkstemp "/mnt/datastore/backup/.chunks/fe4f/fe4f929f37f248a3dab3a5f0b962ecd89f3a3a7a0b49559cb9b35339fe8570ae.tmp_XXXXXX" failed: EIO: I/O error

2024-01-05T13:10:32-05:00: POST /dynamic_chunk: 400 Bad Request: inserting chunk on store 'bran-backup' failed for ddfb726fc0f79047d9f3f274b097f02fa20011569387cf4a57e573d8ca411a1f - mkstemp "/mnt/datastore/backup/.chunks/ddfb/ddfb726fc0f79047d9f3f274b097f02fa20011569387cf4a57e573d8ca411a1f.tmp_XXXXXX" failed: EIO: I/O error

2024-01-05T13:10:32-05:00: POST /dynamic_chunk: 400 Bad Request: inserting chunk on store 'bran-backup' failed for f4a0b9204e830299f8bc1fcb04f3e795547e2889eca96453973e22839af6cbcc - mkstemp "/mnt/datastore/backup/.chunks/f4a0/f4a0b9204e830299f8bc1fcb04f3e795547e2889eca96453973e22839af6cbcc.tmp_XXXXXX" failed: EIO: I/O error

2024-01-05T13:10:32-05:00: POST /dynamic_chunk: 400 Bad Request: inserting chunk on store 'bran-backup' failed for c29448fdd907843f30b2375d505ad98994ed70e8b38caf656b765c4880557fef - mkstemp "/mnt/datastore/backup/.chunks/c294/c29448fdd907843f30b2375d505ad98994ed70e8b38caf656b765c4880557fef.tmp_XXXXXX" failed: EIO: I/O error

2024-01-05T13:10:32-05:00: POST /dynamic_chunk: 400 Bad Request: inserting chunk on store 'bran-backup' failed for 2373fa108515b30657d2fe5110da4b400593de31b167116f7432849b08e411eb - mkstemp "/mnt/datastore/backup/.chunks/2373/2373fa108515b30657d2fe5110da4b400593de31b167116f7432849b08e411eb.tmp_XXXXXX" failed: EIO: I/O error

2024-01-05T13:10:32-05:00: POST /dynamic_chunk: 400 Bad Request: inserting chunk on store 'bran-backup' failed for bc7cdb0d188014f8953075535c179d86e18c7f3298c788a32ea508b08dfd6f80 - mkstemp "/mnt/datastore/backup/.chunks/bc7c/bc7cdb0d188014f8953075535c179d86e18c7f3298c788a32ea508b08dfd6f80.tmp_XXXXXX" failed: EIO: I/O error

2024-01-05T13:10:32-05:00: backup failed: connection error: not connected

2024-01-05T13:10:32-05:00: removing failed backup

2024-01-05T13:10:32-05:00: TASK ERROR: removing backup snapshot "/mnt/datastore/backup/ct/102/2024-01-05T16:29:32Z" failed - Input/output error (os error 5)

 

[DaveInDeed]

The one piece of hardware that does not seem to have been changed is the USB cable
Also, are there any errors in the syslog that might be a clue?
Maybe messages that indicate there are power issues or something similar ?

I have made some tests using an external unpowered USB SSD for datastore without problems (well beside poor performance) so it should work

 

[sb-jw]

ERROR: VM 103 qmp command 'backup' failed - backup connect failed: command error: namespace not found

Does your PBS run permanently and is its disk always available or do you sometimes switch off the VM or disk? Have you set up the appropriate jobs for Verify and GC and are they running without any problems? What is the condition of your PBS?

 

[i-try-pxmx]

Thanks both for your input.

The one piece of hardware that does not seem to have been changed is the USB cable
Also, are there any errors in the syslog that might be a clue?
Maybe messages that indicate there are power issues or something similar ?

I have made some tests using an external unpowered USB SSD for datastore without problems (well beside poor performance) so it should work

I have switched out the USB cable as part of the drive enclosure. Should have mentioned that. Haven't looked at syslogs yet but...

Does your PBS run permanently and is its disk always available or do you sometimes switch off the VM or disk? Have you set up the appropriate jobs for Verify and GC and are they running without any problems? What is the condition of your PBS?

Yes, it's running permanently on my host machine alongside PVE. The disk should never turn off... i think. And verify and GC jobs are running, but this whole setup is brand new so they haven't had much of a chance to do their thing.

HOWEVER... I just nuked the whole backup setup and re-did it. Wiped the disk, destroyed the directory, deleted the mount point and started from scratch. But this time I formatted the external SSD as ext4 instead of xfs. Three automatic backup runs and several manual runs later, no failures. I wonder if this had something to do with xfs?

Anyway, I'll keep an eye out and report back, and if I don't see anything else amiss, I'll mark this thread as solved.

 

[StackIOI]

I'm facing the same issue, from 24 LXCs, only one backup task is failing. I have pretty much the same setup as OP:

• I've got PVE and PBS running on the same host.
• I backup to a *brand new* 4TB Seagate BarraCuda HDD in an external (powered) USB3 enclosure

Here my logs:

Spoiler: BackUp Task Log

INFO: starting new backup job: vzdump 212 --notes-template '{{guestname}}, {{node}}, {{vmid}}' --node tundra --remove 0 --mode stop --storage prxmxbs
INFO: Starting Backup of VM 212 (lxc)
INFO: Backup started at 2024-01-09 10:31:34
INFO: status = running
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: bazarr
INFO: including mount point rootfs ('/') in backup
INFO: stopping virtual guest
INFO: creating Proxmox Backup Server archive 'ct/212/2024-01-09T15:31:34Z'
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp507409_212/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 212 --backup-time 1704814294 --repository user@pbs@192.168.99.51:pbs
INFO: Starting backup: ct/212/2024-01-09T15:31:34Z
INFO: Client name: tundra
INFO: Starting backup protocol: Tue Jan  9 10:31:37 2024
INFO: No previous manifest available.
INFO: Upload config file '/var/tmp/vzdumptmp507409_212/etc/vzdump/pct.conf' to 'user@pbs@192.168.99.51:8007:pbs' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'user@pbs@192.168.99.51:8007:pbs' as root.pxar.didx
INFO: catalog upload error - channel closed
INFO: Error: error at "usr/share/i18n/charmaps/CP1258.gz": Input/output error (os error 5)
INFO: restarting vm
INFO: guest is online again after 17 seconds
ERROR: Backup of VM 212 failed - command '/usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp507409_212/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 212 --backup-time 1704814294 --repository user@pbs@192.168.99.51:pbs' failed: exit code 255
INFO: Failed at 2024-01-09 10:31:51
INFO: Backup job finished with errors
TASK ERROR: job errors

Spoiler: Node SysLog

Jan 09 03:05:41 tundra pvescheduler[340086]: ERROR: Backup of VM 212 failed - command '/usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp340086_212/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 212 --backup-time 1704787523 --repository user@pbs@192.168.99.51:pbs' failed: exit code 255

Spoiler: PBS Log

Jan 09 03:05:39 prxmxpbs proxmox-backup-proxy[179]: successfully added chunk 9215ed90d568f2ae4c4d3142fb177736d0187d44ecb319f89f2533bb8014c6e9 to dynamic index 2 (offset 1127777112, size 4217411)
Jan 09 03:05:40 prxmxpbs proxmox-backup-proxy[179]: backup ended and finish failed: backup ended but finished flag is not set.
Jan 09 03:05:40 prxmxpbs proxmox-backup-proxy[179]: removing unfinished backup
Jan 09 03:05:40 prxmxpbs proxmox-backup-proxy[179]: TASK ERROR: backup ended but finished flag is not set.
Jan 09 03:05:40 prxmxpbs proxmox-backup-[179]: prxmxpbs proxmox-backup-proxy[179]: removing backup snapshot "/media/usb-hdd/pbs/ct/212/2024-01-09T08:05:23Z"

I have looked into this file's timestamp and attributes, but could see anything out of place:
INFO: Error: error at "usr/share/i18n/charmaps/CP1258.gz": Input/output error (os error 5)

Spoiler: CP1258.gz

>/usr/share/i18n/charmaps#stat CP1258.gz
  File: CP1258.gz
  Size: 2728          Blocks: 8          IO Block: 4096   regular file
Device: fd0ah/64778d    Inode: 6765        Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-01-08 12:12:43.492673572 -0500
Modify: 2023-10-02 15:22:57.000000000 -0500
Change: 2024-01-08 13:06:11.644043667 -0500
 Birth: 2023-10-04 06:50:04.977858418 -0500

for comparison
>/usr/share/i18n/charmaps#stat CP1257.gz
  File: CP1257.gz
  Size: 2813          Blocks: 8          IO Block: 4096   regular file
Device: fd0ah/64778d    Inode: 6764        Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-01-08 12:12:43.492673572 -0500
Modify: 2023-10-02 15:22:57.000000000 -0500
Change: 2023-10-04 06:50:05.369860211 -0500
 Birth: 2023-10-04 06:50:04.977858418 -0500

Any help is greatly appreciated!

 

[sb-jw]

But the affected VM or CT is not the PBS itself?

 

[StackIOI]

But the affected VM or CT is not the PBS itself?

Not in my case, it is a different container.

 

[Chris]

INFO: Error: error at "usr/share/i18n/charmaps/CP1258.gz": Input/output error (os error 5)

Hi,
this seems to be an error while reading the file, not the upload/write to the PBS.

• Do you see this error in a reproducible manner on exactly this file?
• Is there any other errors around the time of the backup in the systemd journal?

Please post the journal around that timespan from the host journalctl --since <DATETIME> --until <DATETIME> > journal.txt and the container config pct config <VMID> as well as the storage config cat /etc/pve/storage.cfg. Also, please check the smart values of your disks.

 

[StackIOI]

Hi,
this seems to be an error while reading the file, not the upload/write to the PBS.

• Do you see this error in a reproducible manner on exactly this file?
• Is there any other errors around the time of the backup in the systemd journal?

Please post the journal around that timespan from the host journalctl --since <DATETIME> --until <DATETIME> > journal.txt and the container config pct config <VMID> as well as the storage config cat /etc/pve/storage.cfg. Also, please check the smart values of your disks.

@Chris thank you for the reply. I was able to sort it out by reinstalling locales package, so the input/output error went away, and I was able to backup successfully this LXC.

Thanks again.