Backups Failing Input/output error (os error 5)

[Wasca]

Hi Guys

I'm getting Input/output error (os error 5) when trying to backup a VM using Proxmox backup and my NAS drive.

When I do a stat on the folder in question I get,

stat /mnt/backup/.chunks/a103
  File: /mnt/backup/.chunks/a103
  Size: 12288         Blocks: 32         IO Block: 4096   directory
Device: 0,42    Inode: 333774947   Links: 2
Access: (0777/drwxrwxrwx)  Uid: (   34/  backup)   Gid: (   34/  backup)
Access: 2024-07-24 11:35:17.589336693 +1000
Modify: 2025-01-22 20:10:37.529916391 +1000
Change: 2025-01-24 17:34:23.087445393 +1000
 Birth: -

If I do an ls -la on the folder it starts spitting out this...

/mnt/backup/.chunks/a103# ls -la
ls: cannot access 'a10371d38824a05c379bb26aa2178532592c5a61234876766942d9443dc4e8c9': Input/output error
ls: cannot access 'a103c36c1790491f3a40328e3b9c040a093cafbf8472369a9fb9379cd6d9397f': Input/output error
ls: cannot access 'a1037f80abca905dd91c6651fcd9e107e469de6ba3b1fb6dd5fab4efb0e380e0': Input/output error
ls: cannot access 'a103ac4477cb62e4e9463d96ad90c68272445c9e53e9ec6d1a0cab141d805e9f': Input/output error
ls: cannot access 'a10378b32d59cb7f84e1c15c00433954a6c7d232a6fa2ee29ab2a81ac88d4218': Input/output error
ls: cannot access 'a10353b89afebdf423f30cb42eac7923da6d6828286b7b403a629b66b1cbcb2e': Input/output error
ls: cannot access 'a1032cd3ad0180f532d52e6d5f0fdab114c1e69fedf2e5333b500f629a44d166': Input/output error
ls: cannot access 'a1036f68d3ad55b7d28e4d86d16c5e070cf128367682be7959fab9dbeca2bc19': Input/output error
ls: cannot access 'a103f463130e4d7390a6cb061d2d87ff349d5f0c5a17a0c825eb25ad8851868e': Input/output error
total 62660
drwxrwxrwx+ 2 backup backup   12288 Jan 22 20:10 .
drwxrwxrwx+ 1 backup backup 1073152 Jan 24 17:34 ..
-rw-r--r--+ 1 backup backup 1061856 Jan 12 22:50 a10318929dd4967720e79981dc9e081b1b7dc7328b30acbc4ad7a141faee8e9a
-rw-r--r--+ 1 backup backup  742731 Dec  8 22:11 a1031c4f5d63baf948910484d9afca9de62a10643703816535c5c30b69893e3f
-rw-r--r--+ 1 backup backup 1069581 Dec 19 23:09 a10326c0a3c3aa6339ac9129ca243d9728fec00ad77cada5b4a21d44ed277f37
-rw-r--r--+ 1 backup backup 1029736 Oct 27 03:34 a103273cbca86ca0e07a96a20b395c0e551f9c40afd53116e48d1c72a7874f86
-rw-r--r--+ 1 backup backup 1014758 Jan 20 23:06 a1032857dd58bc0c47a5a256b1ec1201c8070a090025f3ca96bec5c754f0c6cc
-rw-r--r--+ 1 backup backup  690415 Dec  1 22:33 a10329d3584271b2d8d9aa72d1db343a789b6fd5fe91008f3623bead514171ac
??????????? ? ?      ?            ?            ? a1032cd3ad0180f532d52e6d5f0fdab114c1e69fedf2e5333b500f629a44d166
-rw-r--r--+ 1 backup backup 2336422 Dec 18 19:01 a10332f86d749218722293495d8737ad7e6e32b1cfdb003ca39110af0895f33b
-rw-r--r--+ 1 backup backup   29524 Jan 22 20:10 a10338dc40b87881195498f5fda20cdf55d264c4c85148e6d2f6e6616730397d
-rwxrwxrwx+ 1 backup backup 4194316 Jul 24  2024 a10338fe33698e89edd74b438f8a3f500a51607bf189cd628dec4edc7dd0fd10
-rw-r--r--+ 1 backup backup  579447 Jan 11 22:37 a1033e2616a84430b52172f8218a9849174fc5e8fdfa71b238e44dc47cbde404
-rw-r--r--+ 1 backup backup  709347 Jan  9 22:33 a1033e33d6ef180ab069f44a7c8d76da06438d8ca509e8c1b74e087fff9e993b
-rw-r--r--+ 1 backup backup  746298 Jan 13 22:24 a10343f085d9212e0b649d4f7d4e3101fa0cdbd99a14c3ff2ae7fe27aafc2514
-rw-r--r--+ 1 backup backup  633446 Dec  1 22:31 a1034b91269e12d9df372a828a6911c5518c7f4edeb1d332efac738e2e9522f1
-rw-r--r--+ 1 backup backup  667515 Dec  8 22:48 a1034cf303f22d4c6986e40bc2afdd991d192d86a60580bcaa6ad4806e0b3a09
-rw-r--r--+ 1 backup backup  656468 Aug 19 03:11 a10353401ea69290307d95fdc998919091878f30598a24e0e915169a583134c4
??????????? ? ?      ?            ?            ? a10353b89afebdf423f30cb42eac7923da6d6828286b7b403a629b66b1cbcb2e
-rw-r--r--+ 1 backup backup  158161 Dec 20 01:01 a10356501a51a5f8f35f85f3ddee04e903711f9cbada01af48de55d8e595bcaa
-rw-r--r--+ 1 backup backup  858357 Dec 17 23:44 a103567ea8bb19766870019f11e258dc1f4b66715c1a17fbdd09f82bef72a847
-rw-r--r--+ 1 backup backup   31749 Sep 25 22:15 a1035e4bc15f97c6a5de16b1e3624d6931335a1c7084a52d4a6f343b997f97dc
-rw-r--r--+ 1 backup backup   21662 Jan  6 22:45 a1036219e969d24e013a8a521e52fc8cd67ce5a2570db1a789918bbdc0d3efbf
-rwxrwxrwx+ 1 backup backup 4194316 Jul 24  2024 a1036a0f82e4a510d78b1cf67d2c96c1d711f148204f2762924958049440b1e7
-rw-r--r--+ 1 backup backup 2276227 Aug 24 23:11 a1036adacda56033c8a426f0e1f61cdab1e718a1b3f559bd4cb0109a2271e998
-rw-r--r--+ 1 backup backup 1377958 Sep 29 03:47 a1036bf23378aea7c65dd9cc97c2f048a42a24d50f3adeff549c055f1ad4d8e5
-rw-r--r--+ 1 backup backup 1691992 Nov  8 11:31 a1036bff91f85b1b289eb604551723c9e0083a2e0789f6276d7ffe07e2368187
-rw-r--r--+ 1 backup backup 1090031 Dec 20 23:39 a1036c29967cdfd6b541d2754a51b39d7ed51bb13b654fad5ddbf6b160cad804
??????????? ? ?      ?            ?            ? a1036f68d3ad55b7d28e4d86d16c5e070cf128367682be7959fab9dbeca2bc19
??????????? ? ?      ?            ?            ? a10371d38824a05c379bb26aa2178532592c5a61234876766942d9443dc4e8c9
-rw-r--r--+ 1 backup backup  835825 Nov 17 03:16 a1037229ca64fe5f58c6f98ec4bbdc5888eeab733daab206ad4db5e9f4060290
-rw-r--r--+ 1 backup backup  886267 Jan 20 22:15 a103730c1ccc1dc57d8ec106cdd36369f94a7770334e47c4841f185adf281559
-rw-r--r--+ 1 backup backup  378288 Nov 21 22:33 a1037329b529412bd1b95605fba9a8fb26eac3a086391700b096f600a718d133
-rw-r--r--+ 1 backup backup 1114250 Aug 25 03:19 a103770eb71a9983146093e727f3dc68e8779cabab9fdd380b238dcb54dcdddb
??????????? ? ?      ?            ?            ? a10378b32d59cb7f84e1c15c00433954a6c7d232a6fa2ee29ab2a81ac88d4218
-rw-r--r--+ 1 backup backup  983301 Jan 21 22:42 a1037d1326f168358d59d698f2b7f4b87ab1ce4531ed5e746233bc99cf08b230
??????????? ? ?      ?            ?            ? a1037f80abca905dd91c6651fcd9e107e469de6ba3b1fb6dd5fab4efb0e380e0
-rw-r--r--+ 1 backup backup 2431079 Dec 18 19:26 a1037ffbe7de233ec65120452d54d614009989e8a762e0465ae1ac16671b24d6
-rw-r--r--+ 1 backup backup  579991 Jan 13 20:03 a1038077878b784e35cc8a53d1ac2ac4efe9ffbc07030e21e7411ef9cb9b834f
-rw-r--r--+ 1 backup backup  819181 Dec 18 23:53 a10394d60a5cc11b3327669dbb18f2f724980b09709e686dafd8fe0f14a933ef
-rw-r--r--+ 1 backup backup 1767532 Dec 13 19:11 a1039b1c7447e5eed391a0af3b6861030217edca0af7eeec5e7c1b1c81e88b1c
-rw-r--r--+ 1 backup backup  425830 Nov 17 23:08 a1039ef36a90fd9bbcb20884a7d6e5821817ede13c320cf2cf91e3a1050ca4e0
-rw-r--r--+ 1 backup backup   75801 Dec 14 21:25 a103a1a28744e2acba50de28be19e0cd6371d708c63e8f3c633f31b3b76aa667
-rw-r--r--+ 1 backup backup 1133050 Aug 23 21:16 a103aa1809c7d085b9fc13ccad4159b15a436df3cde180793559b3933603e45d
??????????? ? ?      ?            ?            ? a103ac4477cb62e4e9463d96ad90c68272445c9e53e9ec6d1a0cab141d805e9f
-rw-r--r--+ 1 backup backup  786926 Dec 23 22:29 a103acbfa5f923be05afca4486afc754e4d5d381045564525850477d97b8ba3c
-rw-r--r--+ 1 backup backup 1512446 Jan 17 19:01 a103ad0aa9da9ec5c14991f8b6d2a9c7696a67ae485795bc8e1e017a77f68c3b
-rw-r--r--+ 1 backup backup 1143415 Sep 28 23:08 a103b104b08588e5d2bcaf6783e44cf30d75b99cf766504e3d05ba1a04e0fc2b
-rw-r--r--+ 1 backup backup 2076103 Nov 22 20:02 a103b20bc6df1365ba7aae5612cb384ce891f009d95bd5e770e8bf3cd65cfe6b
-rw-r--r--+ 1 backup backup 1802472 Sep 20 22:05 a103bf24dcfbc53d04680c8612748bf78f7a2ba36dcd1257dec4c3c3b46b7d2f
-rw-r--r--+ 1 backup backup 2150248 Jan  5 19:16 a103c0718d8093d3e552fdfd1c3e98871b5a514d0de9012bcce8bcc557a56f54
-rw-r--r--+ 1 backup backup 4194316 Oct 31 00:45 a103c2ca5d4a31c74f98301ed76890147225fdf1f7d887b8507556d6c3bd806a
-rw-r--r--+ 1 backup backup  805431 Sep 29 03:46 a103c2ee74992329640b9b2f9c11f835922a52c18ed79425db3705a3409fc96c
??????????? ? ?      ?            ?            ? a103c36c1790491f3a40328e3b9c040a093cafbf8472369a9fb9379cd6d9397f
-rw-r--r--+ 1 backup backup  674174 Dec 16 22:12 a103c463c8cc45145cf062af0d122765f106aadae9b4a564d5d6e222bea27e21
-rw-r--r--+ 1 backup backup 1009547 Dec 15 23:14 a103c8ca54e9a1de7cd4c292a81541794342271e38264b26ff49eab4b90767c2
-rw-r--r--+ 1 backup backup 1007651 Jan  8 22:34 a103c9c51876e8dc377a43fa9ab9ab4df368ac4c5c580bb48826de9c2192dbb9
-rw-r--r--+ 1 backup backup  118789 Jan 13 22:26 a103d04ee308ef7497d6fb2636091d2bdffb5f7297ae91825976801c83cd5bc7
-rw-r--r--+ 1 backup backup  441649 Dec  8 22:26 a103d7aef9abb9d25e2751a0b3226cabfa8352527af68387c85be1e17f540a6c
-rw-r--r--+ 1 backup backup  697421 Jan 15 22:16 a103dde9c08892322f9fe25e2c6925942f7e9941f06323051dddfaeb5f986eeb
-rw-r--r--+ 1 backup backup  921179 Jan  7 23:07 a103dffd54de30af01a6eab296547fc764b105e5828e0ddf685e94ab9452c8c9
-rw-r--r--+ 1 backup backup 1074145 Sep 29 04:12 a103e4124d4376c4f20b067ac5b1f3eb6cc6f1d2463619ed36666fbdebe403bf
-rw-r--r--+ 1 backup backup 2236075 Dec 18 20:16 a103e8d7c6c30e02c741bd5de39ea6d68157e64aff271b290cbf8dfd0ccecf64
-rw-r--r--+ 1 backup backup  995782 Jan  9 19:11 a103ee9f30e19fbf1b002b24a3b7cfb8a43c1cfbcf7970e1c44397c87819f21c
??????????? ? ?      ?            ?            ? a103f463130e4d7390a6cb061d2d87ff349d5f0c5a17a0c825eb25ad8851868e

I'm assume that means I've lost those chunks?

These bad files are preventing me from doing a backup and I'm not sure how to move forward from here. Would really appreciate some help.

 

[Wasca]

Okay so I've done a bit more reading and it looks like I can restore an index file which I have done by running this command.

proxmox-backup-debug recover index  /mnt/backup/vm/2206/2025-01-23T12:43:46Z/drive-scsi0.img.fidx /mnt/backup/.chunks --ignore-missing-chunks --ignore-corrupt-chunks

I can now see this extra file - drive-scsi0.img

drwxrwxrwx+  2 backup backup        4096 Jan 24 19:29 .
drwxrwxrwx+ 16 backup backup        4096 Jan 23 22:45 ..
-rw-r--r--+  1 backup backup        1013 Jan 23 22:45 client.log.blob
-rw-rw-rw-+  1 root   root   34359738368 Jan 24 20:30 drive-scsi0.img
-rw-rw-rw-+  1 backup backup      266240 Jan 23 22:45 drive-scsi0.img.fidx
-rw-r--r--+  1 backup backup         428 Jan 23 22:45 index.json.blob
-rw-r--r--+  1 backup backup         424 Jan 23 22:43 qemu-server.conf.blob

Now that I have this file How can I restore that VM?