Backup failed: "Input/output error (os error 5)"

[jedie]

I try to run a backup of a stopped LXC container. It ended very fast with:

...
INFO: catalog upload error - channel closed
INFO: Error: error at "media/servershare/Backups/000-Software/PyCharmCE2019.3/.PyCharmCE2019.3/config/plugins/Docker/lib/rt/jersey-hk2-2.27.jar": Input/output error (os error 5)
ERROR: Backup of VM 100 failed - command '...' failed: exit code 255
INFO: Failed at 2022-12-11 11:54:16
INFO: Backup job finished with errors
TASK ERROR: job errors

Think the interesting part is: error at "media/servershare/Backups/000-Software/PyCharmCE2019.3/.PyCharmCE2019.3/config/plugins/Docker/lib/rt/jersey-hk2-2.27.jar": Input/output error (os error 5)

I looked at this file, and i can't read it in pve shell:

root@pve:/tank/vmdata/subvol-100-disk-1/Backups/000-Software/PyCharmCE2019.3/.PyCharmCE2019.3/config/plugins/Docker/lib/rt# cat jersey-hk2-2.27.jar
cat: jersey-hk2-2.27.jar: Eingabe-/Ausgabefehler

Try to check this:

root@pve:~# zpool status -v
  pool: tank
 state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 01:01:25 with 1 errors on Sun Dec 11 01:25:26 2022
config:

    NAME        STATE     READ WRITE CKSUM
    tank        ONLINE       0     0     0
      sdb       ONLINE       0     0    16
      sdc       ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        /tank/vmdata/subvol-100-disk-1/Backups/000-Software/PyCharmCE2019.3/.PyCharmCE2019.3/config/plugins/Docker/lib/rt/jersey-hk2-2.27.jar

Currently a zpool scrub tank is running... I'm curious to see what comes out of it.

This file not very important for me: But is this an indication of serious problems?

Questions:

• What may cause this file corruption? (We had a power outage last week. However, it is very, very unlikely that this file was created/modified during that period.)
• Should the backup run not skip this file and continue backup and send a information to the user? (In my case, this error occurred very early at the beginning of the backup. But that happens if it's already backuped tons of data and just before the end the error occurs? All work is lost, then, isn't it?)
• There is no way to check this and run "scrub" in Web GUI, isn't it?
• Should i setup a periodic "scrub" run?!?

 

[jedie]

Hm. Scrub done, but nothing repaired:

root@pve:~# zpool status -v
  pool: tank
 state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 01:00:37 with 1 errors on Sun Dec 11 13:12:36 2022
config:

    NAME        STATE     READ WRITE CKSUM
    tank        ONLINE       0     0     0
      sdb       ONLINE       0     0    20
      sdc       ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        /tank/vmdata/subvol-100-disk-1/Backups/000-Software/PyCharmCE2019.3/.PyCharmCE2019.3/config/plugins/Docker/lib/rt/jersey-hk2-2.27.jar
        <0x15c8e>:<0x46d>

What means the additional entry: <0x15c8e>:<0x46d> ?!?

I just remove the file, but it's still in error list:

root@pve:~# rm /tank/vmdata/subvol-100-disk-1/Backups/000-Software/PyCharmCE2019.3/.PyCharmCE2019.3/config/plugins/Docker/lib/rt/jersey-hk2-2.27.jar
root@pve:~# zpool status -v
  pool: tank
 state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 01:00:37 with 1 errors on Sun Dec 11 13:12:36 2022
config:

    NAME        STATE     READ WRITE CKSUM
    tank        ONLINE       0     0     0
      sdb       ONLINE       0     0    20
      sdc       ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        tank/vmdata/subvol-100-disk-1:<0x46d>
        <0x15c8e>:<0x46d>

Looks like that the second entry relates to the origin file?!?