ZFS-8000-8A error backup

Thatoo · Apr 11, 2023

I got an email from my proxmox saying there was an error during backup.
I run the command


zpool status -x -v

and here is the result

Code:

pool: backuppool
 state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 01:33:18 with 1 errors on Sun Apr  9 01:57:19 2023
config:

    NAME                               STATE     READ WRITE CKSUM
    backuppool                         ONLINE       0     0     0
      nvme-CT4000P3PSSD8_2242E67C9265  ONLINE       0     0     3

errors: Permanent errors have been detected in the following files:

        backuppool/backups:<0x43>

I don't know what is backuppool/backups:<0x43> and what I should do now.

If anyone could explain me what is this error, how bad it is and what I should do, it would be really nice.

LnxBil · Apr 11, 2023

Is backups a dataset or a zvolume? If may be that something inside of the dataset is corrupt, you may need to debug ZFS with zdb.

leesteken · Apr 11, 2023

If I remember correctly, your "backups" is probably a zvol on the "backuppool" pool and you'll see this permanent error until those blocks with invalid checksums are overwritten (or deleted and trimmed). You'll keep getting read errors when you try to read the block. I don't remember how to find the file inside the zvol that uses that block. It depends on the used filesystem but I found some guides at the time for ext4. You could also try to read all files inside the filesystem on that zvol and see which ones fail and overwrite them.

Thatoo · Apr 12, 2023

I don't know if it's a dataset or a zvolume.
In Datacenter/Storage, "backups" is a Directory type and "backuppool" is a ZFS type.
I think backuppool is indeed a zfs pool.
However, I don't know if backups is a dataset or a zvolume, only that iti s a Directory type.

Thatoo · Apr 12, 2023

If I do zdb backuppool/backups , here is what I get

Code:

Dataset backuppool/backups [ZPL], ID 899, cr_txg 2827, 1.42T, 237 objects

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
         0    6   128K    16K   288K     512  1.33M    8.71  DMU dnode
        -1    1   128K    512      0     512    512  100.00  ZFS user/group/project used
        -2    1   128K    512      0     512    512  100.00  ZFS user/group/project used
        -3    1   128K    512      0     512    512  100.00  ZFS user/group/project used
         1    1   128K    512     8K     512    512  100.00  ZFS master node
         3    1   128K    512      0     512    512  100.00  ZFS directory
         6    2   128K    16K    48K     512    48K  100.00  ZFS directory
         7    1   128K    512      0     512    512  100.00  ZFS plain file
         8    3   128K   128K   414M     512   414M  100.00  ZFS plain file
         9    1   128K    512      0     512    512  100.00  ZFS plain file
        10    1   128K     1K     4K     512     1K  100.00  ZFS plain file
        11    1   128K    512      0     512    512  100.00  ZFS plain file
        12    1   128K  4.50K     4K     512  4.50K  100.00  ZFS plain file
        13    3   128K   128K  7.46G     512  7.45G  100.00  ZFS plain file
        15    1   128K    512      0     512    512  100.00  ZFS plain file
        17    3   128K   128K   568M     512   568M  100.00  ZFS plain file
        18    1   128K     1K     4K     512     1K  100.00  ZFS plain file
        20    3   128K   128K  2.84G     512  2.84G  100.00  ZFS plain file
        25    1   128K    512      0     512    512  100.00  ZFS plain file
        32    1   128K    512      0     512    512  100.00  SA master node
        33    1   128K    512      0     512    512  100.00  ZFS delete queue
        34    1   128K    512      0     512    512  100.00  ZFS directory
        35    1   128K  1.50K     8K     512  1.50K  100.00  SA attr registration
        36    1   128K    16K    16K     512    32K  100.00  SA attr layouts
        41    3   128K   128K  2.72G     512  2.72G  100.00  ZFS plain file
        42    3   128K   128K  7.09G     512  7.08G  100.00  ZFS plain file
        43    1   128K     1K     4K     512     1K  100.00  ZFS plain file
        45    1   128K    512      0     512    512  100.00  ZFS plain file
        46    1   128K    512      0     512    512  100.00  ZFS plain file
        47    1   128K  1.50K     4K     512  1.50K  100.00  ZFS plain file
        48    1   128K  1.50K     4K     512  1.50K  100.00  ZFS plain file
        49    1   128K     1K     4K     512     1K  100.00  ZFS plain file
        59    1   128K  3.50K     4K     512  3.50K  100.00  ZFS plain file
        60    3   128K   128K   375M     512   375M  100.00  ZFS plain file
        61    1   128K     1K     4K     512     1K  100.00  ZFS plain file
        62    1   128K    512      0     512    512  100.00  ZFS plain file
        63    1   128K    512      0     512    512  100.00  ZFS plain file
        64    3   128K   128K   902M     512   901M  100.00  ZFS plain file
        66    1   128K    512      0     512    512  100.00  ZFS plain file
        69    1   128K    512      0     512    512  100.00  ZFS plain file
        70    1   128K    512      0     512    512  100.00  ZFS plain file
       128    3   128K   128K   404M     512   404M  100.00  ZFS plain file
       135    1   128K    512      0     512    512  100.00  ZFS plain file
       138    1   128K     3K     4K     512     3K  100.00  ZFS plain file
       151    3   128K   128K   269M     512   269M  100.00  ZFS plain file
       152    1   128K    512      0     512    512  100.00  ZFS plain file
       157    1   128K    512      0     512    512  100.00  ZFS plain file
       158    3   128K   128K   413M     512   413M  100.00  ZFS plain file
       159    1   128K    512      0     512    512  100.00  ZFS plain file
       160    3   128K   128K   373M     512   373M  100.00  ZFS plain file
       161    1   128K    512      0     512    512  100.00  ZFS plain file
       162    3   128K   128K   569M     512   569M  100.00  ZFS plain file
       172    3   128K   128K   961M     512   961M  100.00  ZFS plain file
       174    3   128K   128K   436M     512   435M  100.00  ZFS plain file
       175    3   128K   128K   569M     512   569M  100.00  ZFS plain file
       176    3   128K   128K   269M     512   269M  100.00  ZFS plain file
       177    1   128K    512      0     512    512  100.00  ZFS plain file
       178    1   128K  5.50K     4K     512  5.50K  100.00  ZFS plain file
       182    3   128K   128K   122G     512   122G  100.00  ZFS plain file
       256    1   128K    512      0     512    512  100.00  ZFS plain file
       257    3   128K   128K   403M     512   403M  100.00  ZFS plain file
       259    3   128K   128K   418M     512   417M  100.00  ZFS plain file
       260    1   128K    512      0     512    512  100.00  ZFS directory
       262    1   128K    512      0     512    512  100.00  ZFS directory
       274    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       276    3   128K   128K   901M     512   901M  100.00  ZFS plain file
       277    1   128K  1.50K     4K     512  1.50K  100.00  ZFS plain file
       278    1   128K  1.50K     4K     512  1.50K  100.00  ZFS plain file
       279    1   128K    512      0     512    512  100.00  ZFS plain file
       280    1   128K  1.50K     4K     512  1.50K  100.00  ZFS plain file
       282    3   128K   128K  10.6G     512  10.6G  100.00  ZFS plain file
       295    3   128K   128K   512M     512   512M  100.00  ZFS plain file
       296    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       297    1   128K    512      0     512    512  100.00  ZFS plain file
       298    1   128K  1.50K     4K     512  1.50K  100.00  ZFS plain file
       385    3   128K   128K  2.90G     512  2.90G  100.00  ZFS plain file
       387    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       388    1   128K     6K     4K     512     6K  100.00  ZFS plain file
       389    3   128K   128K  10.5G     512  10.5G  100.00  ZFS plain file
       391    3   128K   128K   957M     512   956M  100.00  ZFS plain file
       398    3   128K   128K   504M     512   504M  100.00  ZFS plain file
       399    3   128K   128K   405M     512   405M  100.00  ZFS plain file
       400    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       407    1   128K  1.50K     4K     512  1.50K  100.00  ZFS plain file
       409    1   128K    512      0     512    512  100.00  ZFS plain file
       412    3   128K   128K   122G     512   122G  100.00  ZFS plain file
       413    1   128K    11K     4K     512    11K  100.00  ZFS plain file
       512    1   128K    512      0     512    512  100.00  ZFS plain file
       513    3   128K   128K  1.17G     512  1.17G  100.00  ZFS plain file
       514    1   128K    512      0     512    512  100.00  ZFS plain file
       515    1   128K    512      0     512    512    0.00  ZFS plain file
       516    1   128K    512      0     512    512  100.00  ZFS directory
       517    1   128K    512      0     512    512  100.00  ZFS directory
       519    3   128K   128K   954M     512   953M  100.00  ZFS plain file
       520    1   128K    512     4K     512    512  100.00  ZFS plain file
       524    1   128K    512     4K     512    512  100.00  ZFS plain file
       525    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       530    3   128K   128K   373M     512   373M  100.00  ZFS plain file
       532    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       547    3   128K   128K   363M     512   363M  100.00  ZFS plain file
       548    1   128K    512      0     512    512  100.00  ZFS plain file
       549    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       550    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       551    3   128K   128K   415M     512   415M  100.00  ZFS plain file
       552    3   128K   128K   425M     512   424M  100.00  ZFS plain file
       553    3   128K   128K  2.71G     512  2.71G  100.00  ZFS plain file
       554    3   128K   128K   383M     512   383M  100.00  ZFS plain file
       563    1   128K    512     4K     512    512  100.00  ZFS plain file
       565    3   128K   128K   561M     512   561M  100.00  ZFS plain file
       569    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       571    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       572    3   128K   128K   407M     512   407M  100.00  ZFS plain file
       573    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       574    1   128K    512      0     512    512  100.00  ZFS plain file
       577    3   128K   128K  10.6G     512  10.5G  100.00  ZFS plain file
       578    1   128K    512      0     512    512  100.00  ZFS plain file
       582    1   128K  11.5K     4K     512  11.5K  100.00  ZFS plain file
       584    4   128K   128K   206G     512   206G  100.00  ZFS plain file
       585    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       640    3   128K   128K  2.84G     512  2.84G  100.00  ZFS plain file
       641    1   128K    512      0     512    512  100.00  ZFS plain file
       643    1   128K  2.50K     4K     512  2.50K  100.00  ZFS plain file
       644    3   128K   128K   228M     512   516M  100.00  ZFS plain file
       645    3   128K   128K  5.09G     512  5.20G   99.96  ZFS plain file
       646    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       649    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       650    1   128K    512      0     512    512  100.00  ZFS plain file
       651    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       652    3   128K   128K  2.71G     512  2.71G  100.00  ZFS plain file
       654    3   128K   128K   349M     512   349M  100.00  ZFS plain file
       663    3   128K   128K   543M     512   543M  100.00  ZFS plain file
       664    1   128K  1.50K     4K     512  1.50K  100.00  ZFS plain file
       665    3   128K   128K   406M     512   406M  100.00  ZFS plain file
       666    1   128K  1.50K     4K     512  1.50K  100.00  ZFS plain file
       667    3   128K   128K   269M     512   269M  100.00  ZFS plain file
       668    3   128K   128K   364M     512   364M  100.00  ZFS plain file
       669    1   128K    512      0     512    512  100.00  ZFS plain file
       683    1   128K    512      0     512    512  100.00  ZFS plain file
       684    1   128K    512      0     512    512  100.00  ZFS plain file
       685    3   128K   128K  2.71G     512  2.71G  100.00  ZFS plain file
       686    3   128K   128K   729M     512   729M  100.00  ZFS plain file
       687    1   128K  1.50K     4K     512  1.50K  100.00  ZFS plain file
       689    1   128K    512      0     512    512  100.00  ZFS plain file
       690    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       768    1   128K    512      0     512    512  100.00  ZFS plain file
       770    1   128K    512      0     512    512  100.00  ZFS plain file
       776    1   128K    512      0     512    512  100.00  ZFS plain file
       779    1   128K    512      0     512    512  100.00  ZFS plain file
       782    1   128K    512      0     512    512  100.00  ZFS plain file
       783    1   128K    512      0     512    512  100.00  ZFS plain file
       786    3   128K   128K  51.1G     512  51.1G  100.00  ZFS plain file
       787    1   128K    512      0     512    512  100.00  ZFS plain file
       788    1   128K  1.50K     4K     512  1.50K  100.00  ZFS plain file
       789    1   128K    512      0     512    512  100.00  ZFS plain file
       794    3   128K   128K   267M     512   266M  100.00  ZFS plain file
       802    1   128K    512      0     512    512  100.00  ZFS plain file
       806    1   128K    512      0     512    512  100.00  ZFS plain file
       807    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       809    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       810    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       811    1   128K    512      0     512    512  100.00  ZFS plain file
       812    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       813    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       814    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       821    1   128K    512      0     512    512  100.00  ZFS plain file
       822    4   128K   128K   206G     512   206G  100.00  ZFS plain file
       902    1   128K    512      0     512    512  100.00  ZFS plain file
       903    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       904    1   128K     1K     4K     512     1K  100.00  ZFS plain file
       907    4   128K   128K   206G     512   206G  100.00  ZFS plain file
       908    4   128K   128K   206G     512   206G  100.00  ZFS plain file
      1025    1   128K    512      0     512    512  100.00  ZFS plain file
      1026    1   128K    512      0     512    512  100.00  ZFS plain file
      1155    1   128K    512      0     512    512  100.00  ZFS plain file
      1156    1   128K     1K     4K     512     1K  100.00  ZFS plain file
      1282    1   128K    512      0     512    512  100.00  ZFS plain file
      1283    1   128K    512      0     512    512  100.00  ZFS plain file
      1290    1   128K    512      0     512    512  100.00  ZFS plain file
      1294    1   128K     1K     4K     512     1K  100.00  ZFS plain file
      1295    1   128K    512      0     512    512  100.00  ZFS plain file
      1301    1   128K    512      0     512    512  100.00  ZFS plain file
      1302    1   128K    512      0     512    512  100.00  ZFS plain file
      1410    3   128K   128K   413M     512   413M  100.00  ZFS plain file
      1412    1   128K    512      0     512    512  100.00  ZFS plain file
      1413    1   128K  1.50K     4K     512  1.50K  100.00  ZFS plain file
      1414    3   128K   128K   507M     512   507M  100.00  ZFS plain file
      1415    3   128K   128K   901M     512   900M  100.00  ZFS plain file
      1419    3   128K   128K  51.1G     512  51.1G  100.00  ZFS plain file
      1536    3   128K   128K  51.1G     512  51.1G  100.00  ZFS plain file
      1537    1   128K     1K     4K     512     1K  100.00  ZFS plain file
      1538    1   128K     1K     4K     512     1K  100.00  ZFS plain file
      1539    3   128K   128K   543M     512   543M  100.00  ZFS plain file
      1546    3   128K   128K   378M     512   378M  100.00  ZFS plain file
      1553    3   128K   128K   415M     512   415M  100.00  ZFS plain file
      1554    1   128K    512      0     512    512  100.00  ZFS plain file
      1556    1   128K  2.50K     4K     512  2.50K  100.00  ZFS plain file
      1557    1   128K  3.50K     4K     512  3.50K  100.00  ZFS plain file
      1558    1   128K  1.50K     4K     512  1.50K  100.00  ZFS plain file
      1560    1   128K    512      0     512    512  100.00  ZFS plain file
      1576    1   128K    512      0     512    512  100.00  ZFS plain file
      1579    3   128K   128K  7.06G     512  7.06G  100.00  ZFS plain file
      1580    3   128K   128K   460M     512   460M  100.00  ZFS plain file
      1581    3   128K   128K   414M     512   414M  100.00  ZFS plain file
      1583    1   128K     1K     4K     512     1K  100.00  ZFS plain file
      1584    3   128K   128K   545M     512   545M  100.00  ZFS plain file

Thatoo · Apr 12, 2023

Code:

1585    1   128K    512      0     512    512  100.00  ZFS plain file
      1586    1   128K    512      0     512    512  100.00  ZFS plain file
      1587    1   128K    512      0     512    512  100.00  ZFS plain file
      1588    1   128K     1K     4K     512     1K  100.00  ZFS plain file
      1589    3   128K   128K   354M     512   354M  100.00  ZFS plain file
      1590    1   128K    512      0     512    512  100.00  ZFS plain file
      1591    1   128K    512      0     512    512  100.00  ZFS plain file
      1592    3   128K   128K   562M     512   562M  100.00  ZFS plain file
      1596    1   128K    512      0     512    512  100.00  ZFS plain file
      1792    3   128K   128K  7.07G     512  7.07G  100.00  ZFS plain file
      1793    1   128K    512      0     512    512  100.00  ZFS plain file
      1923    1   128K     1K     4K     512     1K  100.00  ZFS plain file
      1931    1   128K  11.5K     8K     512  11.5K  100.00  ZFS plain file
      2053    1   128K    512      0     512    512  100.00  ZFS plain file
      2054    1   128K    512      0     512    512  100.00  ZFS plain file
      2057    1   128K  1.50K     4K     512  1.50K  100.00  ZFS plain file
      2058    3   128K   128K   350M     512   350M  100.00  ZFS plain file
      2059    1   128K  1.50K     4K     512  1.50K  100.00  ZFS plain file
      2082    3   128K   128K  2.48G     512  2.48G  100.00  ZFS plain file
      2083    1   128K     4K     4K     512     4K  100.00  ZFS plain file
      2084    1   128K     1K     4K     512     1K  100.00  ZFS plain file
      2085    1   128K    512      0     512    512  100.00  ZFS plain file
      2087    1   128K    512      0     512    512  100.00  ZFS plain file
      2092    3   128K   128K   562M     512   562M  100.00  ZFS plain file
      2094    1   128K  1.50K     4K     512  1.50K  100.00  ZFS plain file
      2177    1   128K     1K     4K     512     1K  100.00  ZFS plain file
      2178    1   128K     5K     4K     512     5K  100.00  ZFS plain file
      2180    1   128K     1K     4K     512     1K  100.00  ZFS plain file
      2309    3   128K   128K   365M     512   365M  100.00  ZFS plain file
      2310    1   128K     1K     4K     512     1K  100.00  ZFS plain file
      2436    1   128K     1K     4K     512     1K  100.00  ZFS plain file
      2437    1   128K  1.50K     4K     512  1.50K  100.00  ZFS plain file
      2438    3   128K   128K   267M     512   266M  100.00  ZFS plain file
      2562    1   128K    512      0     512    512  100.00  ZFS plain file
      2688    3   128K   128K   123G     512   123G  100.00  ZFS plain file
      2689    1   128K    512      0     512    512  100.00  ZFS plain file

    Dnode slots:
    Total used:           237
    Max used:            2689
    Percent empty:  91.186315


ZFS_DBGMSG(zdb) START:
spa.c:5181:spa_open_common(): spa_open_common: opening backuppool/backups
spa_misc.c:418:spa_load_note(): spa_load(backuppool, config trusted): LOADING
vdev.c:160:vdev_dbgmsg(): disk vdev '/dev/disk/by-id/nvme-CT4000P3PSSD8_2242E67C9265-part1': best uberblock found for spa backuppool. txg 899426
spa_misc.c:418:spa_load_note(): spa_load(backuppool, config untrusted): using uberblock with txg=899426
spa.c:8360:spa_async_request(): spa=backuppool async request task=2048
spa_misc.c:418:spa_load_note(): spa_load(backuppool, config trusted): LOADED
ZFS_DBGMSG(zdb) END

How to link anything to the error

Code:

errors: Permanent errors have been detected in the following files:
backuppool/backups:<0x43>

Thatoo · Apr 12, 2023

Is this

Code:

backuppool/backups:<0x43>

the same as if I do

Code:

root@xxx:/# zdb -d backuppool/backups 43
Dataset backuppool/backups [ZPL], ID 899, cr_txg 2827, 1.42T, 237 objects

    Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
        43    1   128K     1K     4K     512     1K  100.00  ZFS plain file

?

Thatoo · Apr 12, 2023

Here is what I have done thank to the ZFS community (mailing liste)

Code:

$ zpool clear backuppool
$ zpool scrub -w backuppool; zpool scrub -w backuppool
$ zpool status -v backuppool
pool: backuppool
state: ONLINE
scan: scrub repaired 0B in 01:00:27 with 0 errors on Wed Apr 12 16:18:53 2023
config:

NAME                               STATE     READ WRITE CKSUM
backuppool                         ONLINE       0     0     0
nvme-CT4000P3PSSD8_2242E67C9265  ONLINE       0     0     0

errors: No known data errors


# zdb backuppool/backups 0x43
Dataset backuppool/backups [ZPL], ID 899, cr_txg 2827, 1.42T, 237 objects

Object  lvl   iblk   dblk  dsize  dnsize  lsize   %full  type
zdb: dmu_object_info() failed, errno 2

ZFS_DBGMSG(zdb) START:
spa.c:5181:spa_open_common(): spa_open_common: opening backuppool/backups
spa_misc.c:418:spa_load_note(): spa_load(backuppool, config trusted): LOADING
vdev.c:160:vdev_dbgmsg(): disk vdev '/dev/disk/by-id/nvme-CT4000P3PSSD8_2242E67C9265-part1': best uberblock found for spa backuppool. txg 902781
spa_misc.c:418:spa_load_note(): spa_load(backuppool, config untrusted): using uberblock with txg=902781
spa.c:8360:spa_async_request(): spa=backuppool async request task=2048
spa_misc.c:418:spa_load_note(): spa_load(backuppool, config trusted): LOADED
ZFS_DBGMSG(zdb) END

It looks like the problem is solved.
Thank you all for your help.

Search

Search

ZFS-8000-8A error backup

Thatoo

Member

LnxBil

Distinguished Member

leesteken

Distinguished Member

Thatoo

Member

Thatoo

Member

Thatoo

Member

Thatoo

Member

Thatoo

Member