Permanent errors have been detected in the following files

[vassa]

Good afternoon, please tell me how to fix this error?

pool: rpool
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://zfsonlinux.org/msg/ZFS-8000-8A
scan: scrub repaired 0B in 0h13m with 3 errors on Sun Jul 18 17:21:13 2021
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 4
mirror-0 ONLINE 0 0 2
sdb2 ONLINE 0 0 2
sdc2 ONLINE 0 0 2
mirror-1 ONLINE 0 0 6
ata-Samsung_SSD_850_120GB_S3WBNX0K821254W ONLINE 0 0 6
ata-Samsung_SSD_850_120GB_S3WBNX0K810091B ONLINE 0 0 6

errors: Permanent errors have been detected in the following files:

rpool/data/vm-100-disk-1:<0x1>

 

[leesteken]

Restore local-zfs:vm-100-disk-1 of VM 100 from backup. It is probably much easier to restore the VM completely from backup. Once the disk-1 of VM 100 is overwritten (by the restore) or deleted (if you restore to another VM ID and delete VM 100), the error should disappear automatically.

 

[vassa]

I only have a backup from June 29th, if run now the backup throws this error

2021-07-18 18:22:47 INFO: status: 28% (54138175488/193273528320), sparse 1% (3158142976), duration 733, read/write 53/50 MB/s
2021-07-18 18:23:10 INFO: status: 28% (55390240768/193273528320), sparse 1% (3159470080), duration 756, read/write 54/54 MB/s
2021-07-18 18:23:10 ERROR: job failed with err -5 - Input/output error
2021-07-18 18:23:10 INFO: aborting backup job
2021-07-18 18:23:13 ERROR: Backup of VM 100 failed - job failed with err -5 - Input/output error

 

[leesteken]

ZFS has detected an error inside the virtual disk that it cannot recover from (that is why it is permanent). The error appears to be presents on both mirror drives (maybe cause by unexpected power loss?). Therefore, it gives an error whenever you try to read those blocks inside the virtual disk.

The easiest way to restore from backup before the error. Doing a backup now is too late.
The other way is to treat this inside the VM as bad sectors. I don't know which OS you are using but: scan the (virtual) disk for bad sectors (from inside the VM) and determine the disk blocks with errors. Then find out which files use those block and replace those files with correct ones (or delete them if you don't care). There are probably guides on the internet about how to find bad sectors on (physical) disk or broken files that can help.

 

[vassa]

Will replacing the failed disks solve the problem?

 

[leesteken]

Will replacing the failed disks solve the problem?

Do you mean the physical Samsung drives? The error is on all of the two disks, so you cannot fix it because you do not have a drive that does not have the error.
Do you mean the virtual VM disk? Yes. replacing the virtual disk-1 of the VM 100 should solve the problem (but where will you get a good copy of that virtual disk?).

 

[vassa]

Yes physical disks. I have a RAID10 of four drives.

 

[leesteken]

Sorry, if that was possible then ZFS would not report the error as permanent (and would have fixed it for you automatically). You only have two copies of the data, and both have the (same) error. You need to correct copy of the data from somwhere, or delete the VM 100 disk-1 (and recreate the VM from backup or manually).

 

[vassa]

Thank you very much for your help, apparently I lost my data forever.

 

[leesteken]

Thank you very much for your help, apparently I lost my data forever.

Maybe it is just a log file inside the VM as I once had. As I said before: run a bad blocks check inside the VM and try to determine which files are broken. Maybe it is not that bad.

 

[vassa]

I identified a faulty file, I have PostgreSQL and one table is damaged in it, now I am trying to load this table from an old copy.

 

[leesteken]

I identified a faulty file, I have PostgreSQL and one table is damaged in it, now I am trying to load this table from an old copy.

If you manage to replace the broken data, you need to trim the virtual disk to let the underlying ZFS know that the broken data is no longer used. And maybe you need a scrub of your pool.

 

[vassa]

If I manage to restore the database, then I'll just dump and change disks and reinstall proxmox and the virtual machine. Here is the error:
COPY public._inforg13267 (_fld13268_type, _fld13268_rtref, _fld13268_rrref, _fld13269, _fld770) TO stdout;