CT's on LVM disk get bad message error in turn

[achirkov]

Hi, i have phyzical hetzner server with proxmox 6.3. Мonth ago i got error https://forum.proxmox.com/threads/backup-lxc-bad-message.80474/ .
fsck fix container disk, but the next day another container received an same error. I delete all containers and restore from backup. Everything is working fine, the backup jobs have been working successfully until today, when I received the same bad message error from the container already restored from the backup.
What can be done to prevent the problem from appearing again?
Disk in lvdisplay:

--- Logical volume ---
  LV Path                /dev/vg0/vm-102-disk-0
  LV Name                vm-102-disk-0
  VG Name                vg0
  LV UUID                YYzwUi-7qlV-VIwe-ufsU-uIe1-hKQg-eC7MSw
  LV Write Access        read/write
  LV Creation host, time server, 2020-12-21 06:30:03 +0100
  LV Pool name           virt-hdd
  LV Status              available
  # open                 1
  LV Size                8.00 GiB
  Mapped size            40.30%
  Current LE             2048
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     4096
  Block device           253:7

in syslog i see message like this:
[2817590.943700] EXT4-fs error (device dm-10): ext4_lookup:1701: inode #175475: comm tar: iget: checksum invalid
Backup task:

INFO: Starting Backup of VM 101 (lxc)
INFO: Backup started at 2020-12-28 00:37:44
INFO: status = running
INFO: CT Name: search
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "snap_vm-101-disk-0_vzdump" created.
INFO: creating vzdump archive '/mnt/pve/Backup/dump/vzdump-lxc-101-2020_12_28-00_37_44.tar.lzo'
INFO: tar: ./home/user/src/project/noop.js: Cannot stat: Bad message
INFO: tar: ./home/user/src/project/sd/_util.js: Cannot stat: Bad message
INFO: tar: ./home/user/src/project/sd/forOwn.js: Cannot stat: Bad message
INFO: tar: ./home/user/src/project/sd/update.js: Cannot stat: Bad message
INFO: tar: ./home/user/src/project/sd/defer.js: Cannot stat: Bad message
INFO: tar: ./home/user/src/project/sd/overArgs.js: Cannot stat: Bad message
INFO: tar: ./home/user/src/project/sd/negate.js: Cannot stat: Bad message
INFO: tar: ./home/user/src/project/sd/_convertBrowser.js: Cannot stat: Bad message
INFO: tar: ./home/user/src/project/sd/symmetricDifferenceBy.js: Cannot stat: Bad message
INFO: tar: ./home/user/src/project/sd/startsWith.js: Cannot stat: Bad message
INFO: tar: ./home/user/src/project/sd/cloneDeep.js: Cannot stat: Bad message
INFO: tar: ./home/user/src/project/sd/snakeCase.js: Cannot stat: Bad message
INFO: tar: ./home/user/src/project/sd/uniqBy.js: Cannot stat: Bad message
INFO: tar: ./home/user/src/project/sd/stubTrue.js: Cannot stat: Bad message
INFO: tar: ./home/user/src/project/sd/identical.js: Cannot stat: Bad message
INFO: tar: ./usr/share/icons/ubuntu-mono-light/categories: Cannot stat: Bad message
INFO: tar: ./usr/share/icons/ubuntu-mono-light/places/48/folder-home.svg: Cannot stat: Bad message
INFO: tar: ./usr/share/icons/ubuntu-mono-light/places/48/gnome-fs-home.svg: Cannot stat: Bad message
INFO: tar: ./usr/share/icons/ubuntu-mono-light/status: Cannot stat: Bad message
INFO: Total bytes written: 6867732480 (6.4GiB, 172MiB/s)
INFO: tar: Exiting with failure status due to previous errors
INFO: cleanup temporary 'vzdump' snapshot
  Logical volume "snap_vm-101-disk-0_vzdump" successfully removed
ERROR: Backup of VM 101 failed - command 'set -o pipefail && tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/var/tmp/vzdumptmp20483_101' ./etc/vzdump/pct.conf ./etc/vzdump/pct.fw '--directory=/mnt/vzsnap0' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' ./ | lzop >/mnt/pve/Backup/dump/vzdump-lxc-101-2020_12_28-00_37_44.tar.dat' failed: exit code 2
INFO: Failed at 2020-12-28 00:41:33

 

[apoc]

That looks like some Hardware issue to me.
Something seems to go wrong in your disk/filesystem or memory

Does the system use ECC memory? In this case this almost can be ruled out.

 

[achirkov]

looked in dmidecode - system use ECC.
SMART not have any error on disk.

 

[apoc]

Can you correlate your issue with the uptime of the server?
I experience a strange behavior where my storage goes crazy (one pool) after 24 days of uptime.
on the 25th day of uptime I experience strange errors and the ZFS pool gets suspended.
If I reboot before that all is ok. Scrubs are fine.

SMART does not necessarily mean you dont have an issue in the hardware. This could be bad cables or such.
Of course this is a tough one to rule out...

 

[achirkov]

Can you correlate your issue with the uptime of the server?
I experience a strange behavior where my storage goes crazy (one pool) after 24 days of uptime.
on the 25th day of uptime I experience strange errors and the ZFS pool gets suspended.
If I reboot before that all is ok. Scrubs are fine.

SMART does not necessarily mean you dont have an issue in the hardware. This could be bad cables or such.
Of course this is a tough one to rule out...

After reboot server my issue is not gone (uptime before reboot - 45 days).