Bad Chunks that aren't bad???

[Cayuga]

Hi,

If I look for chunks that are named "*.bad" and then run "proxmox-backup-debug inspect file" on the ".bad" files, I will get a zero exit status and size and encryption status on almost all of them and only occasionally will I get a "real" error.

Are these chunks really bad? If so, why are there so many of them? If not, why are they being marked bad?

Thanks in advance for any help/suggestions!

Jeff

 

[dcsapak]

the rename to '.bad' happens when the content does not coincide with the checksum anymore, and 'proxmox-backup-debug instpect file' would not print an error if e.g. only a character changed (but the checksum won't match)

only occasionally will I get a "real" error.

which errors do you get?

in any case, if verify marks chunks bad and some reads return errors, i'd check the underlying storage + memory for errors/defects

 

[Cayuga]

I haven't had any bad chunks in the last 2 days. I'll post the error(s) when they re-appear.

 

[Cayuga]

They have re-appeared...I don't see any disk errors...(output is ls -l and proxmox-backup-debug inspect file)

-rw-r--r-- 1 backup backup 980131 Jun 10 13:58 /rpool/ROOT/backups/.chunks/3060/30600d38ebd67a0ae7a24d155795a328de0d391051ee8f4336f80501ca02fac4.0.bad
size: 980131
encryption: none

-rw-r--r-- 1 backup backup 594152 Jun 25  2022 /rpool/ROOT/backups/.chunks/3068/3068f8cffdd1568af16ebe2a668f71b6b7f8d20f9f70b0679d9e94eedf2f7723.0.bad
size: 594152
encryption: none

-rw-r--r-- 1 backup backup 4175819 Jun  7  2022 /rpool/ROOT/backups/.chunks/3070/30706bbd5d86b989406616493a717bff09d826a1f19692b989f505350ceb26fb.0.bad
size: 4175819
encryption: none

-rw-r--r-- 1 backup backup 4182429 Jun  7  2022 /rpool/ROOT/backups/.chunks/308d/308d08551c485611bad9f7b2c345e42a92db29392cab20b4136dfb04fff44514.0.bad
size: 4182429
encryption: none

-rw-r--r-- 1 backup backup 4194316 Aug 10  2022 /rpool/ROOT/backups/.chunks/30b8/30b899566a8cced6202753f34518971724a8f6f1b0f7984caad5be93ff616839.0.bad
size: 4194316
encryption: none

-rw-r--r-- 1 backup backup 1093382 Jun 10 13:57 /rpool/ROOT/backups/.chunks/3142/314229d3f5f5e482d7e4163fa05081495547eb0bc2a94ae6d2fce730ac5855e7.0.bad
size: 1093382
encryption: none

-rw-r--r-- 1 backup backup 4187889 Jun  6  2022 /rpool/ROOT/backups/.chunks/3181/3181c0bfa0f6cd30dda707635dcd52d3988c9ebcc8f41816b8c8ac6fd0548b41.0.bad
size: 4187889
encryption: none

-rw-r--r-- 1 backup backup 4166871 Jun  7  2022 /rpool/ROOT/backups/.chunks/3198/31987334cef469ada5ab4358996a96885f21d67190fa815d87c8b4e35a443b4e.0.bad
size: 4166871
encryption: none

-rw-r--r-- 1 backup backup 2809117 Sep 22  2022 /rpool/ROOT/backups/.chunks/31ba/31ba4a35d35a803d6f1f96cf4d9a2fb0fd4d8e4b9693debb7a4cef5fa609480c.0.bad
size: 2809117
encryption: none

-rw-r--r-- 1 backup backup 4191171 Jun  6  2022 /rpool/ROOT/backups/.chunks/31c7/31c7b292979db986860f65fd19d7a757264341c624a681ef55473d4c61077ea5.0.bad
size: 4191171
encryption: none

-rw-r--r-- 1 backup backup 4169306 Jun  6  2022 /rpool/ROOT/backups/.chunks/31ec/31ecfdfef3f86ea6f56d3fc825555980d201f0cb6506c310d5b3a02ef17a9e6b.0.bad
size: 4169306
encryption: none

-rw-r--r-- 1 backup backup 4186949 Jun  7  2022 /rpool/ROOT/backups/.chunks/3206/32068a936cc2913d4181baeded011bc6e9a86dd520b2a9d50f3345479fa5931a.0.bad
size: 4186949
encryption: none

-rw-r--r-- 1 backup backup 4165768 Jun  6  2022 /rpool/ROOT/backups/.chunks/320a/320a0704b22de4b28d25e0b6baa54cfaa3f141d8f9ed683f68d0166fbc6a5ece.0.bad
size: 4165768
encryption: none

-rw-r--r-- 1 backup backup 4163280 Jun  7  2022 /rpool/ROOT/backups/.chunks/323b/323beab67ac6188458a7f16ddbd2631d8b3becc40551b9b6f11facefaeeb28b2.0.bad
size: 4163280
encryption: none

-rw-r--r-- 1 backup backup 4174398 Jun  7  2022 /rpool/ROOT/backups/.chunks/329d/329dc904cb4314516eed33612235aa620c2ade7d0cea80a53d400c8d927a338d.0.bad
size: 4174398
encryption: none

-rw-r--r-- 1 backup backup 4106569 Jun  6  2022 /rpool/ROOT/backups/.chunks/329e/329ea8a5bc996ca66f0a90a71b6c606cc7af8c22ee9e725510cbaa2b660ec0b6.0.bad
size: 4106569
encryption: none

-rw-r--r-- 1 backup backup 4132090 May 13 00:13 /rpool/ROOT/backups/.chunks/32b1/32b15b7cadca4bfe71e19b8e1343519298f761b31f93e75bae63b7ff56f44e59.0.bad
size: 4132090
encryption: none

-rw-r--r-- 1 backup backup 4194316 Jan 25  2022 /rpool/ROOT/backups/.chunks/32da/32da6a6eb91f5101ca86ea3c6159ed562f3accd472e19b309988bcdc218619d8.0.bad
size: 4194316
encryption: none

-rw-r--r-- 1 backup backup 4160864 Jun  7  2022 /rpool/ROOT/backups/.chunks/32db/32dbdb2fb75234c691a163e62433bbcba6224ef435aa20bd0010960c5ccd632c.0.bad
size: 4160864
encryption: none

 

[dcsapak]

as i said, the backup debug tool would not print an error in that case since it does not check the checksum that way

if you want that, copy the file somwhere else without the '.0.bad' extension and do

proxmox-backup-debug inspect chunk /path/to/chunk --decode /tmp/decoded

if the checksum matches it will print the crc and encryption info and put the decoded part into /tmp/decoded
if the checksum does not match the contet you get an error:

Error: detected chunk with wrong digest.

 

[Cayuga]

Dominic,

Thanks for that. I had clearly misunderstood and thought that inspect file would check the CRC. I just tried the inspect chunk on 20 "bad" chunks and they look "OK" to me.... Am I missing something? If not, any idea how/why they were marked as bad?

Thanks again,
Jeff

'/rpool/ROOT/backups/.chunks/fd27/fd27b32a5e6a247b271fc6902e702660a8652ee866576602c447b4cce8b50c65.0.bad' -> '/tmp/fd27b32a5e6a247b271fc6902e702660a8652ee866576602c447b4cce8b50c65'
CRC: "3132916419(OK)"
encryption: "none"

'/rpool/ROOT/backups/.chunks/fd8f/fd8fe7cd7535af3abf77c315fceb6fdcdcd1bf55ca804e234e2e6bd677f6a91a.0.bad' -> '/tmp/fd8fe7cd7535af3abf77c315fceb6fdcdcd1bf55ca804e234e2e6bd677f6a91a'
CRC: "1753571366(OK)"
encryption: "none"

'/rpool/ROOT/backups/.chunks/fd9f/fd9fe3db6fc59b9e3dcc21e47eaeed1476dd16cce66f404c5933c86d2f62d558.0.bad' -> '/tmp/fd9fe3db6fc59b9e3dcc21e47eaeed1476dd16cce66f404c5933c86d2f62d558'
CRC: "2852792139(OK)"
encryption: "none"

'/rpool/ROOT/backups/.chunks/fde8/fde8e0876692eb1c68d8bc526747b2b2f7f931c7785964e7d0cab10ff8ed17bf.0.bad' -> '/tmp/fde8e0876692eb1c68d8bc526747b2b2f7f931c7785964e7d0cab10ff8ed17bf'
CRC: "697055274(OK)"
encryption: "none"

'/rpool/ROOT/backups/.chunks/fdf3/fdf30d43cb04f39464b87f54f87ac2fab97644d33bcd4ce944b6e98654253bf3.0.bad' -> '/tmp/fdf30d43cb04f39464b87f54f87ac2fab97644d33bcd4ce944b6e98654253bf3'
CRC: "2783017447(OK)"
encryption: "none"

'/rpool/ROOT/backups/.chunks/fdfb/fdfb2660b2a34393968e759829e2e23693e62ef3d3822cb536a69f0fcde71fb4.0.bad' -> '/tmp/fdfb2660b2a34393968e759829e2e23693e62ef3d3822cb536a69f0fcde71fb4'
CRC: "286715537(OK)"
encryption: "none"

'/rpool/ROOT/backups/.chunks/fe06/fe0664a53de0c8698a6b34b8a8aa74d691b955ae142d326b1181a8f01d1e2241.0.bad' -> '/tmp/fe0664a53de0c8698a6b34b8a8aa74d691b955ae142d326b1181a8f01d1e2241'
CRC: "443431380(OK)"
encryption: "none"

'/rpool/ROOT/backups/.chunks/fe3b/fe3b20aa9f36481cbf2b4bffdb971f600b03042f6894abb8231d64deec7d086c.0.bad' -> '/tmp/fe3b20aa9f36481cbf2b4bffdb971f600b03042f6894abb8231d64deec7d086c'
CRC: "2405126243(OK)"
encryption: "none"

'/rpool/ROOT/backups/.chunks/fecf/fecf7d021dbd0227961bd5446e657146e6c91b193d055c15a4baeb5bf7825d0e.0.bad' -> '/tmp/fecf7d021dbd0227961bd5446e657146e6c91b193d055c15a4baeb5bf7825d0e'
CRC: "3377243505(OK)"
encryption: "none"

'/rpool/ROOT/backups/.chunks/fedc/fedc53d2d18ac100a083fe483d8a7b6541fb580174916e36f37ce4ffb195f582.0.bad' -> '/tmp/fedc53d2d18ac100a083fe483d8a7b6541fb580174916e36f37ce4ffb195f582'
CRC: "3922195058(OK)"
encryption: "none"

'/rpool/ROOT/backups/.chunks/fef4/fef49bf67d22933b47c00f528edb85f4babdde7a230ef06baaab61115ebad8c2.0.bad' -> '/tmp/fef49bf67d22933b47c00f528edb85f4babdde7a230ef06baaab61115ebad8c2'
CRC: "4153129803(OK)"
encryption: "none"

'/rpool/ROOT/backups/.chunks/ff0d/ff0d99e91ba4e598b4e747132f103de54dad461379af283224dc541e5621b5dd.0.bad' -> '/tmp/ff0d99e91ba4e598b4e747132f103de54dad461379af283224dc541e5621b5dd'
CRC: "2270073844(OK)"
encryption: "none"

'/rpool/ROOT/backups/.chunks/ff1e/ff1ef7646f705aadcf6edefd37ccc6025b288946e8aa8e893fd1d67a71cb7957.0.bad' -> '/tmp/ff1ef7646f705aadcf6edefd37ccc6025b288946e8aa8e893fd1d67a71cb7957'
CRC: "27666291(OK)"
encryption: "none"

'/rpool/ROOT/backups/.chunks/ff21/ff210295ccdb1a9cde7aa52dcb559f098f0bf365b106dc5852bc86dd0012ff23.0.bad' -> '/tmp/ff210295ccdb1a9cde7aa52dcb559f098f0bf365b106dc5852bc86dd0012ff23'
CRC: "506491234(OK)"
encryption: "none"

'/rpool/ROOT/backups/.chunks/ff5d/ff5db5f354b010cda61d435f0971929630172c9b10dc083c4bb864db9073c873.0.bad' -> '/tmp/ff5db5f354b010cda61d435f0971929630172c9b10dc083c4bb864db9073c873'
CRC: "1343402709(OK)"
encryption: "none"

'/rpool/ROOT/backups/.chunks/ff63/ff630694a5ad63df76adb3ecb1a196e01875cfe8113446f41a6f71faf7532831.0.bad' -> '/tmp/ff630694a5ad63df76adb3ecb1a196e01875cfe8113446f41a6f71faf7532831'
CRC: "2233371329(OK)"
encryption: "none"

'/rpool/ROOT/backups/.chunks/ff73/ff737ab1bed488e53ad2df41b50e82313a4255e9ff923cc8e1abc105ee6d1b76.0.bad' -> '/tmp/ff737ab1bed488e53ad2df41b50e82313a4255e9ff923cc8e1abc105ee6d1b76'
CRC: "3557605235(OK)"
encryption: "none"

'/rpool/ROOT/backups/.chunks/ff75/ff75c3974828c906a34a0261e58cdf54c1561e2f52005d8141262e182e0e828d.0.bad' -> '/tmp/ff75c3974828c906a34a0261e58cdf54c1561e2f52005d8141262e182e0e828d'
CRC: "2132791359(OK)"
encryption: "none"

'/rpool/ROOT/backups/.chunks/ffc8/ffc88d70d6ce48781cac9b8a7edff94fced348f09d7918044adba21b92e01e90.0.bad' -> '/tmp/ffc88d70d6ce48781cac9b8a7edff94fced348f09d7918044adba21b92e01e90'
CRC: "691304680(OK)"
encryption: "none"

'/rpool/ROOT/backups/.chunks/ffd6/ffd61bb99203a9e3bfc267dd97d168b1aac1385a665c0b89c7478fdc0e955ba6.0.bad' -> '/tmp/ffd61bb99203a9e3bfc267dd97d168b1aac1385a665c0b89c7478fdc0e955ba6'
CRC: "1830703685(OK)"
encryption: "none"

 

[dcsapak]

you have to give '--decode' otherwise it won't check the checksum.... and if the .0.bad is not removed you also have to give '--digest <digest>' to check it

 

[Cayuga]

I just looped on a list of .bad.0 files using this code.....

x=$(basename $f | sed -e s/.0.bad//) ; cp -v $f /tmp/$x ; proxmox-backup-debug inspect chunk /tmp/$x --decode /tmp/decoded

this should have checked the CRC, correct?

 

[dcsapak]

yes, the crc and the digest

if that succeeded, i'd strongly recommend that you check your ram, because it would indicate that sometimes the verification fails even though the data on the disk is ok....

EDIT: it could still be your storage that is somehow broken if it sometimes gives back wrong data...

 

[Cayuga]

Thanks - I'll keep an eye on the memory (I checked this morning and there are no ECC/memory errors in the system logs or the HP-ILO logs).