Is there any kind of incompatible data?

[omgs]

Hi. I'm using PBS to backup my data. There's a ct running a nextcloud instance, and that ct has a separate mount point for the data (~100GB). I run at os level the proxmox backup client for that directory, and found out that there are errors that I'm currently debugging. Also, take into account that users can run into "bad practice", but that shouldn't be a technical problem from a datastore storage point of view as long as ext4 can manage the file. SMART says the disks are OK.

At this time I get errors like

catalog upload error - channel closed
Error: inserting chunk on store 'XXXX' failed for 26fb44e8d7f18c2776a2a515b3ce773a94a3136931b400e6aec4c0f7cab37f51 - mkstemp "/mnt/datastore/XXXX/.chunks/26fb/26fb44e8d7f18c2776a2a515b3ce773a94a3136931b400e6aec4c0f7cab37f51.tmp_XXXXXX" failed: EUCLEAN: Structure needs cleaning

and I don't know which file is having the problem, so I'm testing with trees in a test and error work.
I've found that a user has uploaded a copy of the thunderbird folder (.thunderbird), and this causes problems for pbs.

Also, I've had to create a tar with the whole data, restored this in a separate lv for these tests. First, I formatted the lv with ext4, and run into several weird issues with tested files becoming corrupted, so I'm repeating using now btrfs, and I'm currently on the way, finding the issues explained above, and surely more yet to come.


Any help is appreciated. Thanks in advance.

 

[Chris]

Hi,
this error points towards an issue with the filesystem backing your datastore storage. Please run a fsck on that filesystem.

 

[omgs]

Hi,
this error points towards an issue with the filesystem backing your datastore storage. Please run a fsck on that filesystem.

Do you mean fsck on datastore or on where the data is stored? As I said, it's recentrly reformatted in btrfs and I've checked it. Unless you mean it can become corrupted within a short period of time with no use.

 

[Chris]

Do you mean fsck on datastore or on where the data is stored? As I said, it's recentrly reformatted in btrfs and I've checked it. Unless you mean it can become corrupted within a short period of time with no use.

You should run it on the btrfs filesystem (the datastore is agnostic to the underlying filesystem). Also please check if you see errors in the PBS dmesg output related to btrfs and post your promox-backup-manager version --verbose output.

Far fetched, but this could also be related to something like https://tavianator.com/2024/btrfs_bug.html

 

[mow]

Could also be bad RAM. Do you have ECC?

 

[omgs]

Well, I found the problem. The underlying fs for datastore had problems. I noticed when started verifying the jobs, and all were corrupt indicating errors in the datastore. I've repaired the fs and created a new datastore, and I'm doing them all and verifying. I hope this time it's ok.

 

[Chris]

Well, I found the problem. The underlying fs for datastore had problems. I noticed when started verifying the jobs, and all were corrupt indicating errors in the datastore. I've repaired the fs and created a new datastore, and I'm doing them all and verifying. I hope this time it's ok.

Yes, so that was the initial suspicion, would you be willing to share the steps you took to resolve the errors for others to find? I think not many users use btrfs as storage backing the datastore.

 

[omgs]

Maybe I didn't explain clearly. I used btrfs as second step for temporarily storing a manual backup of the source data, once that the first temporary lv was with ext4 and showed corruptions. Then, I unmounted and repaired the failing failsystem, and mounted it back after checking. The datastore, according to the docs, doesn't support btrfs, just ext4, zfs or xfs.
In order to start over, I didn't find in the gui the option to remove or rename the datastore, so I edited manually the cfg file, renamed there the storage and the mount point. Then, I renamed the directory, created a datastore with the original name. Once it started working and sucessfully verified the first backup, deleted manually the entry in the cfg and removed the old tree.
I've done all the backups again and everything seems ok now, getting rid of the weird messages.

Thanks for your help.