No previous backup found, cannot do incremental backup

[SiJux]

I'll try that in the next days.

 

[Chris]

Please do also check the systemd journal or dmesg output on the PBS host for errors related to CephFS around the time of the backups.

 

[Chris]

I'll try that in the next days.

Another user in the other thread regarding this issue tried to use the ceph-fuse client, and reported that this fixed the issue for him. So while the fuse client might be used as a workaround, it would be nice to find the root cause. If anyone able to reproduce this issue is willing to debug, please try to generate a dynamic debugging output for the in-kernel ceph client as described here https://forum.proxmox.com/threads/backup-suceeds-but-ends-up-failing-verification.149249/post-690579

 

[SiJux]

Hi Chris,

I've switched now to fuse-mounts. The issue is fixed now. Also the backups are faster.
I'll try to collect the debug-Information and provide them to you.

Best regards
Simon

 

[Chris]

Hi Chris,

I've switched now to fuse-mounts. The issue is fixed now. Also the backups are faster.
I'll try to collect the debug-Information and provide them to you.

Best regards
Simon

Hi,
please do also try to see if the issue is present if booting into kernel 6.5.13-6-pve on the PBS, using the in-kernel cephfs client.

 

[SiJux]

I'll try.

 

[SiJux]

please do also try to see if the issue is present if booting into kernel 6.5.13-6-pve on the PBS, using the in-kernel cephfs client.

after switching to proxmox-kernel-6.5.13-5-pve (proxmox-kernel-6.5.13-6-pve is not available in the repo) the backup works fine again with in-kernel cephfs.

Best regards
Simon

 

[SiJux]

Another user in the other thread regarding this issue tried to use the ceph-fuse client, and reported that this fixed the issue for him. So while the fuse client might be used as a workaround, it would be nice to find the root cause. If anyone able to reproduce this issue is willing to debug, please try to generate a dynamic debugging output for the in-kernel ceph client as described here https://forum.proxmox.com/threads/backup-suceeds-but-ends-up-failing-verification.149249/post-690579

Chris: Do you need the described debug-information?

Best Regards
Simon

 

[Chris]

Chris: Do you need the described debug-information?

Best Regards
Simon

Please do not go trough the effort as of now, I am currently investigating an issue with the in-kernel cephfs client which might be related. Allow me to come back to you if more information is needed or this turns out to be unrelated.

Thanks a lot for your availability!

 

[Chris]

Hi,

the CephFS issue currently under investigation is related to the readahead size, as configurable when mounting the ceph filesystem via the rasize option [0]. Please check if your issues persist even if the CephFS backing the datastore is mounted by adding an additional rasize=0. This would help in gaining confidence that the two issues are indeed related, thx!

[0] https://docs.ceph.com/en/reef/man/8/mount.ceph/#advanced

 

[Chris]

Hi,
please try to see if the fixed index file corruption persists on the latest proxmox-kernel in version 6.8.12-3, which at the time of writing is available in the PBS test repository [0]. This version includes a tentative fix for a related issue.
Required steps to get the latest kernel are:

1. Adapt the sources to enable the pbstest repo as described in [0]
2. apt update
3. apt install proxmox-kernel-6.8
4. reboot

Please do not forget to reboot into the new kernel after installing the upgrade, feedback is very much appreciated!

[0] https://pbs.proxmox.com/docs/installation.html#proxmox-backup-test-repository

 

[SiJux]

Hi Chris,

we have tested now and the problem is not reproducible right now.

We will have a look on it.

Best regards
Simon