apply rrd journal failed - stream did not contain valid utf-8

[chrispage1]

Hi,

Our backup servers RAID array hit a bit of an issue this morning and it ultimately caused the PBS to crash.

Now I can't view the datastore and my journal is giving the following errors -

Jun 24 04:01:52 pbs proxmox-backup-proxy[191]: GET /api2/json/admin/datastore/default/status/?verbose=true: 400 Bad Request: [client [::ffff:217.>
Jun 24 04:01:55 pbs proxmox-backup-proxy[191]: apply rrd journal failed - stream did not contain valid UTF-8
Jun 24 04:01:55 pbs proxmox-backup-proxy[191]: unable to parse rrd journal 'rrd.journal' line 165 (skip) - unable to parse time
Jun 24 04:01:58 pbs proxmox-backup-proxy[191]: GET /api2/json/admin/datastore/default/status/?verbose=true: 400 Bad Request:

I've had a look at my journal file and there is a line with a big blob of invalid text (as attached).

So it definitely seems there is a level of corruption going on here - I'm hoping this file was just mid write (as there was a large backup going on).

Is there a way to recover from this without losing the datatstore?

Thanks!

 

[Chris]

Hi,

Our backup servers RAID array hit a bit of an issue this morning and it ultimately caused the PBS to crash.

Now I can't view the datastore and my journal is giving the following errors -

Jun 24 04:01:52 pbs proxmox-backup-proxy[191]: GET /api2/json/admin/datastore/default/status/?verbose=true: 400 Bad Request: [client [::ffff:217.>
Jun 24 04:01:55 pbs proxmox-backup-proxy[191]: apply rrd journal failed - stream did not contain valid UTF-8
Jun 24 04:01:55 pbs proxmox-backup-proxy[191]: unable to parse rrd journal 'rrd.journal' line 165 (skip) - unable to parse time
Jun 24 04:01:58 pbs proxmox-backup-proxy[191]: GET /api2/json/admin/datastore/default/status/?verbose=true: 400 Bad Request:

I've had a look at my journal file and there is a line with a big blob of invalid text (as attached).

So it definitely seems there is a level of corruption going on here - I'm hoping this file was just mid write (as there was a large backup going on).

Is there a way to recover from this without losing the datatstore?

Thanks!

Hi,
seems like our error handling does not skip over lines for cases where non-utf8 contents are dumped to the rrd journal, a was the case for the crash (note that this is not the systemd journal, you find the rrd journal files under /var/lib/proxmox-backup/rrdb). You might want to check and remove non-utf8 content in rrd.journal

I sent a patch for this to the developer mailing list [0], this does however not lead to the 400 bad request error in my test case. So maybe something else is still off after the crash? I noticed that there is increased system journal output in my case so maybe you ran out of space?
Please do check the response for these requests in your browsers development tools in the network tab by selecting the corresponding datastore summary page.

[0] https://lists.proxmox.com/pipermail/pbs-devel/2024-June/009954.html