PBS Task history creating "bad" files at times

[BloodyIron]

I've recently had a need to copy off a PBS system the Task History Logs and a bunch of those Task History files have been created with "bad" naming conventions. This has caused me problems for syncing the files onto my nextCloud, and I'm sure you'll see the problem with this example:

"UPID:redacted:000289CB:3BEF2DE3:000003B1:64B24022:backup:Redacted/x3act-182:root@pam:"

That is the full, and exact, (apart from redacted) filename. The problem probably lies to there being a slash in the filename at all.

I don't know why these files are being generated with slashes at all, as that causes filesystem interpretation problems. Such as, it can be interpreted that it's a folder before the slash. So naturally copying onto nextCloud fails because it cannot create this file, as its very naming is nonsensical.

I just wanted to pass this along so the developers can improve PBS to avoid creating any files that are nonsensical like this. I can't really theorise why a slash was used at all by PBS to create any files. Really hoping this gets fixed.

Thanks for hearing me out!

 

[dcsapak]

are you sure that it's a '/' and not a backslash '\' ?

can you post the output of 'ls -lh' of the directory containing such a file?

a '/' in filenames is not even possible on linux AFAIK

what we do here is using systemd's escape mechanism to encode special characters such as /,: etc. but this should never produce a forward slash in the output

 

[BloodyIron]

I copied the name of the file exactly (except changing two sensitive words to redacted, which did not alter any slash characters at all in the process).

So yes, I am 100% confident the slash is as I have shown above. That is what was in the filename both before and after the redaction.

There are multiple other files of the same nature in that folder, and this is client data so I do not want to paste an "ls -lh" as that would be sensitive information.

As to how this came to be, I cannot speculate, I simply am reporting what I saw/see. I was not involved in the setup of the Proxmox Backup Server that this came from, as I am stepping in as a "hire muscle" support person, so to say.

 

[dcsapak]

can you post the versions?
proxmox-backup-manager versions --verbose

 

[BloodyIron]

root@redacted:~# proxmox-backup-manager versions --verbose
proxmox-backup unknown running kernel: 6.1.2-1-pve
proxmox-backup-server 2.4.6-1 running version: 2.4.3
pve-kernel-helper 7.3-8
pve-kernel-6.1 7.3-6
pve-kernel-6.1.15-1-pve 6.1.15-1
pve-kernel-6.1.2-1-pve 6.1.2-1
ifupdown2 3.1.0-1+pmx4
libjs-extjs 7.0.0-1
proxmox-backup-docs 2.4.6-1
proxmox-backup-client 2.4.6-1
proxmox-mail-forward 0.1.1-1
proxmox-mini-journalreader 1.2-1
proxmox-offline-mirror-helper unknown
proxmox-widget-toolkit 3.7.3
pve-xtermjs 4.16.0-2
smartmontools 7.2-pve3
zfsutils-linux 2.1.14-pve1

 

[dcsapak]

mhmm looks somewhat ok ? i guess you have pbs 2 installed on a pve 7 with the 6.1 opt in kernel? (i would strongly suggest to upgrade, since the opt-in kernel does not get an update anymore (change to 6.2) and pve7/pbs2 are EOL this summer)

but as i said i don't think it's possible on linux to get a / in the filename

do you have any special filesystem in /var/log ?

can you maybe check one of those files with 'stat' (ofc you can anonymize the filename, if necessary)

 

[BloodyIron]

Oh I'm with you about upgrading that system, but to be clear this system isn't "mine" but one that I have provided support for. So I will be convincing about upgrades likely in the near future, but "how soon" is probably not "soon".

The underlying filesystem for the Datastore is btrfs (not set up by me, yuck), so unsure if that's relevant to the slash aspect.

As for "what is possible", well.... I have birds in my hand here so to say (these files were created by PBS through general operations so far as I can tell). So while it may be "should not have happened", it surely has happened, and again is also part of why I'm posting here. To help $futureHumans have a $betterTomorrow2.

As for the filesystem for /var/log, last I checked that's ext4 but I do not currently have access to that system.

As for "stat"-ing the file, any particular parameters I should use? (unsure when I can get into that system next)

 

[dcsapak]

As for "stat"-ing the file, any particular parameters I should use? (unsure when I can get into that system next)

no just simply `stat /path/to/file`

if possible as little modification as possible please

also you mentioned

This has caused me problems for syncing the files onto my nextCloud,

what exact error did you get?

 

[BloodyIron]

no just simply `stat /path/to/file`

if possible as little modification as possible please

also you mentioned
what exact error did you get?

I don't know exactly when I will be able to get back onto that system next, but I will try my darndest to remember.

As for the error... "Could not create path" since it interprets the slash in the file as referencing "wanting" to make a folder.

 

[Dunuin]

Maybe its not PBS screwing up chunk file names? I remember 2 threads where people wanted migrate a datastore between two PBS installations without sync jobs but by copying the datastore folder. And while copying the files a special character in the filename got replaced with another one (I think it had something todo with the network share of the NAS or similar) so the new PBS wasn't able to recognize any chunk files.

 

[BloodyIron]

Maybe its not PBS screwing up chunk file names? I remember 2 threads where people wanted migrate a datastore between two PBS installations without sync jobs but by copying the datastore folder. And while copying the files a special character in the filename got replaced with another one (I think it had something todo with the network share of the NAS or similar) so the new PBS wasn't able to recognize any chunk files.

I have total confidence that these files were created by PBS, all evidence in-hand shows it. And the scenario you describe has not happened for this relevant PBS system.

 

[mow]

I see a very similar filename, but with backslash instead:
'UPID:pvebs:0015178D:03A71DFA:00000002:637686D1:backup:pbsroot\x3act-101:pvebs@pbs:'
ls helpfully puts '' around it.

a '/' in filenames is not even possible on linux AFAIK

... from the shell, because all executables that take a file as parameter, including the shell itself, will split it at the last /. (While the fs should block invalid characters, such as \ on ntfs.)

 

[BloodyIron]

I see a very similar filename, but with backslash instead:
'UPID:pvebs:0015178D:03A71DFA:00000002:637686D1:backup:pbsroot\x3act-101:pvebs@pbs:'
ls helpfully puts '' around it.

... from the shell, because all executables that take a file as parameter, including the shell itself, will split it at the last /. (While the fs should block invalid characters, such as \ on ntfs.)

Sounds like whatever is resulting in that outcome is probably resulting in the same thing on my end too. And yeah it looks like you're also speaking to my original concern that the invalid slash characters should have been blocked before file creation (and I'm not even sure why Proxmox Backup Server tried to do so, it needs to not do that).

 

[fabian]

files with escaped parts (using \xAA notation) are normal and expected (when encoding the task ID/UPID, we escape everything but '_', '.', digits and letters a-z, with everything but '/' being escaped using that notation ('/' is escaped as '-').

I still don't see how files with (unescaped) '/' in their name could be created by our code, since the kernel would block that.

that basically leaves a few options:
- some other OS had access to the disk and mistreated those paths
- a systematic corruption of the on-disk file system data occurred that replace some other character with '/'
- the character in question is not a real '/' but some unicode character that looks like one (unlikely, see escape rules mentioned above)

you mentioned NextCloud before - is it possible that it is the culprit and somehow mangles filenames with backslashes in them? in your original post, was the filename copied from "ls" or similar output on the Linux side, or from a NextCloud log/error? if you still have such paths around, could you try to access their task status/log via the PBS UI/API and report whether that works?

 

[BloodyIron]

files with escaped parts (using \xAA notation) are normal and expected (when encoding the task ID/UPID, we escape everything but '_', '.', digits and letters a-z, with everything but '/' being escaped using that notation ('/' is escaped as '-').

I still don't see how files with (unescaped) '/' in their name could be created by our code, since the kernel would block that.

that basically leaves a few options:
- some other OS had access to the disk and mistreated those paths
- a systematic corruption of the on-disk file system data occurred that replace some other character with '/'
- the character in question is not a real '/' but some unicode character that looks like one (unlikely, see escape rules mentioned above)

you mentioned NextCloud before - is it possible that it is the culprit and somehow mangles filenames with backslashes in them? in your original post, was the filename copied from "ls" or similar output on the Linux side, or from a NextCloud log/error? if you still have such paths around, could you try to access their task status/log via the PBS UI/API and report whether that works?

1. No other operating system has direct access to these disks to any capacity. The storage that PBS uses is local to the OS.
2. The observed naming with slash is observed for multiple backup files, this is not a one-off.
3. nextCloud in no way has mangled the filenames. The involvement was manual, and completely external to the PBS. In that, I had copied the files _off_ the PBS to a computer local to me (as in, not the PBS itself) and then uploaded it to nextCloud. In no way could nextCloud have malformed the files _before_ they were uploaded, and they failed to upload _because_ of the slash in the filename.

The file was copied via the SCP protocol and if I remember correctly I used FileZilla to download it, which last I checked uses a compliant implementation of SCP. And the files _on_ the PBS still have the slash as demonstrated.

As for which "paths" and tasks, I'm not entirely understanding what you want me to check there. Your description doesn't quite make sense. If you could rephrase that would be appreciated.

 

[fabian]

this here "UPID:redacted:000289CB:3BEF2DE3:000003B1:64B24022:backup:Redacted/x3act-182:root@pam:" is a (malformed cause of the '/') "UPID", which is the identifier of a task execution. there should be a corresponding task - I wonder whether
- the corresponding non-malformed path exists (with '/' replaced by '\')
- the corresponding task status and log is queriable over the GUI/API/CLI (e.g., you can check with "proxmox-backup-manager task list --all" and "proxmox-backup-manager task log 'UPID VALUE'")

your original post mentions that you noticed the issue while trying to sync the path in question to NextCloud. so if I understand correctly, then the sequence of events was the following:
- scp PBS -> "local computer"
- NextCloud sync of dir on local computer failed

which OS is running on the local computer?

And the files _on_ the PBS still have the slash as demonstrated.

well, a "stat PATH" or "ls -lh PATH" (or even better, something like find PATH -print -exec sh -c 'echo -n "{}" | xxd -pu' \;, which will print the filename and its bytes in hex) would help shed some light on this, because so far, you haven't actually demonstrated this in any fashion, just said it is so this would also show if the filename in question just looks like it has a slash in it (i.e., because of a unicode character looking like a slash). I am still fairly sure something other than PBS messed with those filenames - there is no way for regular applications to create files with a slash in their name on Linux in general, or BTRFS in particular, and PBS itself doesn't allow and has never allowed a slash in the string representation of a UPID either.

e.g., the following does work: "touch foo∕bar" (this contains the division slash U+2215) while this doesn't: "touch foo/bar" (regular slash)

 

[BloodyIron]

this here "UPID:redacted:000289CB:3BEF2DE3:000003B1:64B24022:backup:Redacted/x3act-182:root@pam:" is a (malformed cause of the '/') "UPID", which is the identifier of a task execution. there should be a corresponding task - I wonder whether
- the corresponding non-malformed path exists (with '/' replaced by '\')
- the corresponding task status and log is queriable over the GUI/API/CLI (e.g., you can check with "proxmox-backup-manager task list --all" and "proxmox-backup-manager task log 'UPID VALUE'")

your original post mentions that you noticed the issue while trying to sync the path in question to NextCloud. so if I understand correctly, then the sequence of events was the following:
- scp PBS -> "local computer"
- NextCloud sync of dir on local computer failed

which OS is running on the local computer?



well, a "stat PATH" or "ls -lh PATH" (or even better, something like find PATH -print -exec sh -c 'echo -n "{}" | xxd -pu' \;, which will print the filename and its bytes in hex) would help shed some light on this, because so far, you haven't actually demonstrated this in any fashion, just said it is so this would also show if the filename in question just looks like it has a slash in it (i.e., because of a unicode character looking like a slash). I am still fairly sure something other than PBS messed with those filenames - there is no way for regular applications to create files with a slash in their name on Linux in general, or BTRFS in particular, and PBS itself doesn't allow and has never allowed a slash in the string representation of a UPID either.

e.g., the following does work: "touch foo∕bar" (this contains the division slash U+2215) while this doesn't: "touch foo/bar" (regular slash)

Local computer in particular example is/was Ubuntu 23.10.

I keep meaning to do the stat thing but have not yet, that's on my end, sorry! I need to do that.

 

[BloodyIron]

no just simply `stat /path/to/file`

if possible as little modification as possible please

also you mentioned
what exact error did you get?

root@REDACTED:/var/log/proxmox-backup/tasks/E3# stat 'UPID:REDACTED:000289CB:3BEF2DE3:000003B1:64B24022:backup:REDACTED\x3act-182:root@pam:'
  File: UPID:REDACTED:000289CB:3BEF2DE3:000003B1:64B24022:backup:REDACTED\x3act-182:root@pam:
  Size: 455331          Blocks: 896        IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 6428481     Links: 1
Access: (0644/-rw-r--r--)  Uid: (   34/  backup)   Gid: (   34/  backup)
Access: 2024-03-05 14:34:17.376729619 -0700
Modify: 2023-07-15 00:44:52.509272617 -0600
Change: 2023-07-15 00:44:52.509272617 -0600
 Birth: 2023-07-15 00:43:46.220776104 -0600

Uhhh hopefully that forum formatting comes out nice. But yeah that's the REDACTED result.

 

[dcsapak]

REDACTED\x3act-182

so as i asked in my first answer, this is a backslash '\' and not a forward slash '/'
and that is perfectly fine in filenames on linux...

 

[BloodyIron]

so as i asked in my first answer, this is a backslash '\' and not a forward slash '/'
and that is perfectly fine in filenames on linux...

I honestly cannot explain how the slash changed between the stat command, and my first comment. I would not have intentionally changed that slash.