Previously working proxmox-backup-client command now fails

F

Fischer-HWH

Member
Apr 21, 2022
12
0
6
Hi,

I have been using proxmox-backup-client to create backups, which I can write to tapes.
It's a pretty simple setup where I have a few SMB shares mounted locally (on the PBS itself), and then run a command in the style of (names changed obviously):
Bash: 
proxmox-backup-client backup archive1.pxar:/mnt/remotefolder1 archive2.pxar:/mnt/remotefolder2 archive3.pxar:/mnt/remotefolder3

This exact command was used in the past and was working fine, but it's now been not working for over a year, so I thought I'd ask here. Last successful run was august 2023.
Additionally, environment variables PBS_PASSWORD_FILE and PBS_REPOSITORY are set (the latter just contains the name of the datastore). Something about the file access seems to fail and it won't tell me where or with what path or anything. The shares are mounted read-only, but they always have been and it was fine.

Output is from the command looks as follows:
Code: 
Starting backup: host/MyPbsHostname/2025-03-19T13:51:42Z
Client name: MyPbsHostname
Starting backup protocol: Wed Mar 19 14:51:42 2025
storing login ticket failed: $XDG_RUNTIME_DIR must be set
Downloading previous manifest (Thu Aug 31 11:03:53 2023)
Upload directory '/mnt/remotefolder1' to 'archive' as archive1.pxar.didx
catalog upload error - channel closed
Error: failed to get metadata for source directory

Caused by:
    0: failed to read xattrs
    1: EACCES: Permission denied
 
Chris said:
This part of the error message suggests that you lack permissions on this folder.
Click to expand...
I have permissions on the folder. This was run as root, and the folder is mounted as root. I can list contents, read files. The shares come from a Windows host. I have a .pxarexclude, which I can also cat just fine. It excludes the usual (System Volume Information and $RECYCLE.BIN). I did spot-check for a bit, and I can read all files I tried, and list all folders. They aren't system-partitions, but all of them just contain data (so no Windows, ProgramData or Users folders on any of the shares)
Chris said:
Check the output of stat /mnt/remotefolder1, and getfacl /mnt/remotefolder1. Also check getfattr /mnt/remotefolder1
Click to expand...

Output below.

Code: 
# stat /mnt/remotefolder1/
  File: /mnt/remotefolder1/
  Size: 8192            Blocks: 16         IO Block: 1048576 directory
Device: 0,54    Inode: 1407374883553285  Links: 2
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-04-23 19:12:35.927161600 +0200
Modify: 2024-04-23 19:12:35.927161600 +0200
Change: 2024-04-23 19:12:35.927161600 +0200
Birth: 2022-05-08 00:24:03.228098500 +0200

I had to install acl and attr to get the remaining commands. Output below.

Code: 
# getfacl /mnt/remotefolder1
getfacl: Removing leading '/' from absolute path names
# file: mnt/remotefolder1
# owner: root
# group: root
user::rwx
group::r-x
other::r-x

Code: 
#getfattr /mnt/remotefolder1
getfattr: /mnt/remotefolder1: Permission denied

So how or why does that fail, exactly? And why is it suddenly a problem NOW, when it was working a fine a while ago? I mean I can't really give myself more permissions: I'm already root with owner and group also set to root and rwx set as well. The folders/shares are mounted from fstab with an entry roughly looking like:
Code: 
//ActualFileServer/remotefolder1   /mnt/remotefolder1  cifs credentials=/root/.smbcredentials,uid=0,gid=0,ro 0 0
The Windows-User used to access the share has read permission, and again it's mounted readonly anyway as it's just meant to be read and written to the backup.
 
Last edited:
Since there hasn't been any reply or answer anymore, am I right to assume that this is just weird and nobody knows why or how this is happening?

Is there some way to disable or circumvent the getfattr check? I mean I can't tell what it's supposed to do, and it failing just stops the backup from running for no reason that I can see. And again, it worked just fine a while ago with no changes to the mounts, shares, or configuration that I know of (updates to PBS notwithstanding).
 
Fischer-HWH said:
getfattr: /mnt/remotefolder1: Permission denied
Click to expand...
Fischer-HWH said:
Since there hasn't been any reply or answer anymore, am I right to assume that this is just weird and nobody knows why or how this is happening?
Click to expand...
It seems that reading extended file attributes fails on that shares mountpoint, so maybe you disabled xattr support for that share? E.g. samba allows to set ea support = yes to enable/disable extended attributes. Who provides the share in your case, did you check on the share provider configuration.

Fischer-HWH said:
Is there some way to disable or circumvent the getfattr check? I mean I can't tell what it's supposed to do, and it failing just stops the backup from running for no reason that I can see.
Click to expand...
This is not a check, extended attributes are part of a files metadata and important, therefore part of a backup. In your case however, reading of these attributes fails.
 
the underlying code has an option to skip xattrs (and other things), we just don't expose it in proxmox-backup-client (yet). it might make sense to offer those (`--no-XXX`) options for opting-out of including certain properties/parts of the backup input?
 
fabian said:
the underlying code has an option to skip xattrs (and other things), we just don't expose it in proxmox-backup-client (yet). it might make sense to offer those (`--no-XXX`) options for opting-out of including certain properties/parts of the backup input?
Click to expand...
Yes, the client exposes such flags for the restore already. Can have a look in exposing these for the backup as well, when time allows. @Fischer-HWH can you open an issue at https://bugzilla.proxmox.com/ with an enhancement request for exposing the `exclude-xattr` flag (among others) and referencing this thread, so you will get status notifications on progress, thanks!
 
Chris said:
It seems that reading extended file attributes fails on that shares mountpoint, so maybe you disabled xattr support for that share? E.g. samba allows to set ea support = yes to enable/disable extended attributes. Who provides the share in your case, did you check on the share provider configuration.
Click to expand...
I have absolutely no idea, sorry. I had potsed the fstab entry used to mount the shares above, and I'm not actively disabling anything in regards to extended attributes there. The configuration is whatever is default for PBS out of the box. And again: that entry (and the share) are unchanged from when the backup was working fine with exactly that command 1+ years ago. So either backing up extended attributes was added, or it somehow worked before?
Googling around for extended attributes and samba/cifs didn't yield any obvious results as to what I might need to do or change, or even how to check.

The output of mount for these shares looks like this, and I also don't see anything that would suggest xattrs are disabled here. The server hosting the shares is running Windows Server 2022, if that matters in any way.
Code: 
//ActualFileServer/remotefolder1 on /mnt/remotefolder1 type cifs (ro,relatime,vers=3.1.1,cache=strict,username=thesmbuser,domain=ActualDomainGoesHere,uid=0,noforceuid,gid=0,noforcegid,addr=192.168.66.60,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1,closetimeo=1)

The whole point of this is just to be able to use PBS to backup that data, and to be able to write it to tapes. If it was just about a backup copy I'd just rsync them or something. But to my knowledge this is basically the only way to do tape backups of generic "data" with PBS, by creating backups in a datastore that I can then write to tape.
 
Last edited:
Fischer-HWH said:
The server hosting the shares is running Windows Server 2022, if that matters in any way.
Click to expand...
Yes, that is what I intended with share provider. Does the file server or the filesystem where the data is located on maybe have extended attributes support disabled? it is supported by the SMB protocol, see https://learn.microsoft.com/en-us/o...s/ms-smb/65e0c225-5925-44b0-8104-6b91339c709f

Fischer-HWH said:
nounix
Click to expand...
your mount output shows the `nounix` flag, that disables the unix extensions for this mount. So maybe you could try to explicitly enable these in your mount options by explicitely setting unix and see if that allows you to back up data from the share?
 
Chris said:
Yes, that is what I intended with share provider. Does the file server or the filesystem where the data is located on maybe have extended attributes support disabled? it is supported by the SMB protocol, see https://learn.microsoft.com/en-us/o...s/ms-smb/65e0c225-5925-44b0-8104-6b91339c709f


your mount output shows the `nounix` flag, that disables the unix extensions for this mount. So maybe you could try to explicitly enable these in your mount options by explicitely setting unix and see if that allows you to back up data from the share?
Click to expand...
I guess the nounix is there for a reason, as trying to mount with specifying unix as a mont option leads to this (via dmesg):
Code: 
[32138292.051338] CIFS: VFS: Server does not support mounting with posix SMB3.11 extensions
[32138292.052143] CIFS: VFS: cifs_mount failed w/return code = -95

Edit forgot to answer the first part: as for the extended attributes being disabled server side that seems highly unlikely. I certainly didn't, and wouldn't know how or why, and nobody else here would either. The link is to the protocol spec, but even googling this seems to be so obscure that I can't figure out how to even check if it's enabled, let alone how to en- or disable it explcitly.

The Windows server installation is from 2022, and is also mostly default in terms of settings and setup (especially for file sharing). It is of course updated normally via windows updates, but not even configuration changes were done on that machine as far as I can remember: it's just a file server, it shares files. as long as that works nobody is gonna bother touching it.
 
Last edited:
One more update on "getfattr": I just tried to run it on files and folders on the mounted share, and there the command runs fine. It doesn't return anything, but also doesn't error out.

So the only command that fails (with permission denied error as described above) is getfattr /mnt/remotefolder1, but getfattr /mnt/remotefolder1/someFolder and getfattr /mnt/remotefolder1/someFolder/SomeFile both work perfectly fine. So it seems extended attribute support via SMB is working, but only fails on the mount point itself for some reason?
 
What about the top folder/network share as seen on the Windows side? Any difference in set of permissions or attributes as compared to other shares? (I assume also the other shares you backup up are hosted on Windows)
 
Yes, all shares to be backed up are from the same server. Even compared to shares from other servers, including normal PCs/workstations, they look the same. Permissions are there (ACL), and all tabs in the dialog reachable from the security tab when clicking on advanced are filled like normal. This includes the share permissons (on the actual share, which are distinct from the permissions on the file system). The permissions for one of the shares are also very broad, almost universal, as that's the kind of shares these are (for general data exchange and such).

When mounting some random other shares (from servers and normal windows client machines), the behavior is the same: I can getfattr a file/folder in the share, but not for the folder the share is mounted to. So this is not unique to shares mounted from that server in particular, but applies to any share.

I have also mounted some shares, including the ones that I want to back up, on a PVE host that we have. There the getfattr command completes just fine even for the shares mount folder. The exact same mount command was used when comparing, so this includes the same (windows) user for accessing the share and all options. So it's definitely an issue on this PBS machine in particular, and ONLY on this machine. Both PBS and PVE are running natively (not virtualized) and are reasonably current in terms of updates.

I have no recollection of ever changing any smb-related settings on the PBS (or why I would even want to), or which setting might even affect this. It seems like such a niche issue, since accessing the share works completely fine.
 
Fischer-HWH said:
I have also mounted some shares, including the ones that I want to back up, on a PVE host that we have. There the getfattr command completes just fine even for the shares mount folder. The exact same mount command was used when comparing, so this includes the same (windows) user for accessing the share and all options. So it's definitely an issue on this PBS machine in particular, and ONLY on this machine. Both PBS and PVE are running natively (not virtualized) and are reasonably current in terms of updates.
Click to expand...
Did you already try to use a completely different mountpoint on the pbs, e.g. create a temp dir in /tmp and mount the share to that, then test the getfattr call
 
I hadn't until now, but the other mounts (from the other machines) were tested on other dirctories, but all were in /mnt. I have now tested creating a directory in /tmp/ to mount to, and still got the same issue there as well, so that isn't it either (of course also only fails if anything is mounted there): getfattr: /tmp/test-mount/: Permission denied
 
Please post the output of proxmox-backup-manager version --verbose. One thing you could check further is see if the same happens when booting the PBS with a different kernel, e.g. install the opt-in Linux kernel version 6.14 via apt install proxmox-kernel-6.14 and boot selecting that.
 
Here is the requested output:
Code: 
# proxmox-backup-manager version --verbose
proxmox-backup                     3.4.0         running kernel: 6.5.11-7-pve
proxmox-backup-server              3.4.0-1       running version: 3.3.7
proxmox-kernel-helper              8.1.1
pve-kernel-5.13                    7.1-9
proxmox-kernel-6.8                 6.8.12-9
proxmox-kernel-6.8.12-9-pve-signed 6.8.12-9
proxmox-kernel-6.8.12-8-pve-signed 6.8.12-8
proxmox-kernel-6.5.13-6-pve-signed 6.5.13-6
proxmox-kernel-6.5                 6.5.13-6
proxmox-kernel-6.5.11-7-pve-signed 6.5.11-7
pve-kernel-5.13.19-6-pve           5.13.19-15
pve-kernel-5.13.19-2-pve           5.13.19-4
pve-kernel-5.13.19-1-pve           5.13.19-3
ifupdown2                          3.2.0-1+pmx11
libjs-extjs                        7.0.0-5
proxmox-backup-docs                3.4.0-1
proxmox-backup-client              3.4.0-1
proxmox-mail-forward               0.3.2
proxmox-mini-journalreader         1.4.0
proxmox-offline-mirror-helper      0.6.7
proxmox-widget-toolkit             4.3.10
pve-xtermjs                        5.5.0-2
smartmontools                      7.3-pve1
zfsutils-linux                     2.2.7-pve2

I have now installed the opt-in 6.14 Kernel. This actually seems to have instantly fixed the issue with getfattr. It also wasn't just the reboot, as I'd previously rebooted the machine as well, just cause why not.
I am currently running the same backup command that was used in previous runs. So far it's running fine (it'll take a while to complete of course, but I'm optimistic).

Thank you very much for your time, and patience. I would still be curious as to why it broke, and how/what exactly the issue is, but I'll guess we'll never know?
 
You must log in or register to reply here.
Top Bottom