Problem with zfs and setfattr

S

skraw

Well-Known Member
Proxmox Subscriber
Aug 13, 2019
105
1
58
59
Hello,

I just came across a problem with zfs handling of xattrs. My zfs has xattr=sa and is mounted with xattr. But using setfattr on a file results in "operation not supported". Why is this?
 
You didn't say what xattr you were trying to set, so I can only guess that you are getting this message because you aren't naming your attribute in the user namespace. If you look in "man xattr" there are several namespaces that have special meaning to the system, and there is the user namespace for any attribute you want to set that doesn't have special meaning to the system. Some filesystems handle this differently but as far as I know, zfs doesn't let normal users set xattrs outside of the user namespace.

So this will fail with "Operation not supported":
Code: 
touch test.file
setfattr -n test -v "test" test.file

But this will succeed:
Code: 
touch test.file
setfattr -n user.test -v "test" test.file 
getfattr -n user.test test.file
 
Well, in fact I am coming from another side to this problem. I take backup by rsync from a lot of hosts. when the backup drive was built into the backup server (with ext4 fs) everything went without problems. Now I virtualised the backup server and replaced this drive with an nfs-mounted zfs from a proxmox server and had to find that almost all hosts have rsync problems like this one:

rsync: [receiver] rsync_xal_set: lsetxattr("/usr/bin/ping" (in some-host),"security.capability") failed: Operation not supported (95)

Since this is nfs 4.2 and all tools tell me that nfs is indeed handling the xattrs I tried from command line on backup server, and it does not work. So it is no rsync problem. Then I went back to the proxmox exporting the zfs and all docs about zfs tell me that with xattr=sa this should work. But it does not. Even on the server console a setfattr does not work.
Btw all this was done as root. So there should be no some-user-cant-do-this.

The basic question is: how can I replace a hardware drive with ext4 fs by nfs over zfs, so that it is really compatible?

Btw I assumed that this is a common problem in virtualised environments, but there is no real info about it...
 
skraw said:
rsync: [receiver] rsync_xal_set: lsetxattr("/usr/bin/ping" (in some-host),"security.capability") failed: Operation not supported (95)
Click to expand...
So not quite what I was thinking. The property security.capability is in the security namespace for xattrs, which I believe can be controlled by a security extension in the kernel (like selinux or apparmor). Even root can't write to that without permission of that security system. I think if you don't have one of those in your kernel at all then you can write anything you like to the security xattr namespace, but it's been a long time since I played with that, so I am not sure.

The normal way to set the capabilites that are stored in the security.capability xattr is with the capset or setcap command, and it displayed with the getcap command. These xattrs look like random text in getfattr. It might be nice to know that if you want to make a small test case.

I have 2 ideas you can try. First one is to make a small zvol (a block device inside of zfs) and create on it an ext4 filesystem to test on. The basic steps would be something like this
Code: 
zfs create -V 20G tank/test_ext4
mkdir /mnt/test_ext4
mount /dev/zvol/tank/test_ext4 /mnt/test_ext4 -o user_xattr
cd /mnt/test_ext4
touch test_file
setcap 'cap_net_bind_service=+ep' test_file
And if that works, try to rsync just that ping file over.

I think that will fail because I think writing to that xattr is being denied by the extra security module in the kernel and not the filesystem, but it would be interesting to try it anyway, since if it works that might be your easiest option (except with an adequate size zvol, etc). To clean up after that, unmount that and then zfs destroy that zvol.


The other option is to use the --fake-super on rsync. You need to use this on the receiving side (so you might need to specify it as -M --fake-super if you are running rsync from the sending machine). This option tells rsync not to do things that need root, but instead fake it with some xattrs. So instead of changing the owner of a file, it will store the owner info in an xattr (in the user namespace). I believe that also means that your source file's security.capability xattr will be stored as user.rsync.security.capability (or something like that), so the writing should succeed.

Make sure you try a restore too. When restoring a backup, you will need the --fake-super on the sending rsync. That way the info in those special rsync xattrs will get translated back to their original values (so the restored file will get back the original owner).
 
Last edited:
  • Like
Reactions: Onslow
First of all let me say thank you for the time you take to answer in this thread.
Unfortunately this does not quite make it to a point where I want to go. Since the backup host is exactly the same as it was when using a local ext4 drive. So I would think that there must be some way to let it do the same thing (without problems) it did before the virtualisation.
So, when I replace an ext4 with an NFS-mounted zfs, and all parts tell in the docs that xattrs should work, why these errors in the logs now? and what is the precise reason for them? I mean NFS 4.2 clearly states it is capable of xattrs. zfs even has some options to enhance the speed (sa).
And the rsync (receiver daemon) is now root as it has been before. I see no need to fake-super if I am already root. And there is no selinux or apparmor either, never has been.
I ran across other peculiarities before in this setup. It seemed that sometimes (and I have not found a cause until now) the file dates get trashed to current time during rsync. This leads to the problem that a client rsync retransmits all data again, every time. Although the files did not change, but the filedate on the backup server got "trashed" by the rsync-daemon to current. The clients always used avxAHX since decades and there has never been a problem.
Interestingly there is none now if I set the destination away from the nfs drive to a "local" ext4. Then nothing gets trashed and xattrs get set.
Honestly, I don't get it..
 
I am sorry, I guess I wasn't clear. I didn't mean that either of those options are a fix. Those are things you can do quick to get backups working. They might also provide some additional information that could help determine if this is a bug in rsync, a bug in zfs, or a result of something that apparmor is doing.

And yes, AFAIK if apparmor is loaded, then it has to approve any wrote or modification to the "security" xattrs. So for example, there might be a rule in apparmor that allows access to the security xattrs for the setcap command, but not rsync or setfattr or nfsd. I don't know for sure that is what is happening here. Like I said it seems possible that there's a bug in the way zfs handles xattrs (though it doesn't feel likely, it is possible).
 
skraw said:
Ok, I did not think about that so far, is it possible that the problem is related to apparmor on proxmox?
Click to expand...
Not likely. the error is from the destination file system, not the source.

While it does sound like an interesting problem, what is the use case for this? tar.zstd will take the backup, compress it regardless of destination capabilities, and would be 100% native restorable whereas rsync is slower, heavier, and more accessible for granular modification at the destination, none of which are desirable.

Or, you know, just use pbs.
 
alexskysilk said:
Not likely. the error is from the destination file system, not the source.

While it does sound like an interesting problem, what is the use case for this? tar.zstd will take the backup, compress it regardless of destination capabilities, and would be 100% native restorable whereas rsync is slower, heavier, and more accessible for granular modification at the destination, none of which are desirable.

Or, you know, just use pbs.
Click to expand...
Hm, does not sound like you understood the problem setup. Maybe I did not explain well.
I have a VM backup guest. This is the destination for numerous rsync backups from _elsewhere_. The advantage in using rsync is exactly to be able to access all files in the backups as single files, e.g. for copying/restoring back maybe one or few of them. The advantage of using zfs here is that it makes snapshots every night, so that I can hold hundreds of days back a completely accessible file oriented backup history. Tar is not really an option here. And compression by whatever (zstd or zip or something else) is not useful either because zfs does that on its own. And on top of that is dedup:

NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
backup 58.2T 55.0T 3.19T - - 76% 94% 4.51x ONLINE -

We are not talking about guest vms to backup here.
 
Last edited:
skraw said:
Hm, does not sound like you understood the problem setup. Maybe I did not explain well.
I have a VM backup guest. This is the destination for numerous rsync backups from _elsewhere_. The advantage in using rsync is exactly to be able to access all files in the backups as single files, e.g. for copying/restoring back maybe one or few of them. The advantage of using zfs here is that it makes snapshots every night, so that I can hold hundreds of days back a completely accessible file oriented backup. Tar is not really an option here. And compression by whatever (zstd or zip or something else) is not useful either because zfs does that on its own. And on top of that is dedup:

NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
backup 58.2T 55.0T 3.19T - - 76% 94% 4.51x ONLINE -

We are not talking about guest vms to backup here.
Click to expand...
Makes sense.

I think that leaves you with these options:
- zfs set xattr=on
- zvol with originating filesystem
- or drop xattr at the rsync level.
 
You must log in or register to reply here.
Top Bottom
Back