Samba cephfs gateway fails erraticly when reading/writing xattr

[Thomas Hukkelberg]

Hi

We run samba in privileged containers with CTDB utilizing CephFS storage. Samba is version 4.12.8-SerNet-Ubuntu-8.focal running on ubuntu 20.04 in Proxmox LXCs and Ceph is version 14.2.11 also running on Proxmox 6.2. The CephFS volumes is bind-mounted into the container and shared with Samba.

We utilize the vfs_fruit module in conjunction with vfs_streams_xattr and lately we have experienced that when copying files to the samba share with macOS Finder, files are being randomly deleted after successful copy or marked grayed out.

When disabling the vfs_fruit + vfs_streams_xattr modules we don't experience problems, but performance is awful in comparison -- and also Finder is creating the dreaded dot-files (._) that burden the filesystem...

Another test we tried was to bind-mount a local ext4 folder directly from the hypervisor into the container -- then we don't experience these problems at all, even with fruit enabled.

The issue with grayed out files is due to the fact that the extended attribute "user.DosStream.AFP_AfpInfo:$DATA" is not being updated/removed after the file is copied. Manually removing the xattr 'AFP_AfpInfo' makes the file become avaliable to the user.


In log.smbd we see the following errors:

[2020/10/19 14:49:08.694441,  0] ../../source3/modules/vfs_fruit.c:2200(fruit_pread_meta_stream)
 fruit_pread_meta_stream: Removing [Endclient/2020/20260_projectname/Icons/filename_icon_v3.ai:AFP_AfpInfo] after short read [0]
[2020/10/19 14:49:08.695018,  0] ../../source3/modules/vfs_fruit.c:2208(fruit_pread_meta_stream)
 fruit_pread_meta_stream: Removing [Endclient/2020/20260_projectname/Icons/filename_icon_v3.ai:AFP_AfpInfo] failed
[2020/10/19 14:49:08.779410,  0] ../../source3/modules/vfs_fruit.c:2200(fruit_pread_meta_stream)
 fruit_pread_meta_stream: Removing [Endclient/2020/20260_projectname/Icons/filename_icon_v3.svg:AFP_AfpInfo] after short read [0]
[2020/10/19 14:49:08.779772,  0] ../../source3/modules/vfs_fruit.c:2208(fruit_pread_meta_stream)
 fruit_pread_meta_stream: Removing [Endclient/2020/20260_projectname/Icons/filename_icon_v3.svg:AFP_AfpInfo] failed
[2020/10/19 14:49:08.859609,  0] ../../source3/modules/vfs_fruit.c:2200(fruit_pread_meta_stream)
 fruit_pread_meta_stream: Removing [Endclient/2020/20260_projectname/Rebalance_icon.pptx:AFP_AfpInfo] after short read [0]
[2020/10/19 14:49:08.859959,  0] ../../source3/modules/vfs_fruit.c:2208(fruit_pread_meta_stream)
 fruit_pread_meta_stream: Removing [Endclient/2020/20260_projectname/Rebalance_icon.pptx:AFP_AfpInfo] failed
[2020/10/19 14:49:09.655440,  0] ../../source3/modules/vfs_fruit.c:2200(fruit_pread_meta_stream)
 fruit_pread_meta_stream: Removing [Endclient/2020/20260_projectname/Rebalance_icon.pptx:AFP_AfpInfo] after short read [0]
[2020/10/19 14:49:09.656351,  0] ../../source3/modules/vfs_fruit.c:2208(fruit_pread_meta_stream)
 fruit_pread_meta_stream: Removing [Endclient/2020/20260_projectname/Rebalance_icon.pptx:AFP_AfpInfo] failed

We discovered that the AFP_AfpInfo xattr on files copied to one samba gateway is non-existent, but when looking at the xattr from another hypervisor og samba gateway, the Apfinfo is still visible and is causing the files to be grayed out and unusable to macOS. The issue with files being deleted (apparently by Finder) after copying is a mystery.

At the moment we are a bit lost and struggle to find exactly what’s causing vfs_fruit to being unable to remove the FinderInfo extended attributes but other xattr operations are read/written to/from cephfs apparently without problems. Is there any known issues using xattr streams with cephfs? Is it because we use samba in containers?

Any suggestions or pointers how to solve this issue would be welcome!


--

smb.conf is as follows:

[global]
 workgroup = STORAGE
 netbios name = STORAGE
 realm = AD.STORAGE.NET
 security = ADS
 idmap config * : backend = autorid
 idmap config * : range = 10000-24999999
 map acl inherit = Yes

 bind interfaces only = yes
 interfaces = lo eth1
 passdb backend = tdbsam
 username map = /etc/samba/user.map

 smbd: backgroundqueue = no
 clustering = yes
 ctdbd socket = /run/ctdb/ctdbd.socket

 winbind use default domain = yes
 winbind enum users = yes
 winbind enum groups = yes

 vfs objects = catia fruit streams_xattr acl_xattr recycle
 min protocol = SMB2
 ea support = yes

 fruit:metadata = stream
 fruit:resource = stream

 fruit:model = Xserve
 fruit:posix_rename = yes
 fruit:zero_file_id = yes
 fruit:veto_appledouble = no
 fruit:wipe_intentionally_left_blank_rfork = yes
 fruit:delete_empty_adfiles = yes

 #disable printing
 load printers = no
 printing = bsd
 printcap name = /dev/null
 disable spoolss = yes

 #performance tuning
 socket options = TCP_NODELAY
 aio read size = 1
 aio write size = 1
 min receivefile size = 16384
 use sendfile = true
 max xmit = 65535
 log level = 1
 read raw = yes
 write raw = yes
 dead time = 15
 getwd cache = yes

 #enable spotlight search for macOS
 spotlight backend = elasticsearch
 elasticsearch:address = 127.0.0.1
 elasticsearch:port = 9200


[storage]
 path = /srv/storage
 read only = no
 posix locking = no
 strict locking = no
 oplocks = no
 level2 oplocks = no
 kernel share modes = no
 recycle:repository = .recycle
 recycle:keeptree = yes
 recycle:versions = yes
 spotlight = yes
 elasticsearch:index = storage

Container config:

arch: amd64
cores: 8
cpulimit: 0
cpuunits: 1024
hostname: samba-01
memory: 32768
mp0: /mnt/pve/cephfs/.ctdb/storage,mp=/var/lib/ctdb/shared,shared=1
mp1: /mnt/pve/cephfs/volumes/storage/d4a36e59-6f24-4f14-a193-44b7bb1707b4,mp=/srv/storage,shared=1
net0: name=eth0,bridge=vmbr0,gw=10.0.19.1,hwaddr=E2:0C:B7:D7:50:BB,ip=10.0.19.53/24,ip6=auto,tag=19,type=veth
net1: name=eth1,bridge=vmbr0,hwaddr=02:49:c8:17:f5:e1,ip=10.0.20.53/24,ip6=auto,tag=20,type=veth,mtu=9000
onboot: 1
ostype: ubuntu
rootfs: fast-replicated_ct:vm-28933-disk-0,size=16G
swap: 512

 

[Alwin]

We run samba in privileged containers with CTDB utilizing CephFS storage. Samba is version 4.12.8-SerNet-Ubuntu-8.focal running on ubuntu 20.04 in Proxmox LXCs and Ceph is version 14.2.11 also running on Proxmox 6.2. The CephFS volumes is bind-mounted into the container and shared with Samba.

How is the CephFS mounted on the node (kernel, fuse)? And are there messages in the dmesg?

vfs objects = catia fruit streams_xattr acl_xattr recycle

There is alos a vfs_ceph that utilizes libceph to connect to Ceph directly. That might be worth to look at.

 

[Thomas Hukkelberg]

Hi,

CephFS is mounted via kernel on hypervisor with proxmox GUI and the 'mount' command returns following for the mouted cephfs:

10.40.28.151,10.40.28.151:6789,10.40.28.152,10.40.28.152:6789,10.40.28.153,10.40.28.153:6789,10.40.28.157,10.40.28.158:/ on /mnt/pve/cephfs type ceph (rw,relatime,name=admin,secret=<hidden>,acl)

dmesg gives no error messages regarding this.

We have tried using the vfs_ceph module directly with samba but experience really bad read performance compared to kernel mounted mode. End clients for the mount point is Mac video production workstations with 10GbE interfaces that mainly work with mulit-gigabyte files and good read thruput is important. Kernel mounted export gives ~800MB/s write and 650MB/s read (measured using the single-threaded BlackMagic Speed Test). With vfs_ceph we get ~600MB/s write and 150MB/s read which is a bit too low. Changing the ceph.dir.layout to 8 stripes/512kb units gives a bit better read performance when using vfs_ceph but not nearly as good as kernel mounted mode. What other options could be done to optimize vfs_ceph? Please note that write operations are higher due to NVME journals.

We have investigated the original issue a bit more and found a decent workaround for now. We believe the smb.conf options min receivefile size = 16384 and max xmit = 65535 make Finder delete random files after copying, so removing them fixes that problem. When it comes to the problem of files being marked grayed out, we set readdir_attr:aapl_finder_info = no to make samba ignore the flag so it is not relayed to the end client. This has other side effects as other macOS Finder flags are not being updated and the underlying problem with random xattrs not being updated/removed is still present.

Any input on how to troubleshoot this would be great as the problem seem not to be samba but rather between container<->hypervizor<->ceph/mds.

 

[Alwin]

Please note that write operations are higher due to NVME journals.

Reads of rados objects are usually around twice as fast as writes, since the can be done in parallel. Hm... is that the case if you run a rados bench?

Any input on how to troubleshoot this would be great as the problem seem not to be samba but rather between container<->hypervizor<->ceph/mds.

Only in general, that you could try to use fuse to mount cephfs.
https://pve.proxmox.com/pve-docs/chapter-pct.html#_fuse_mounts