samba share incosistencies

[hereitis]

Hi all,
I am running 4 node cluster, v8.2.4. Recently I added nfs and smb shares (with shared option inside proxmox storage) from truenas to store "shared" data among cluster nodes.

I am using proxmox-shared samba share to store lxc and vm images (maybe doing it wrong, but at least migration is almost instant).

Lately, I started having issues with some of lxc/vm images and its owners...

For example, first_node is stuck listing image 131, second_node is stuck at 137, while other two nodes can list all images from that proxmox-shared share.

Is there some process or service to restart and fix this lockups? I searched for samba-client but didnt find anything. Since it works normally on two out of four nodes, I think samba server is out of the picture?

Forgot to mention I tried to disable and enable share using pvesm tool, but it didnt fix the issue.

Share is mounted as:

//192.168.0.156/proxmox-shared on /mnt/pve/proxmox-shared type cifs (rw,relatime,vers=3.1.1,cache=strict,username=proxmox,uid=0,noforceuid,gid=0,noforcegid,addr=192.168.0.156,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,retrans=1,echo_interval=60,actimeo=1,closetimeo=1)

Only clue I could find on second node related to samba is following:

2024-07-16T00:50:09.212419+02:00 pve2 kernel: [1201407.741302] CIFS: VFS: Autodisabling the use of server inode numbers on \\192.168.0.156\proxmox-backups
2024-07-16T00:50:09.212428+02:00 pve2 kernel: [1201407.741306] CIFS: VFS: The server doesn't seem to support them properly or the files might be on different servers (DFS)
2024-07-16T00:50:09.212429+02:00 pve2 kernel: [1201407.741308] CIFS: VFS: Hardlinks will not be recognized on this mount. Consider mounting with the "noserverino" option to silence this message.
2024-07-16T00:50:22.069394+02:00 pve2 kernel: [1201420.597848] CIFS: VFS: \\192.168.0.156\proxmox-backups Close unmatched open for MID:35242177

first_node# ls -alR /mnt/pve/proxmox-shared/images/
/mnt/pve/proxmox-shared/images/:
total 0
drwxr-xr-x 2 root root 0 Jul 15 13:50 .
drwxr-xr-x 2 root root 0 Jul  2 01:07 ..
drwxr-xr-x 2 root root 0 Jul  2 01:09 127
drwxr-xr-x 2 root root 0 Jul  2 01:11 131
drwxr-xr-x 2 root root 0 Jul  2 02:54 132
drwxr-xr-x 2 root root 0 Jul  5 00:17 135
drwxr-xr-x 2 root root 0 Jul  5 17:46 136
drwxr-xr-x 2 root root 0 Jul  9 13:11 137
drwxr-xr-x 2 root root 0 Jul 11 11:29 138
drwxr-xr-x 2 root root 0 Jul  9 10:39 8001

/mnt/pve/proxmox-shared/images/127:
total 93937
drwxr-xr-x 2 root root          0 Jul  2 01:09 .
drwxr-xr-x 2 root root          0 Jul 15 13:50 ..
-rwxr-xr-x 1 root root 8589934592 Jul 16 00:45 vm-127-disk-0.raw

/mnt/pve/proxmox-shared/images/131:
^C

second_node# ls -alR /mnt/pve/proxmox-shared/
/mnt/pve/proxmox-shared/:
total 4
drwxr-xr-x 2 root root    0 Jul  2 01:07 .
drwxr-xr-x 7 root root 4096 Jul  2 00:57 ..
drwxr-xr-x 2 root root    0 Jul  2 01:07 dump
drwxr-xr-x 2 root root    0 Jul 15 13:50 images
drwxr-xr-x 2 root root    0 Jul  2 01:07 private

/mnt/pve/proxmox-shared/dump:
total 0
drwxr-xr-x 2 root root 0 Jul  2 01:07 .
drwxr-xr-x 2 root root 0 Jul  2 01:07 ..

/mnt/pve/proxmox-shared/images:
total 0
drwxr-xr-x 2 root root 0 Jul 15 13:50 .
drwxr-xr-x 2 root root 0 Jul  2 01:07 ..
drwxr-xr-x 2 root root 0 Jul  2 01:09 127
drwxr-xr-x 2 root root 0 Jul  2 01:11 131
drwxr-xr-x 2 root root 0 Jul  2 02:54 132
drwxr-xr-x 2 root root 0 Jul  5 00:17 135
drwxr-xr-x 2 root root 0 Jul  5 17:46 136
drwxr-xr-x 2 root root 0 Jul  9 13:11 137
drwxr-xr-x 2 root root 0 Jul 11 11:29 138
drwxr-xr-x 2 root root 0 Jul  9 10:39 8001

/mnt/pve/proxmox-shared/images/127:
total 82325
drwxr-xr-x 2 root root          0 Jul  2 01:09 .
drwxr-xr-x 2 root root          0 Jul 15 13:50 ..
-rwxr-xr-x 1 root root 8589934592 Jul 16 01:00 vm-127-disk-0.raw

/mnt/pve/proxmox-shared/images/131:
total 13694349
drwxr-xr-x 2 root root            0 Jul  2 01:11 .
drwxr-xr-x 2 root root            0 Jul 15 13:50 ..
-rwxr-xr-x 1 root root 107374182400 Jul 14 22:26 vm-131-disk-0.raw

/mnt/pve/proxmox-shared/images/132:
total 54600665
drwxr-xr-x 2 root root            0 Jul  2 02:54 .
drwxr-xr-x 2 root root            0 Jul 15 13:50 ..
-rwxr-xr-x 1 root root 107374182400 Jul 16 01:00 vm-132-disk-0.raw

/mnt/pve/proxmox-shared/images/135:
total 0
drwxr-xr-x 2 root root 0 Jul  5 00:17 .
drwxr-xr-x 2 root root 0 Jul 15 13:50 ..

/mnt/pve/proxmox-shared/images/136:
total 1502301
drwxr-xr-x 2 root root           0 Jul  5 17:46 .
drwxr-xr-x 2 root root           0 Jul 15 13:50 ..
-rwxr-xr-x 1 root root 21474836480 Jul  5 17:46 vm-136-disk-0.raw

/mnt/pve/proxmox-shared/images/137:
^C

# pvesm status
Name                  Type     Status           Total            Used       Available        %
backups-smb           cifs     active     13628550912      1869821440     11758729472   13.72%
iso-template           nfs     active     11772713984        13983744     11758730240    0.12%
local                  dir     active       951541116       138443092       773383088   14.55%
proxmox-shared        cifs     active     11856373248        97643776     11758729472    0.82%

# pveversion -v
proxmox-ve: 8.2.0 (running kernel: 6.8.8-2-pve)
pve-manager: 8.2.4 (running version: 8.2.4/faa83925c9641325)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.8-2
proxmox-kernel-6.8.8-2-pve-signed: 6.8.8-2
proxmox-kernel-6.8.4-3-pve-signed: 6.8.4-3
proxmox-kernel-6.5.13-5-pve-signed: 6.5.13-5
proxmox-kernel-6.5: 6.5.13-5
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
proxmox-kernel-6.2.16-19-pve: 6.2.16-19
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
intel-microcode: 3.20240514.1~deb12u1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.3
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.9
libpve-storage-perl: 8.2.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.7-1
proxmox-backup-file-restore: 3.2.7-1
proxmox-firewall: 0.4.2
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.1.12
pve-docs: 8.2.2
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.1
pve-firewall: 5.0.7
pve-firmware: 3.12-1
pve-ha-manager: 4.0.5
pve-i18n: 3.2.2
pve-qemu-kvm: 9.0.0-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.4-pve1