CIFS/SMB Unable to activate storage - directory does not exist or is unrecheable

[Mihai]

PVE Versions:

proxmox-ve: 7.0-2 (running kernel: 5.11.22-5-pve)
pve-manager: 7.0-13 (running version: 7.0-13/7aa7e488)
pve-kernel-helper: 7.1-2
pve-kernel-5.11: 7.0-8
pve-kernel-5.4: 6.4-5
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.11.22-4-pve: 5.11.22-9
pve-kernel-5.4.128-1-pve: 5.4.128-1
ceph: 15.2.15-pve1
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve1
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-12
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-3
libpve-storage-perl: 7.0-13
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.13-1
proxmox-backup-file-restore: 2.0.13-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.1-1
pve-docs: 7.0-5
pve-edk2-firmware: 3.20210831-1
pve-firewall: 4.2-5
pve-firmware: 3.3-2
pve-ha-manager: 3.3-1
pve-i18n: 2.5-1
pve-qemu-kvm: 6.0.0-4
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-16
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.5-pve1

Task:

I was able to mount an SMB/Cifs share from TrueNAS using the GUI:



Problem:

When running a backup or uploading an iso file, from any host, I get the following error:

Dec 02 15:57:07 VMHost2 pvestatd[1638]: unable to activate storage 'proxmox-share' - directory '/mnt/pve/proxmox-share' does not exist or is unreachable
Dec 02 15:57:07 VMHost2 pvestatd[1638]: status update time (35.895 seconds)
Dec 02 15:57:10 VMHost2 pvestatd[1638]: got timeout

Dec 02 15:45:26 VMHost2 kernel: CIFS: VFS: \\backups.internal.foo.org sends on sock 000000008ba2532e stuck for 15 seconds
Dec 02 15:45:26 VMHost2 kernel: CIFS: VFS: \\backups.internal.foo.org Error -11 sending data on socket to server

Troubleshooting:

I can read and write to a simple text file from the CLI.



I tried to upload an iso: virtio-win-0.1.208.iso. It began uploading but then could not copy from the tmp directory. As you can see above, it remained as a 0 size file.

Ran the following commands:

pvesm scan cifs backups.internal.foo.org --username proxmox --password foo
backups       Backups
proxmox-share Share

nc backups.internal.foo.org 445 -v
backups.internal.foo.org [10.9.1.10] 445 (microsoft-ds) open

I did not see any issues with the firewall blocking any traffic.

When I try to access the file that was created by pve it also freezes for a bit but then I can download it.

Anyone have any idea what might be going on?

 

[Mihai]

I'm suspecting it might be a permissions issue. ProxmoxVE writes to these shares as the root user, correct?

PS: I tried NFS sharing and I had a similar issue.

 

[Mihai]

I just tried to mount a TrueNAS Core NFS share via NFS v3 and I am having the same issue!

When I attempt to upload an ISO for example, Proxmox freezes at the Copy data step.

The server is not responding, but it mounts fine!

I do not have this problem with Raspbian systems, or Ubuntu server systems or even veeam from Windows clients.

This only happens to my proxmox cluster and I cannot figure out what is going on.

Anyone have any troubleshooting steps?

I've tried looking at system logs and firewall logs and I cannot find out what is going on.

 

[Mihai]

So far I discovered the following:

The TrueNAS server has two interfaces:

one for management GUI only via bonded 1 Gbe
one for storage access via bonded 10 Gbe

If I try to mount and write via the 1Gbe management interface, it works find.
If I try to mount and write via the 10 Gbe storage interface it times out,

Same issue occurs if mounted via IP or DNS.

This problem occurs only for Proxmox hosts.

See my post over at the TrueNAS forums.

 

[kalasnikov]

So far I discovered the following:

The TrueNAS server has two interfaces:

one for management GUI only via bonded 1 Gbe
one for storage access via bonded 10 Gbe

If I try to mount and write via the 1Gbe management interface, it works find.
If I try to mount and write via the 10 Gbe storage interface it times out,

Same issue occurs if mounted via IP or DNS.

This problem occurs only for Proxmox hosts.

See my post over at the TrueNAS forums.

Could it be an MTU Setting? Did anyone get a chance to figure out what's going on?

 

[Mihai]

MTU is the same at 1500 (defaults).

Haven't even had time to go back and diagnose, partly also because I don't know what else to check.

 

[kalasnikov]

MTU is the same at 1500 (defaults).

Haven't even had time to go back and diagnose, partly also because I don't know what else to check.

So I have 2 Interfaces on my NAS. One is a 10G fiber and one is a LAGG with 4 1G. I switched to the LAGG port and it worked for me without an issue.... Been struggling with this for like ever now... Not sure if MTU has anything to do with the issues? I've tried all different configurations for NFS/SMB and nothing worked up until today when I switched.. Weird