SMB/CIFS unable to activate storage after share restarted

[sloppyjoe]

Hi!

I've been running Proxmox about a year and absolutely love it. However, I've been having a problem since I first installed that I can't solve (I've searched here and Google too).

Proxmox (fully updated 8.1.10) is connected via SMB/CIFS to a Windows box that has most of my additional storage. When I restart the Windows box Proxmox immediately notices that the share is disconnected and keeps trying to reconnect (which is great), however even when the Windows machine comes back online, Proxmox doesn't seem to be able to reconnect.

Here are the logs:

Apr 13 08:49:01 pve pvestatd[1001]: storage 'hercules' is not online
Apr 13 08:49:16 pve pvestatd[1001]: storage 'hercules' is not online
Apr 13 08:49:16 pve pvestatd[1001]: status update time (5.230 seconds)
Apr 13 08:49:27 pve pvestatd[1001]: storage 'hercules' is not online
Apr 13 08:49:27 pve pvestatd[1001]: status update time (5.223 seconds)
Apr 13 08:49:36 pve pvestatd[1001]: storage 'hercules' is not online
Apr 13 08:49:36 pve pvestatd[1001]: status update time (5.182 seconds)
Apr 13 08:49:44 pve pvestatd[1001]: storage 'hercules' is not online
Apr 13 08:49:54 pve pvestatd[1001]: storage 'hercules' is not online
Apr 13 08:50:02 pve pvestatd[1001]: storage 'hercules' is not online
Apr 13 08:50:14 pve pvestatd[1001]: storage 'hercules' is not online
Apr 13 08:50:24 pve pvestatd[1001]: got timeout
Apr 13 08:50:24 pve pvestatd[1001]: unable to activate storage 'hercules' - directory '/mnt/pve/hercules' does not exist or is unreachable
Apr 13 08:50:33 pve pvestatd[1001]: got timeout
Apr 13 08:50:33 pve pvestatd[1001]: unable to activate storage 'hercules' - directory '/mnt/pve/hercules' does not exist or is unreachable
Apr 13 08:50:43 pve pvestatd[1001]: got timeout
Apr 13 08:50:43 pve pvestatd[1001]: unable to activate storage 'hercules' - directory '/mnt/pve/hercules' does not exist or is unreachable
Apr 13 08:50:53 pve pvestatd[1001]: got timeout
Apr 13 08:50:53 pve pvestatd[1001]: unable to activate storage 'hercules' - directory '/mnt/pve/hercules' does not exist or is unreachable
Apr 13 08:50:54 pve pvedaemon[518112]: got timeout
Apr 13 08:50:54 pve pvedaemon[518112]: unable to activate storage 'hercules' - directory '/mnt/pve/hercules' does not exist or is unreachable
Apr 13 08:51:03 pve pvestatd[1001]: got timeout
Apr 13 08:51:03 pve pvestatd[1001]: unable to activate storage 'hercules' - directory '/mnt/pve/hercules' does not exist or is unreachable
Apr 13 08:51:13 pve pvestatd[1001]: got timeout
Apr 13 08:51:13 pve pvestatd[1001]: unable to activate storage 'hercules' - directory '/mnt/pve/hercules' does not exist or is unreachable
Apr 13 08:51:23 pve pvestatd[1001]: got timeout
Apr 13 08:51:23 pve pvestatd[1001]: unable to activate storage 'hercules' - directory '/mnt/pve/hercules' does not exist or is unreachable
Apr 13 08:51:33 pve pvestatd[1001]: got timeout
Apr 13 08:51:33 pve pvestatd[1001]: unable to activate storage 'hercules' - directory '/mnt/pve/hercules' does not exist or is unreachable
Apr 13 08:51:43 pve pvestatd[1001]: got timeout
Apr 13 08:51:43 pve pvestatd[1001]: unable to activate storage 'hercules' - directory '/mnt/pve/hercules' does not exist or is unreachable
Apr 13 08:51:53 pve pvestatd[1001]: got timeout
Apr 13 08:51:53 pve pvestatd[1001]: unable to activate storage 'hercules' - directory '/mnt/pve/hercules' does not exist or is unreachable
Apr 13 08:51:57 pve kernel: CIFS: VFS: \\10.10.7.121 has not responded in 180 seconds. Reconnecting...
Apr 13 08:52:01 pve pvestatd[1001]: unable to activate storage 'hercules' - directory '/mnt/pve/hercules' does not exist or is unreachable
Apr 13 08:52:11 pve pvestatd[1001]: unable to activate storage 'hercules' - directory '/mnt/pve/hercules' does not exist or is unreachable
Apr 13 08:52:21 pve pvestatd[1001]: unable to activate storage 'hercules' - directory '/mnt/pve/hercules' does not exist or is unreachable
Apr 13 08:52:31 pve pvestatd[1001]: unable to activate storage 'hercules' - directory '/mnt/pve/hercules' does not exist or is unreachable
Apr 13 08:52:42 pve pvestatd[1001]: unable to activate storage 'hercules' - directory '/mnt/pve/hercules' does not exist or is unreachable

At around 08:51:33 the windows machine is back up but Proxmox doesn't seem to be able to reconnect to it.

The only way I've found to resolve this issue is to restart Proxmox itself.

Is there a way to get Proxmox to automatically reconnect to the share?

Thanks in advanced for any help!

Joe

 

[aweber1nj]

@sloppyjoe Did you ever figure out any command/option to get PVE to reconnect? I'm seeing this with a TrueNAS server that I had to restart post upgrade. (The share is hosted by TrueNAS, so it's not specific to Windows servers.)

 

[aweber1nj]

I made sure my pve is up-to-date. This is what I get:

root@pve:~# ls -l /mnt/pve
ls: cannot access '/mnt/pve/FreenasBackup2': Host is down
ls: cannot access '/mnt/pve/FreenasBackup1': Host is down
total 0
d????????? ? ? ? ?            ? FreenasBackup1
d????????? ? ? ? ?            ? FreenasBackup2
root@pve:~# pvesm scan cifs freenas
backup   
backup1  First
backup2  Second
pbackup2 Secured
root@pve:~#

 

[Gumball6141]

I made sure my pve is up-to-date. This is what I get:

root@pve:~# ls -l /mnt/pve
ls: cannot access '/mnt/pve/FreenasBackup2': Host is down
ls: cannot access '/mnt/pve/FreenasBackup1': Host is down
total 0
d????????? ? ? ? ?            ? FreenasBackup1
d????????? ? ? ? ?            ? FreenasBackup2
root@pve:~# pvesm scan cifs freenas
backup
backup1  First
backup2  Second
pbackup2 Secured
root@pve:~#

I have the exact same symptoms with an OMV CIFS folder mount to Proxmox (at /mnt/pve/). If for any reason OMV is restarted, the mount is not automatically reconnected to Proxmox. Dmesg reports that CIFS mount in unavailable and Proxmox that CIFS storage is not activated. I just restarted Proxmox and the mount completes fine. This is a big problem, since a power failure or/and a maintenance poweroff of the mounted folder results in a relatively silent fail of the mount. I did not even get a notification from Proxmox but noticed it only after a backup to the mounted folder failed. Has anyone managed to find a solution to this? I see that the problem goes back to at least 2024 and relates to mounts of any kind (Windows, TrueNAS and OMV in my case).

 

[Gumball6141]

UPDATE for future readers: I managed to solve this by mounting the share with the option "noserverino" in /etc/pve/storage.cfg (and of course after unmounting and remounting the share in order for the attribute to take effect). Now the share becomes active again (reconnects) after the remote host restarts, albeit approx. 1-1.30 min after the latter's restart is concluded (I haven't managed to solve this). Initially the mount option "cache=loose" (or cache=none) worked too, but I didn't want to lose this feature. In my case the remote host is OMV. I don't know how this problem has not been more widely reported.