storage is not online (cifs)

[Virsacer]

Hi,
I have a PVE 6.4 that I did not want to upgrade directly so I installed a new 7.0 and joined them in a cluster.
On the old one my cifs storage was working perfectly fine but on the new one it does not connect.

My syslog is full with messages like pvedaemon[126626]: storage 'STORAGE' is not online and "pvesm status" also only outputs this:

storage 'STORAGE' is not online
Name                  Type     Status           Total            Used       Available        %
STORAGE               cifs   inactive               0               0               0    0.00%
...

But I can mount the share with mount -t cifs -o credentials=/etc/pve/priv/STORAGE.cred -o username=xxx //hostname/backup /media

Is there a way to find out whats going on/wrong?

BTW: Reboot(s) did not help...

 

[Pifouney]

Hey, did you try to remove/re-enter your cifs share in the datacenter view?

Most smootly, try to see if it's declared in your /etc/pve/storage.cfg file.

 

[Virsacer]

I have disabled/enabled it and also removed/added it from/to the list of nodes multiple times, but also with no success...

...strange thing is that there is no (real) error message anywhere?

Finally I did remove it, mounted via fstab on both nodes and added a new storage of type "Directory"

 

[romansoft]

Same buggy behaviour here. Proxmox & Debian up to date.

The problem is clear: "smbclient" has some kind of incompatibility/problem with Samba libs:

root@proxmox:~# pvesm scan cifs 10.31.3.37 --username leet --password
Enter Password: *****
/usr/bin/smbclient: /usr/lib/x86_64-linux-gnu/samba/libldbsamba.so.0: unsupported version 0 of Verdef record
/usr/bin/smbclient: error while loading shared libraries: /usr/lib/x86_64-linux-gnu/samba/libldbsamba.so.0: unsupported version 0 of Verneed record
root@proxmox:~# smbclient
smbclient: /usr/lib/x86_64-linux-gnu/samba/libldbsamba.so.0: unsupported version 0 of Verdef record
smbclient: error while loading shared libraries: /usr/lib/x86_64-linux-gnu/samba/libldbsamba.so.0: unsupported version 0 of Verneed record
root@proxmox:~#

As I previously said, my system is completely up-to-date:
- Proxmox 7.0-13
- .deb:
Package: smbclient
Version: 2:4.13.5+dfsg-2
Package: samba-libs
Version: 2:4.13.5+dfsg-2

I'd say these ones are the guilty ;-)

I'm going to file a bug in Proxmox's bugzilla, so even this could be unrelated to Proxmox, at least the problem can be tracked.

Cheers,
-r

 

[romansoft]

I researched a little bit and figured out the solution

I compared smbclient and samba lib from a working Debian 11 and discovered than the lib from my Proxmox machine was different!!!

It seems the following lib was corrupted:
/usr/lib/x86_64-linux-gnu/samba/libldbsamba.so.0

So the solution is easy: simply SSH into Proxmox and reinstall the flawed samba lib package:

# apt install --reinstall samba-libs:amd64

Et voilà. Problem solved.


PS: Finally, I didn't file any bug report. But it's odd than we both had the same corruption problem. Maybe Debian 11 (some of its mirrors?) had the faulty package when we upgraded from Buster to Bullseye?

 

[fiona]

Hi,

But I can mount the share with mount -t cifs -o credentials=/etc/pve/priv/STORAGE.cred -o username=xxx //hostname/backup /media

before Proxmox VE 7, it was possible to have CIFS credentials in either /etc/pve/priv/<storeid>.cred (old fallback for backwards compatibility) and /etc/pve/priv/storage/<storeid>.pw (newer path that's used for other storage types too). Wit version 7, the fallback was dropped. There is a hook to automatically move the file when installing the new storage library package, but since you didn't upgrade, that was never executed. So I'm guessing that the fresh node in the cluster just didn't find the credentials where it expected them.

 

[Virsacer]

Ok, that would definitely explain it

But why are there no error messages giving any details?

 

[fiona]

If the share can't be accessed, because of missing privileges, it should actually give an tree connect failed: NT_STATUS_ACCESS_DENIED message. So maybe the connection check for the storage did fail for a different reason. Or the server doesn't return the correct error when login without credentials is attempted. You could check with

time smbclient //<host>/<share> -d 0 -U Guest -N -c 'echo 1 0'

 

[Virsacer]

When manually calling, I do get a session setup failed: NT_STATUS_LOGON_FAILURE

But zgrep NT_STATUS_LOGON_FAILURE /var/log/syslog* shows no entries...

 

[fiona]

When manually calling, I do get a session setup failed: NT_STATUS_LOGON_FAILURE

But zgrep NT_STATUS_LOGON_FAILURE /var/log/syslog* shows no entries...

Thanks! We currently only bubble up the NT_STATUS_ACCESS_DENIED error. Every other error is interpreted as "storage not online". IMHO it makes sense to also bubble up NT_STATUS_LOGON_FAILURE. I'll add that to my to-do list, but it's not a high-priority issue, so it might take a bit until I get around to it.

 

[Jannoke]

I have the same problem except I'm trying to add storage and get the access denied error. I can connect from command line with smbclient just fine. Running proxmox 8.0.4.

 

[fiona]

Hi,

I have the same problem except I'm trying to add storage and get the access denied error. I can connect from command line with smbclient just fine. Running proxmox 8.0.4.

what is the exact error you get? Anything in the system logs/journal? How do you connect from the command line?