CIFS storage is not online

[ccube]

I added a CIFS storage for my backups (Hetzner Storage Box). After a reboot everything is working and seems fine. However, after a couple of days (without touching anything), I am getting the Email that the backup failed, because the storage is not online.

If I check the Proxmox Web Interface, I (sometimes) see a questionmark next to the storage.
If I check the mount in a terminal session, it is mounted and writable.

Any advice where to look whats going on?

Thanks

 

[apoc]

Have you checked the logs? Any messages there which could help?

 

[ccube]

Which logs are you referring to. Journalctl seems quite silent about the issue.

 

[apoc]

Anything in /var/log that could be of interest.

 

[ccube]

le@sb37:/var/log$ sudo ack cifs
messages.1
770:Mar 12 20:13:22 sb37 kernel: [ 1089.518353] FS-Cache: Netfs 'cifs' registered for caching
771:Mar 12 20:13:22 sb37 kernel: [ 1089.531192] Key type cifs.spnego registered
772:Mar 12 20:13:22 sb37 kernel: [ 1089.531225] Key type cifs.idmap registered

auth.log
37154:Mar 15 18:26:00 sb37 sudo:       le : TTY=pts/0 ; PWD=/var/log ; USER=root ; COMMAND=/usr/bin/ack cifs

 

[Eitan]

Just experienced this issue now

syslog isn't gving me anything useful

i just tried removing the storage and adding it again

this is the error i recieved:

create storage failed: error with cfs lock 'file-storage_cfg': storage 'backups' is not online (500)

at least there is a file that we can look at now

also, i have 2 other nodes attached to this storage and the behavior is exactly the same

 

[ccube]

is there a proxmox bugtracker? looks like we should move this thread to over there.

 

[Eitan]

and then suddenly after 15 min it works...wierd...

soon i'll be on ceph anyways...

 

[Eitan]

could these be of interest?

https://forum.proxmox.com/threads/c...fg-working-now-but-shows-question-mark.45962/
https://forum.proxmox.com/threads/issue-with-incorrect-cfs-lock.54345/
https://forum.proxmox.com/threads/add-cifs-storage-error.44349/

 

[ccube]

Doesn't really seem to be related. No idea to be honest. I observed that it seems to recover by itself. Without my intervention I have days, where the backup fails and days where it passes. The mount is always there, however.

 

[Eitan]

From performing backup/restore on about 20 VMs/containers it seems that when the storage target is overwhelmed it stops responding to proxmox, even though it still responds to other network clients. Damn Ceph is complicated...

 

[Eitan]

Wait a second, I just noticed that Hetzner Storage Box is an internet backup service. I would say that the issue the OP is experiencing is related the WAN latency, unless it can be ruled out. or ISP traffic filtering.

 

[ccube]

Both server and Storage are in a Hetzner RZ. The latency is very low. I am aware of Samba/WAN problems. However, since the mount hast generally a very good performance, I do not see any reason why proxmox rejects to backup to it even though it is there....

 

[daemonk]

Hi, I am also with hetzner using a storagebox and getting the exact same issue. Did you ever resolve this? thanks

 

[ccube]

Not really solved. But seems more stable in the last couple of days. No idea what's going on.

 

[daemonk]

Seems the storage boxes on Hetzner are quite jumpy...I have resorted to a manual mount in the shell for now.

Thanks for the reply.

 

[Dr.ooz]

Same issue here when it almost full it goes offline. I have to delete from proxmox and add it again. Any solution?

 

[Eitan]

I would look at logs for clues, but your issue sounds like it is related to being full, rather than just random like the other posts in this thread

 

[Ɍey]

I dont want to come off as a necromancer, reviving the dead in my first post...

https://imgur.com/a/4gNiVms

But I couldn't get around it, as I just set up an instance the same hoster (hetzner) looking at the same behavior. While looking for an answer I also found this threat, so "Hi Guys"

The configuration of the CIFS share offered in the product "storagebox" differs from "most" CIFS shares. Usually those shares are on the local network (private addressing), they adverise their shares and are promiscuous in nature.
Hetzner on the other hand seems to have hardened their shares, almost certainly because they allow users to to grant access to the storage for 0.0.0.0/0 in one click.

Besides shares not being advertised, this unfortunately also seems to apply to the smb version. So I am hopeful these steps will help in such cases:

- Add the storage using the WebUI, but make sure to uncheck the radio button called "Enable"
- Open "/etc/pve/storage.cfg" in your favorite text editor and locate the storage you just added
- As last line of the storage config block add the option "smbversion 3" like the following example

cifs: hetzner-storagebox
        path /mnt/pve/hetzner-storagebox
        server Ɍey.ProviderDomain.tld
        share /backup
        content backup
        prune-backups keep-all=1
        username Ɍey
        smbversion 3

- Go back to the WebUI and enable the storage

EDIT: Coming back to it today I see the exact same behavior as described by the others. I will keep you posted.

Sidenote: After enabling the share the option "smbversion 3" was removed from the config, but I could not reproduce the problem again. Except by fully deleting the configuration for the share. So my guess would be that once the share has been added successfully, the SMB type is remembered somewhere else (maybe in conjunction /w the password).

Maybe it would be helpful to add an SMB version dropdown menu to the GUI, autodetection is great in the majority of cases but I am sure in case of doubt all of us users would try and toggle that version option intuitively. I would have done it certainly.

UPDATE: well, seems it actually has no solution, except a manual mount via CLI


Cheers,
Ɍey