storage is not online....

[manilx]

Hi

I have proxmox connected to a QNAP TVS-h1288X via NFS (or SMB) for a storage.
Use it to backup and to hold iso's.

Lately I have a frequant issue when I access the storage in proxmox with it telling me that the "storage is not online", when it is all the time. When I quit the message and try again it's instantly available.
This has begun with an update to proxmox the last few weeks (can't tell exactly when this happened).

Is there a way to tell what the problem is?

Thx in advance

 

[bbgeek17]

Have you examined /var/log/messages ? journactl ?
Is there anything in them around the time of failure? "Online" checks, in some cases, are simple ICMP port reachouts. If those are dropped storage will be declared offline. You may need to get a network trace on both sides to see what happens during the "outage".

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[manilx]

Have you examined /var/log/messages ? journactl ?
Is there anything in them around the time of failure? "Online" checks, in some cases, are simple ICMP port reachouts. If those are dropped storage will be declared offline. You may need to get a network trace on both sides to see what happens during the "outage".

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

I'm not a lot into CLI.....
But /var/log/messages has nothing in it concerning this.
journalctl I find this:
Jun 28 13:47:42 protecli pvedaemon[4001657]: storage 'qnap' is not online
Jun 28 13:47:43 protecli pvestatd[1242]: storage 'qnap' is not online
Jun 28 13:47:43 protecli pvestatd[1242]: status update time (5.336 seconds)
Jun 28 13:48:41 protecli pvedaemon[4029611]: got timeout
Jun 28 13:48:53 protecli pvedaemon[4000281]: got timeout
Jun 28 13:48:53 protecli pvedaemon[4000281]: unable to activate storage 'qnap' - directory '/mnt/pve/qnap' does not exist or is unreachable
Jun 28 13:48:53 protecli pvestatd[1242]: got timeout
Jun 28 13:48:53 protecli pvestatd[1242]: unable to activate storage 'qnap' - directory '/mnt/pve/qnap' does not exist or is unreachable
Jun 28 13:48:53 protecli pvestatd[1242]: status update time (5.422 seconds)
Jun 28 13:49:48 protecli pvedaemon[4000281]: storage 'qnap' is not online
Jun 28 13:49:53 protecli pvestatd[1242]: storage 'qnap' is not online
Jun 28 13:49:53 protecli pvestatd[1242]: status update time (5.358 seconds)

 

[bbgeek17]

You said you tried both NFS and CIFS and they both fail. These are two protocols that use a completely different healthcheck mechanisms inside PVE.
The common denominator is your NAS and Network. Its unlikely that your NAS cant handle a simple "are you there" request/response.
Its much more likely that your network is not optimal. There could be many reasons:
bad cabling
bad switch
bad port
bad nic
duplicate IP
bad routing
bad firewall
...

One option is to start changing above methodically, test after every change. My advice is to start reading network troubleshooting blogs on the Net and start enjoying CLI.

A quick way to isolate many things out is to direct connect Proxmox and NAS.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[manilx]

Well, will try some things....
Only have issue with proxmox for a short time. Same install, same hardware, same config had no issues a short while ago....

 

[bbgeek17]

Its also possible you are affected by this or similar:
https://bugzilla.proxmox.com/show_bug.cgi?id=4033

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[manilx]

Now this looks exactly similar to what I have encountered. Starting with an update as I mentioned, kernel as it seems.
I have attached the QNAP now via NFS 4.....
Let's see how this goes.

THX!

 

[manilx]

From what I can see now after a few hours I have yet to have the problem again. Seems like the change to use NFS 4 was the fix.