NFS storage errors in logs (pvestatd: WARNING: storage is not online)

[Fleg]

Hi all,
I have a cluster with 6 nodes running kernel 2.6.32.17-pve and manager 2.2.32/389a616
I use 2 NFS storage, the main one (called vmdatastore_pamela2) for the VM images and a second one (called vmdatastore_gina) for the iso distributions files.
vmdatastore_pamela2 is hosted by a netapp machine and vmdatastore_gina is a hosted by a scientific linux box (so two different kind of servers both exporting NFS filesystem).
In the logs of all my nodes (/var/log/syslog) I have the following warnings :

Feb 5 14:01:14 lpnhevirt3 pvestatd[2257]: WARNING: storage 'vmdatastore_gina' is not online
Feb 5 14:02:24 lpnhevirt3 pvestatd[2257]: WARNING: storage 'vmdatastore_pamela2' is not online

No error or warning messages are present on the NFS servers.
The nfs servers also exports some storage to tens of working stations (mounting users home on these servers) without any problem (storage is always available).
Thus it doesn't seems related to the storage servers but to the proxmox clients.

Most of the time, there is no problem despite of the warnings, the storage is accessible.
Sometimes, on the web interface, I cannot access the storage (with error message "storage is not online"), but if I retry a few seconds later it works.
It should not be a big issue, but I had a few times some online migration failing and it seems due to the fact that storage on the target node was not online !

Does anyone had this problem ? Any clue to solve it ?
Thanks for your help.

F.

 

[hotwired007]

Could it be that the NIC connections on your NFS shares are saturated and Proxmox cannot connect? I would have dedicated storage for only Proxmox if i was running VMs from them.

 

[Fleg]

I don't think that the NIC connections are saturated.
On the nodes, I have warnings also on nodes without VM running (so the node is doing almost nothing).
On the NFS shares it seems also unlucky because I only have problems with the proxmox machines and absolutely no pb with other computers.
Moreover, on my 6 nodes I have not necessarily the warnings at the same time ! But this argument is week as the checks do not necessarily occurs exactly at the same time and it seems that the "unavalability" is short !

How pvestatd works ? Does it ping the storage servers regularly ? Is there a timeout thresholds ? Can it be modified ?

 

[Fleg]

The really strange thing is that it seems to happen almost always at the same time (let say between 2pm until 5am), including during the week-end (where the activity is very low).
It affect all the nodes and I have a warning in the logs almost every 2 minutes.

Friday 1 Fev 13h20 -> 2 Fev 3h40
Saturday 2 Fev 13h00 -> 3 Fev 4h25
Sunday 3 Fev 15h40 -> 4 Fev 2h55
Monday 4 Fev 14h55 -> 5 Fev 3h50
Tuesday 5 Fev 14h00 -> 6 Fev 4h50
Wednesday 6 Fev 16h39 -> 7 Fev 4h20

and today thursday 7 it started again at 16h20

We are monitoring our network, but nothing abnormal and no change in the charge of the switchs nor the interfaces of the nodes or the nfs servers!
It really looks like if it was related to the cluster (I was thinking to a cron job starting every days), but again I checked the running tasks and nothing appears to have started recently.

 

[elmacus]

You are not alone. We have seen this also. We have cluster and same error every hour, same time. Cron?

Sent from one of my Nexus Androids or Debian linux.

 

[spirit]

You are not alone. We have seen this also. We have cluster and same error every hour, same time. Cron?

Sent from one of my Nexus Androids or Debian linux.

Hi, the nfs check is:

/usr/bin/rpcinfo -p $server ($server is the server value in your nfs storage config in /etc/pve/storage.cfg)

with a timeout of 2sec.


So maybe the timeout is too short ? (maybe you have some short lags > 2s)

you can try to hack it in
/usr/share/perl5/PVE/Storage/NFSPlugin.pm

then /etc/init.d/pvestatd restart

this timeout is to avoid pvestatd hang if the storage is laggy.