Problem: Shutdown of NFS Server makes Cluster master "hang"

[holgerb]

Hi all,

we have recently bought a 19" based storage solution which we want to use as NFS storage for our Proxmox cluster.
After adding the published NFS shares to our storage pool in the Proxmox web frontend I shut down the storage
server. As a result of this I was not able to log in at the cluster master frontend. I got an error message that
user and/or password is incorrect.

After powering up the storage server I could login again.

Questions:
1) Is it possible that downtime of a attached NFS (or other NAS) causes the login mechanism of the master to fail ?
2) If yes, will this be fixed in a newer version of Proxmox ? To my understanding a solution like Proxmox must be fault-tolerant enough to deal with such events (unless there are really running machines located at the storage).
3) Is there some workaround for this problem ?

Additional info:
We are currently running proxmox V1.5 in our cluster.
The storage server used is a Thecus 8800 Pro (although this shouldn't make a difference).

Regards,
Holger

 

[tom]

storage needs to online, otherwise you have the problems you described. and yes, future versions will have improvements here.

 

[holgerb]

Thanks for your reply ! The major problem I see for now:
In order to access the cluster master web frontend again you either have to power up your storage server or remove the storage server from your storage pool.
Since removing the storage is only possible from the cluster master we run into some sort of chicken-egg-problem

Oki, we´ll have to figure out something ourselves as workaround.

best regards,
Holger

 

[tom]

if you mount storage these storage needs to be online.

just to explain in an example: if you unplug the SATA cable from your desktop hard drive your system will not like it also.

you can remove the storage definition also in /etc/pve/storage.cfg

 

[holgerb]

Hey tom,

just to explain in an example: if you unplug the SATA cable from your desktop hard drive your system will not like it also.

Acknowledged...my car will also not like if my wheels are removed at full speed
I see your point but I was more trying to pinpoint that the web frontend of the master should still be working even when an attached storage device fails unless it´s the host HDD itself where proxmox is installed on.

you can remove the storage definition also in /etc/pve/storage.cfg

That sounds good. So we could manually remove the storage definition if we ssh into the cluster master.

Cheers,
Holger

 

[hk@]

Hi
might i suggest to use the nfs-mount-options hard,intr - current nfs-mounts made by proxmox-gui onyl go for "rw" thus deadlocking any process accessing these mounts if they go away.

regards
hk

 

[dietmar]

Hi
might i suggest to use the nfs-mount-options hard,intr - current nfs-mounts made by proxmox-gui onyl go for "rw" thus deadlocking any process accessing these mounts if they go away.

I will look into that next week - thanks.

 

[hk@]

Hi
just noticed - by accident - rebooting a server that can't connect to it's eg. nfs-backup-mount ends in a looooong hang of this box - looks like this during shutdown (and seems to not end anytime soon):

Stopping OpenVZ: nfs: server 192.168.221.250 not responding, still trying

this is really ugly - a reboot shouldn't go this way.

regards
hk

 

[dietmar]

Stopping OpenVZ: nfs: server 192.168.221.250 not responding, still trying

this is really ugly - a reboot shouldn't go this way.

What do you suggest? If a server does not asnwers, nfs needs to retry - or is there a better solution?

 

[Flo]

I will look into that next week - thanks.

Hi
might i suggest to use the nfs-mount-options hard,intr - current nfs-mounts made by proxmox-gui onyl go for "rw" thus deadlocking any process accessing these mounts if they go away.

regards
hk

i would also like to see such an nfs-mount-options parameter in the proxmox-gui... are there any plans to implement that ? or is there a way to change the default settings ?

regards - Florian

 

[dietmar]

i would also like to see such an nfs-mount-options parameter in the proxmox-gui... are there any plans to implement that ? or is there a way to change the default settings ?

You can set that manually in the 'storage.cfg' file (field is called 'options')

 

[Flo]

exactly what I was looking for!

thanks!