GUI not responding when NFS share is down

[vcp_ai]

Hi, I've not found any similar post under PVE.2.x so here it goes....

One simple config with one 2.1. Server (last upgrade) and one external NFS server (storage server for backups, accesed as NFS share, it is a second disk of a second server, that actually has gone down).

When second server went down, GUI of PVE does not respond:
I can not log-in, or If I was logged, all info about machines can not be accesed.
Consoles are also not available.

Machines keep working without problems, and can be accesed via RDP.

SSH works correctly:
df never ends..
ls /mnt/pve never ends....

oot@cancerbero:/var/log# mount
/dev/mapper/pve-root on / type ext3 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
/dev/mapper/pve-data on /var/lib/vz type ext3 (rw)
/dev/sda1 on /boot type ext3 (rw)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,default_permissions,allow_other)
beancounter on /proc/vz/beancounter type cgroup (rw,name=beancounter)
container on /proc/vz/container type cgroup (rw,name=container)
fairsched on /proc/vz/fairsched type cgroup (rw,name=fairsched)
1xx.1xx.x.xx:/mnt/discoDATOS on /mnt/pve/discoDATOSsantxxxxxx type nfs (rw,vers=3,addr=1xx.1xx.x.xx)
root@cancerbero:/var/log#

Is this a known (correctable) problem ??

What should I do to get back my working server GUI ??? May be umount /mnt/pve/discoDATOSsantxxxxxx

Regards

 

[tom]

first, make sure your NFS never fails. if it fails, bring it online again.

on the proxmox ve side, you can also try to soft-mount the storage (you need to edit the storage.cfg), see example below.

nfs: backup
        path /mnt/pve/backup
        server xxxxxxx
        export /space/backup
        options vers=3,soft,rsize=32768,wsize=32768
        content backup
        maxfiles 10

 

[vcp_ai]

Thanks Tom for your answer.

first, make sure your NFS never fails. if it fails, bring it online again.

PVE and NFS Server are remote, and NFS server has definitevely failed.( How can I assure that NFS server NEVER fails ?)

As NFS is remote, I have no way to know what happens, and no way to bring it up.

What I need is to bring pve GUI up, so:

I've edited storage.cfg to remove nfs section, but it does not help (I can not log-in)

Do I need to restart GUI ? If so, what is the command to restart GUI ??

Regards

 

[tom]

if you cannot bring the NFS server online again, remove it from storage.cfg and reboot your Proxmox VE host.

(high available NFS servers are sold from many companies, also opensource solutions are available.)

 

[vcp_ai]

if you cannot bring the NFS server online again, remove it from storage.cfg and reboot your Proxmox VE host.

I can not reboot Proxmox host, if I can not stop the machines, and some of the do not respond correctly to shutdown command.
Will have to go to the installation and try to find what is the problem with the NFS server....

As I'm quite new to linux, is this problem related to linux (df & ls do not work) or is related to pve ?

Is it the same with other kind of storages (for backup) ? Any other storage solution suggestion ?

Will try to investigate HA NFS servers (any help/start point will be highly appreciated.....)

Thanks and Best Regards

 

[tom]

if you hard-mount a nfs you will have such problems. its on all Linux systems.

just think of a PC and you unplug the hard-drive when the system is running - no system will be happy with this. try soft-mount.

 

[vcp_ai]

Thanks again, did not knew of soft/hard option.

After reading (googling), sure I will try soft, timeo and retrans options when I realice what happend to the server...

 

[vcp_ai]

If tested nfs soft mounting with another system and seems working for my test pourposes.

Can I add this nfs soft mount option in /etc/pve/storage.conf (I can not find info about storage.conf sintax and what is allowed there).


Can I add this option to the line that actually shows options vers=3

options vers=3,soft

Or should it better to mount it in /etc/fstab and use it as a local directory ? If so could it be used in this way by more that one PVE servers ?

 

[dietmar]

Can I add this option to the line that actually shows options vers=3

options vers=3,soft

Yes (same option syntax as in fstab).

 

[vcp_ai]

Tnks