A missing NFS server can destroy HA

[mgiammarco]

Hello,
I have a three node Proxmox/ceph cluster with an external nfs NAS.
I write backups on that nas. Unfortunately the nas has disconnected for an hardware problem.
Linux disconnected nfs (why??) and started to write on mount point in /mnt. So it has filled the root partition blocking the ceph mon and causing several problems to the ceph cluster.
I built a Proxmox cluster to have HA and it is incredibile that a single failure of a NAS can put down the system...
What can I do?
Thanks,
Mario

 

[fiona]

Hi,
please share the output of pveversion -v and cat /etc/pve/storage.cfg.

 

[mgiammarco]

I can send you output of that commands but it happens in many installations with different versions of proxmox. If you want to reproduce it you can configure an external nfs unit with a backup job, then power down the external unit and wait.

 

[fiona]

I cannot reproduce this. For me, the backup job fails with

TASK ERROR: could not activate storage 'mynfs': storage 'mynfs' is not online

Did you by chance configure the storage with type dir instead of type nfs? In that case you should add the option is_mountpoint 1 to the storage's configuration, so Proxmox VE can detect when nothing is mounted at the expected location.