[SOLVED] Unable to activate Storage ... that no longer exist

Mar 31, 2020
7
0
6
Hi,

Yesterday, 3 storage units destined to NFS backup lost connection and their status are gone. With the intention of recovering those states again, they were remounted with the only command that allowed it: umount -f -l ... what happened next, is what I show in the attached picture.

Previously, those units of Backup were given of discharge of the Cluster in "Cluster > Storage tab"

Is there a way to recover the state of the VMs and other NFS drives without having to reboot all 3 nodes?

Could this be a safe solution?

/etc/init.d/pvestatd restart
/etc/init.d/pvedaemon restart

NOTE: For several reasons the intention is not to use those 3 BACKUP drives again.
 

Attachments

  • PVE_Status.jpg
    PVE_Status.jpg
    22.7 KB · Views: 127
  • PVE_TryReactivate.jpg
    PVE_TryReactivate.jpg
    104.7 KB · Views: 123
You can disable those NFS storage (if not needed) in the storage.cfg by writing 'disable' below an entry. Or by trying to pvesm set <STORAGE_ID> --disable 1. And you can try to restart the services.

In the end, you will not get around to resolve the hanging NFS session with a reboot of the node.
 
Are the shares now unmounted? If so, did you restart the services?
 
Yes, now are unmounted. No, I didn`t restart yet that services with:

/etc/init.d/pvestatd restart
/etc/init.d/pvedaemon restart

I wanted to know if it's safe and if that could be the solution to not rebooting the nodes before doing anything. It's a production environment.

Two more questions:

After that restart, is there any loss of connectivity with the VMs and other storage?

After the restart, how long does it take for the whole system to respond again?
 
Last edited:
I wanted to know if it's safe and if that could be the solution to not rebooting the nodes before doing anything. It's a production environment.
Only a reboot will remove the hanging nfs connections, nfs part of the kernel. The service restart might just give you the status back.

After that reboot, is there any loss of connectivity with the VMs and other storage?
If the Proxmox VE nodes can access and mount the storage, then the VM/CT can start. If it's a shared storage, you could migrate the VM/CT away before the reboot.

After the reboot, how long does it take for the whole system to respond again?
This is up to your system and configuration. You need to measure it.
 
HI,

Attached is the result of the operation carried out without the need to reboot the 3 nodes in HA in an production enviroment:

Commands launched ON EACH NODE:

service pve-cluster restart && service pvedaemon restart && service pvestatd restart && service pveproxy restart (Origin URL)

But, I still have one important doubt. When I was executing those commands in each node and as the states of each VM and Storage were being recovered I could check that the storages that caused the problem, in NODE 2 and NODE 3 were still NOT unmounted.

Maybe with the previous application of this command:

umount -f -l /mnt/pve/xxx on ALL NODES, not just on NODE 1 proxmox would have solved the problem on its own.

I'm afraid I'll never know, or at least until I find myself in the same situation again.

NOTE: I was able to see today that pvestatd and pvedaemon are running correctly. Although the attached screenshot shows it as "pvestatd > starting server"
 

Attachments

  • PVE_RestartServicesResult_final.jpg
    PVE_RestartServicesResult_final.jpg
    50.9 KB · Views: 102
  • PVE_Status_final.jpg
    PVE_Status_final.jpg
    36.8 KB · Views: 99
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!