[REVISIT] Proxmox8, datastores on nfs don't reconnect after NAS is rebooted

[manilx]

Hi

Running 3 nodes on proxmox 8 (updated from 7). I have the following issue:
I have the VM disks on a NAS (QNAP TBS464 all nvme NAS. Bought this because it doesn't use a lot of energy unlike my TVSh1288X and so te cluster can run longer on UPS battery).
When I have to reboot the NAS (firmware update or whatever) I now find that (after I shutdown all running VM's before rebootint the NAS) when the NAS comes online the 3 hosts do not reconnect to the NFS share (question mark). This is new, I never had the issue before.
I have to reboot the hosts for them to reconnect.

Any idea on what's wrong?

 

[fiona]

Hi,
I'm rather surprised it worked before to be honest. Instead of rebooting, you can also do an umount -l -f /path/to/mount to get rid of the hanging NFS mount. If the storage is enabled in Proxmox VE, it should be re-mounted automatically after a few seconds.

 

[manilx]

Weird one, so I run "umount -l -f /path/to/mount" and after that it remounts it automatically again, right?
Tried: it has taken a bit longer than a few seconds but it remounted.

 

[manilx]

Having done a lot of tests:

With all the VM's on the hosts internal local-lvm:

When I reboot the NAS and the share disconnects Proxmox UI looses connection to all VM's. Datacenter show big question mark.
No matter how the share is connected NFS/SMB.
The VM's continue running just fine. It's just the proxmox GUI which acts up.

I have to run the above command on the hosts to get connection again.

This makes absolutely no sense. Loosing a connection to a nfs datastore should not brake this havoc!!!

 

[Neobin]

When I reboot the NAS and the share disconnects Proxmox UI looses connection to all VM's. Datacenter show big question mark.
No matter how the share is connected NFS/SMB.
The VM's continue running just fine. It's just the proxmox GUI which acts up.

For reference:
https://bugzilla.proxmox.com/show_bug.cgi?id=3714

 

[manilx]

Well seems we have the bug. NOW why on earth is this not fixed.
Reports sincs 2011 and we're now on version 8?

This seems to me a major issue.

 

[manilx]

A disconnected proxmox-backup-server datastore creates the same issue!

 

[fiona]

It's just the UI and in production systems, you usually don't have disconnected storages (disable and unmount them if you do). There are plans to rework the status daemon, but it hasn't happened yet. We don't have unlimited developer resources.

 

[manilx]

Sorry AND with all due respect but this one hell of a lame excuse. This has been reported since 2021.
While these hosts are at my home lab they have paid licenses, so your revenue doesn't just come from production systems (actually this is a production system and the NAS does not disconnect all the time! Doesn't make this issue less problematic when it happens.

I'm glad I opted for xcp-ng in our factory, was a close call BECAUSE it was running it at my home lab: https://xcp-ng.org/blog/2023/05/15/leading-the-pack-mani-industrias-plasticas-sa-journey-with-vates/